kernel/linux.git/arch/riscv/lib, branch v5.15.208

riscv: uaccess: Return the number of bytes effectively not copied

2023-08-26T12:23:36+00:00

[ Upstream commit 4b05b993900dd3eba0fc83ef5c5ddc7d65d786c6 ] It was reported that the riscv kernel hangs while executing the test in [1]. Indeed, the test hangs when trying to write a buffer to a file. The problem is that the riscv implementation of raw_copy_from_user() does not return the correct number of bytes not written when an exception happens and is fixed up, instead it always returns the initial size to copy, even if some bytes were actually copied. generic_perform_write() pre-faults the user pages and bails out if nothing can be written, otherwise it will access the userspace buffer: here the riscv implementation keeps returning it was not able to copy any byte though the pre-faulting indicates otherwise. So generic_perform_write() keeps retrying to access the user memory and ends up in an infinite loop. Note that before the commit mentioned in [1] that introduced this regression, it worked because generic_perform_write() would bail out if only one byte could not be written. So fix this by returning the number of bytes effectively not written in __asm_copy_[to|from]_user() and __clear_user(), as it is expected. Link: https://lore.kernel.org/linux-riscv/20230309151841.bomov6hq3ybyp42a@debian/ [1] Fixes: ebcbd75e3962 ("riscv: Fix the bug in memory access fixup code") Reported-by: Bo YU Closes: https://lore.kernel.org/linux-riscv/20230309151841.bomov6hq3ybyp42a@debian/#t Reported-by: Aurelien Jarno Closes: https://lore.kernel.org/linux-riscv/ZNOnCakhwIeue3yr@aurel32.net/ Signed-off-by: Alexandre Ghiti Reviewed-by: Björn Töpel Tested-by: Aurelien Jarno Reviewed-by: Aurelien Jarno Link: https://lore.kernel.org/r/20230811150604.1621784-1-alexghiti@rivosinc.com Signed-off-by: Palmer Dabbelt Signed-off-by: Sasha Levin

riscv: lib: uaccess: fix CSR_STATUS SR_SUM bit

2022-08-31T15:16:36+00:00

[ Upstream commit c08b4848f596fd95543197463b5162bd7bab2442 ] Since commit 5d8544e2d007 ("RISC-V: Generic library routines and assembly") and commit ebcbd75e3962 ("riscv: Fix the bug in memory access fixup code"), if __clear_user and __copy_user return from an fixup branch, CSR_STATUS SR_SUM bit will be set, it is a vulnerability, so that S-mode memory accesses to pages that are accessible by U-mode will success. Disable S-mode access to U-mode memory should clear SR_SUM bit. Fixes: 5d8544e2d007 ("RISC-V: Generic library routines and assembly") Fixes: ebcbd75e3962 ("riscv: Fix the bug in memory access fixup code") Signed-off-by: Chen Lifu Reviewed-by: Ben Dooks Link: https://lore.kernel.org/r/20220615014714.1650349-1-chenlifu@huawei.com Cc: stable@vger.kernel.org Signed-off-by: Palmer Dabbelt Signed-off-by: Sasha Levin

riscv: lib: uaccess: fold fixups into body

2022-08-31T15:16:36+00:00

[ Upstream commit 9d504f9aa5c1b76673018da9503e76b351a24b8c ] uaccess functions such __asm_copy_to_user(), __arch_copy_from_user() and __clear_user() place their exception fixups in the `.fixup` section without any clear association with themselves. If we backtrace the fixup code, it will be symbolized as an offset from the nearest prior symbol. Similar as arm64 does, we must move fixups into the body of the functions themselves, after the usual fast-path returns. Signed-off-by: Jisheng Zhang Signed-off-by: Palmer Dabbelt Signed-off-by: Sasha Levin

riscv: Fixed misaligned memory access. Fixed pointer comparison.

2022-04-13T18:59:09+00:00

[ Upstream commit 9d1f0ec9f71780e69ceb9d91697600c747d6e02e ] Rewrote the RISC-V memmove() assembly implementation. The previous implementation did not check memory alignment and it compared 2 pointers with a signed comparison. The misaligned memory access would cause the kernel to crash on systems that did not emulate it in firmware and did not support it in hardware. Firmware emulation is slow and may not exist. The RISC-V spec does not guarantee that support for misaligned memory accesses will exist. It should not be depended on. This patch now checks for XLEN granularity of co-alignment between the pointers. Failing that, copying is done by loading from the 2 contiguous and naturally aligned XLEN memory locations containing the overlapping XLEN sized data to be copied. The data is shifted into the correct place and binary or'ed together on each iteration. The result is then stored into the corresponding naturally aligned XLEN sized location in the destination. For unaligned data at the terminations of the regions to be copied or for copies less than (2 * XLEN) in size, byte copy is used. This patch also now uses unsigned comparison for the pointers and migrates to the newer assembler annotations from the now deprecated ones. Signed-off-by: Michael T. Kloos Signed-off-by: Palmer Dabbelt Signed-off-by: Sasha Levin

riscv: __asm_copy_to-from_user: Fix: Typos in comments

2021-07-24T00:49:12+00:00

Fixing typos and grammar mistakes and using more intuitive label name. Signed-off-by: Akira Tsukamoto Fixes: ca6eaaa210de ("riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall") Signed-off-by: Palmer Dabbelt

riscv: __asm_copy_to-from_user: Remove unnecessary size check

2021-07-24T00:49:07+00:00

Clean up: The size of 0 will be evaluated in the next step. Not required here. Signed-off-by: Akira Tsukamoto Fixes: ca6eaaa210de ("riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall") Signed-off-by: Palmer Dabbelt

riscv: __asm_copy_to-from_user: Fix: fail on RV32

2021-07-24T00:49:01+00:00

Had a bug when converting bytes to bits when the cpu was rv32. The a3 contains the number of bytes and multiple of 8 would be the bits. The LGREG is holding 2 for RV32 and 3 for RV32, so to achieve multiple of 8 it must always be constant 3. The 2 was mistakenly used for rv32. Signed-off-by: Akira Tsukamoto Fixes: ca6eaaa210de ("riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall") Signed-off-by: Palmer Dabbelt

riscv: __asm_copy_to-from_user: Fix: overrun copy

2021-07-24T00:48:52+00:00

There were two causes for the overrun memory access. The threshold size was too small. The aligning dst require one SZREG and unrolling word copy requires 8*SZREG, total have to be at least 9*SZREG. Inside the unrolling copy, the subtracting -(8*SZREG-1) would make iteration happening one extra loop. Proper value is -(8*SZREG). Signed-off-by: Akira Tsukamoto Fixes: ca6eaaa210de ("riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall") Signed-off-by: Palmer Dabbelt

riscv: __asm_copy_to-from_user: Optimize unaligned memory access and pipeline stall

2021-07-06T22:09:48+00:00

This patch will reduce cpu usage dramatically in kernel space especially for application which use sys-call with large buffer size, such as network applications. The main reason behind this is that every unaligned memory access will raise exceptions and switch between s-mode and m-mode causing large overhead. First copy in bytes until reaches the first word aligned boundary in destination memory address. This is the preparation before the bulk aligned word copy. The destination address is aligned now, but oftentimes the source address is not in an aligned boundary. To reduce the unaligned memory access, it reads the data from source in aligned boundaries, which will cause the data to have an offset, and then combines the data in the next iteration by fixing offset with shifting before writing to destination. The majority of the improving copy speed comes from this shift copy. In the lucky situation that the both source and destination address are on the aligned boundary, perform load and store with register size to copy the data. Without the unrolling, it will reduce the speed since the next store instruction for the same register using from the load will stall the pipeline. At last, copying the remainder in one byte at a time. Signed-off-by: Akira Tsukamoto Signed-off-by: Palmer Dabbelt

riscv: Add support for function error injection

2021-01-14T23:09:09+00:00

Inspired by the commit 42d038c4fb00 ("arm64: Add support for function error injection"), this patch supports function error injection for riscv. This patch mainly support two functions: one is regs_set_return_value() which is used to overwrite the return value; the another function is override_function_with_return() which is to override the probed function returning and jump to its caller. Test log: cd /sys/kernel/debug/fail_function echo sys_clone > inject echo 100 > probability echo 1 > interval ls / [ 313.176875] FAULT_INJECTION: forcing a failure. [ 313.176875] name fail_function, interval 1, probability 100, space 0, times 1 [ 313.184357] CPU: 0 PID: 87 Comm: sh Not tainted 5.8.0-rc5-00007-g6a758cc #117 [ 313.187616] Call Trace: [ 313.189100] [] walk_stackframe+0x0/0xc2 [ 313.191626] [] show_stack+0x40/0x4c [ 313.193927] [] dump_stack+0x7c/0x96 [ 313.194795] [] should_fail+0x140/0x142 [ 313.195923] [] fei_kprobe_handler+0x2c/0x5a [ 313.197687] [] kprobe_breakpoint_handler+0xb4/0x18a [ 313.200054] [] do_trap_break+0x36/0xca [ 313.202147] [] ret_from_exception+0x0/0xc [ 313.204556] [] ret_from_syscall+0x0/0x2 -sh: can't fork: Invalid argument Signed-off-by: Guo Ren Reviewed-by: Masami Hiramatsu Cc: Palmer Dabbelt Cc: Paul Walmsley Signed-off-by: Palmer Dabbelt