kernel/linux.git/arch/powerpc/include/asm/uaccess.h, branch linux-7.0.y

x86-64/arm64/powerpc: clean up and rename __copy_from_user_flushcache

2026-04-22T11:32:21+00:00

commit 809b997a5ce945ab470f70c187048fe4f5df20bf upstream. This finishes the work on these odd functions that were only implemented by a handful of architectures. The 'flushcache' function was only used from the iterator code, and let's make it do the same thing that the nontemporal version does: remove the two underscores and add the user address checking. Yes, yes, the user address checking is also done at iovec import time, but we have long since walked away from the old double-underscore thing where we try to avoid address checking overhead at access time, and these functions shouldn't be so special and old-fashioned. The arm64 version already did the address check, in fact, so there it's just a matter of renaming it. For powerpc and x86-64 we now do the proper user access boilerplate. Signed-off-by: Linus Torvalds Signed-off-by: Greg Kroah-Hartman

powerpc: fix KUAP warning in VMX usercopy path

2026-03-12T05:33:47+00:00

On powerpc with PREEMPT_FULL or PREEMPT_LAZY and function tracing enabled, KUAP warnings can be triggered from the VMX usercopy path under memory stress workloads. KUAP requires that no subfunctions are called once userspace access has been enabled. The existing VMX copy implementation violates this requirement by invoking enter_vmx_usercopy() from the assembly path after userspace access has already been enabled. If preemption occurs in this window, the AMR state may not be preserved correctly, leading to unexpected userspace access state and resulting in KUAP warnings. Fix this by restructuring the VMX usercopy flow so that VMX selection and VMX state management are centralized in raw_copy_tofrom_user(), which is invoked by the raw_copy_{to,from,in}_user() wrappers. The new flow is: - raw_copy_{to,from,in}_user() calls raw_copy_tofrom_user() - raw_copy_tofrom_user() decides whether to use the VMX path based on size and CPU capability - Call enter_vmx_usercopy() before enabling userspace access - Enable userspace access as per the copy direction and perform the VMX copy - Disable userspace access as per the copy direction - Call exit_vmx_usercopy() - Fall back to the base copy routine if the VMX copy faults With this change, the VMX assembly routines no longer perform VMX state management or call helper functions; they only implement the copy operations. The previous feature-section based VMX selection inside __copy_tofrom_user_power7() is removed, and a dedicated __copy_tofrom_user_power7_vmx() entry point is introduced. This ensures correct KUAP ordering, avoids subfunction calls while KUAP is unlocked, and eliminates the warnings while preserving the VMX fast path. Fixes: de78a9c42a79 ("powerpc: Add a framework for Kernel Userspace Access Protection") Reported-by: Shrikanth Hegde Closes: https://lore.kernel.org/all/20260109064917.777587-2-sshegde@linux.ibm.com/ Suggested-by: Christophe Leroy (CS GROUP) Reviewed-by: Christophe Leroy (CS GROUP) Co-developed-by: Aboorva Devarajan Signed-off-by: Aboorva Devarajan Signed-off-by: Sayali Patil Tested-by: Shrikanth Hegde Tested-by: Venkat Rao Bagalkote Signed-off-by: Madhavan Srinivasan Link: https://patch.msgid.link/20260304122201.153049-1-sayalip@linux.ibm.com

powerpc/uaccess: Fix inline assembly for clang build on PPC32

2026-03-04T05:36:02+00:00

Test robot reports the following error with clang-16.0.6: In file included from kernel/rseq.c:75: include/linux/rseq_entry.h:141:3: error: invalid operand for instruction unsafe_get_user(offset, &ucs->post_commit_offset, efault); ^ include/linux/uaccess.h:608:2: note: expanded from macro 'unsafe_get_user' arch_unsafe_get_user(x, ptr, local_label); \ ^ arch/powerpc/include/asm/uaccess.h:518:2: note: expanded from macro 'arch_unsafe_get_user' __get_user_size_goto(__gu_val, __gu_addr, sizeof(*(p)), e); \ ^ arch/powerpc/include/asm/uaccess.h:284:2: note: expanded from macro '__get_user_size_goto' __get_user_size_allowed(x, ptr, size, __gus_retval); \ ^ arch/powerpc/include/asm/uaccess.h:275:10: note: expanded from macro '__get_user_size_allowed' case 8: __get_user_asm2(x, (u64 __user *)ptr, retval); break; \ ^ arch/powerpc/include/asm/uaccess.h:258:4: note: expanded from macro '__get_user_asm2' " li %1+1,0\n" \ ^ :7:5: note: instantiated into assembly here li 31+1,0 ^ 1 error generated. On PPC32, for 64 bits vars a pair of registers is used. Usually the lower register in the pair is the high part and the higher register is the low part. GCC uses r3/r4 ... r11/r12 ... r14/r15 ... r30/r31 In older kernel code inline assembly was using %1 and %1+1 to represent 64 bits values. However here it looks like clang uses r31 as high part, allthough r32 doesn't exist hence the error. Allthoug %1+1 should work, most places now use %L1 instead of %1+1, so let's do the same here. With that change, the build doesn't fail anymore and a disassembly shows clang uses r17/r18 and r31/r14 pair when GCC would have used r16/r17 and r30/r31: Disassembly of section .fixup: 00000000 <.fixup>: 0: 38 a0 ff f2 li r5,-14 4: 3a 20 00 00 li r17,0 8: 3a 40 00 00 li r18,0 c: 48 00 00 00 b c <.fixup+0xc> c: R_PPC_REL24 .text+0xbc 10: 38 a0 ff f2 li r5,-14 14: 3b e0 00 00 li r31,0 18: 39 c0 00 00 li r14,0 1c: 48 00 00 00 b 1c <.fixup+0x1c> 1c: R_PPC_REL24 .text+0x144 Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202602021825.otcItxGi-lkp@intel.com/ Fixes: c20beffeec3c ("powerpc/uaccess: Use flexible addressing with __put_user()/__get_user()") Signed-off-by: Christophe Leroy (CS GROUP) Acked-by: Nathan Chancellor Signed-off-by: Madhavan Srinivasan Link: https://patch.msgid.link/8ca3a657a650e497a96bfe7acde2f637dadab344.1770103646.git.chleroy@kernel.org

powerpc/uaccess: Implement masked user access

2026-01-07T04:01:05+00:00

Masked user access avoids the address/size verification by access_ok(). Allthough its main purpose is to skip the speculation in the verification of user address and size hence avoid the need of spec mitigation, it also has the advantage of reducing the amount of instructions required so it even benefits to platforms that don't need speculation mitigation, especially when the size of the copy is not know at build time. So implement masked user access on powerpc. The only requirement is to have memory gap that faults between the top user space and the real start of kernel area. On 64 bits platforms the address space is divided that way: 0xffffffffffffffff +------------------+ | | | kernel space | | | 0xc000000000000000 +------------------+ <== PAGE_OFFSET |//////////////////| |//////////////////| 0x8000000000000000 |//////////////////| |//////////////////| |//////////////////| 0x0010000000000000 +------------------+ <== TASK_SIZE_MAX | | | user space | | | 0x0000000000000000 +------------------+ Kernel is always above 0x8000000000000000 and user always below, with a gap in-between. It leads to a 3 instructions sequence: 150: 7c 69 fe 76 sradi r9,r3,63 154: 79 29 00 40 clrldi r9,r9,1 158: 7c 63 48 78 andc r3,r3,r9 This sequence leaves r3 unmodified when it is below 0x8000000000000000 and clamps it to 0x8000000000000000 if it is above. On 32 bits it is more tricky. In theory user space can go up to 0xbfffffff while kernel will usually start at 0xc0000000. So a gap needs to be added in-between. Allthough in theory a single 4k page would suffice, it is easier and more efficient to enforce a 128k gap below kernel, as it simplifies the masking. e500 has the isel instruction which allows selecting one value or the other without branch and that instruction is not speculative, so use it. Allthough GCC usually generates code using that instruction, it is safer to use inline assembly to be sure. The result is: 14: 3d 20 bf fe lis r9,-16386 18: 7c 03 48 40 cmplw r3,r9 1c: 7c 69 18 5e iselgt r3,r9,r3 On other ones, when kernel space is over 0x80000000 and user space is below, the logic in mask_user_address_simple() leads to a 3 instruction sequence: 64: 7c 69 fe 70 srawi r9,r3,31 68: 55 29 00 7e clrlwi r9,r9,1 6c: 7c 63 48 78 andc r3,r3,r9 This is the default on powerpc 8xx. When the limit between user space and kernel space is not 0x80000000, mask_user_address_32() is used and a 6 instructions sequence is generated: 24: 54 69 7c 7e srwi r9,r3,17 28: 21 29 57 ff subfic r9,r9,22527 2c: 7d 29 fe 70 srawi r9,r9,31 30: 75 2a b0 00 andis. r10,r9,45056 34: 7c 63 48 78 andc r3,r3,r9 38: 7c 63 53 78 or r3,r3,r10 The constraint is that TASK_SIZE be aligned to 128K in order to get the most optimal number of instructions. When CONFIG_PPC_BARRIER_NOSPEC is not defined, fallback on the test-based masking as it is quicker than the 6 instructions sequence but not quicker than the 3 instructions sequences above. As an exemple, allthough barrier_nospec() voids on the 8xx, this change has the following impact on strncpy_from_user(): the length of the function is reduced from 488 to 340 bytes: Start of the function with the patch: 00000000 : 0: 7c ab 2b 79 mr. r11,r5 4: 40 81 01 40 ble 144 8: 7c 89 fe 70 srawi r9,r4,31 c: 55 29 00 7e clrlwi r9,r9,1 10: 7c 84 48 78 andc r4,r4,r9 14: 3d 20 dc 00 lis r9,-9216 18: 7d 3a c3 a6 mtspr 794,r9 1c: 2f 8b 00 03 cmpwi cr7,r11,3 20: 40 9d 00 b4 ble cr7,d4 ... Start of the function without the patch: 00000000 : 0: 7c a0 2b 79 mr. r0,r5 4: 40 81 01 10 ble 114 8: 2f 84 00 00 cmpwi cr7,r4,0 c: 41 9c 01 30 blt cr7,13c 10: 3d 20 80 00 lis r9,-32768 14: 7d 24 48 50 subf r9,r4,r9 18: 7f 80 48 40 cmplw cr7,r0,r9 1c: 7c 05 03 78 mr r5,r0 20: 41 9d 01 00 bgt cr7,120 24: 3d 20 80 00 lis r9,-32768 28: 7d 25 48 50 subf r9,r5,r9 2c: 7f 84 48 40 cmplw cr7,r4,r9 30: 38 e0 ff f2 li r7,-14 34: 41 9d 00 e4 bgt cr7,118 38: 94 21 ff e0 stwu r1,-32(r1) 3c: 3d 20 dc 00 lis r9,-9216 40: 7d 3a c3 a6 mtspr 794,r9 44: 2b 85 00 03 cmplwi cr7,r5,3 48: 40 9d 01 6c ble cr7,1b4 ... 118: 7c e3 3b 78 mr r3,r7 11c: 4e 80 00 20 blr 120: 7d 25 4b 78 mr r5,r9 124: 3d 20 80 00 lis r9,-32768 128: 7d 25 48 50 subf r9,r5,r9 12c: 7f 84 48 40 cmplw cr7,r4,r9 130: 38 e0 ff f2 li r7,-14 134: 41 bd ff e4 bgt cr7,118 138: 4b ff ff 00 b 38 13c: 38 e0 ff f2 li r7,-14 140: 4b ff ff d8 b 118 ... Signed-off-by: Christophe Leroy Signed-off-by: Madhavan Srinivasan Link: https://patch.msgid.link/8f418183d9125cc0bf23922bc2ef2a1130d8b63a.1766574657.git.chleroy@kernel.org

powerpc/uaccess: Refactor user_{read/write/}_access_begin()

2026-01-07T04:01:04+00:00

user_read_access_begin() and user_write_access_begin() and user_access_begin() are now very similar. Create a common __user_access_begin() that takes direction as parameter. In order to avoid a warning with the conditional call of barrier_nospec() which is sometimes an empty macro, change it to a do {} while (0). Signed-off-by: Christophe Leroy Signed-off-by: Madhavan Srinivasan Link: https://patch.msgid.link/2b4f9d4e521e0b56bf5cb239916b4a178c4d2007.1766574657.git.chleroy@kernel.org

powerpc/uaccess: Remove {allow/prevent}_{read/write/read_write}_{from/to/}_user()

2026-01-07T04:01:04+00:00

The six following functions have become simple single-line fonctions that do not have much added value anymore: - allow_read_from_user() - allow_write_to_user() - allow_read_write_user() - prevent_read_from_user() - prevent_write_to_user() - prevent_read_write_user() Directly call allow_user_access() and prevent_user_access(), it doesn't reduce the readability and it removes unnecessary middle functions. Signed-off-by: Christophe Leroy Signed-off-by: Madhavan Srinivasan Link: https://patch.msgid.link/70971f0ba81eab742a120e5bfdeff6b42d08fd98.1766574657.git.chleroy@kernel.org

powerpc/uaccess: Move barrier_nospec() out of allow_read_{from/write}_user()

2026-01-07T04:01:04+00:00

Commit 74e19ef0ff80 ("uaccess: Add speculation barrier to copy_from_user()") added a redundant barrier_nospec() in copy_from_user(), because powerpc is already calling barrier_nospec() in allow_read_from_user() and allow_read_write_user(). But on other architectures that call to barrier_nospec() was missing. So change powerpc instead of reverting the above commit and having to fix other architectures one by one. This is now possible because barrier_nospec() has also been added in copy_from_user_iter(). Move barrier_nospec() out of allow_read_from_user() and allow_read_write_user(). This will also allow reuse of those functions when implementing masked user access which doesn't require barrier_nospec(). Don't add it back in raw_copy_from_user() as it is already called by copy_from_user() and copy_from_user_iter(). Fixes: 74e19ef0ff80 ("uaccess: Add speculation barrier to copy_from_user()") Signed-off-by: Christophe Leroy Signed-off-by: Madhavan Srinivasan Link: https://patch.msgid.link/f29612105c5fcbc8ceb7303808ddc1a781f0f6b5.1766574657.git.chleroy@kernel.org

powerpc/uaccess: Use unsafe wrappers for ASM GOTO

2025-11-03T14:26:09+00:00

ASM GOTO is miscompiled by GCC when it is used inside a auto cleanup scope: bool foo(u32 __user *p, u32 val) { scoped_guard(pagefault) unsafe_put_user(val, p, efault); return true; efault: return false; } It ends up leaking the pagefault disable counter in the fault path. clang at least fails the build. Rename unsafe_*_user() to arch_unsafe_*_user() which makes the generic uaccess header wrap it with a local label that makes both compilers emit correct code. Same for the kernel_nofault() variants. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Mathieu Desnoyers Link: https://patch.msgid.link/20251027083745.356628509@linutronix.de

powerpc/atomic: Use YZ constraints for DS-form instructions

2024-09-17T12:17:57+00:00

The 'ld' and 'std' instructions require a 4-byte aligned displacement because they are DS-form instructions. But the "m" asm constraint doesn't enforce that. That can lead to build errors if the compiler chooses a non-aligned displacement, as seen with GCC 14: /tmp/ccuSzwiR.s: Assembler messages: /tmp/ccuSzwiR.s:2579: Error: operand out of domain (39 is not a multiple of 4) make[5]: *** [scripts/Makefile.build:229: net/core/page_pool.o] Error 1 Dumping the generated assembler shows: ld 8,39(8) # MEM[(const struct atomic64_t *)_29].counter, t Use the YZ constraints to tell the compiler either to generate a DS-form displacement, or use an X-form instruction, either of which prevents the build error. See commit 2d43cc701b96 ("powerpc/uaccess: Fix build errors seen with GCC 13/14") for more details on the constraint letters. Fixes: 9f0cbea0d8cc ("[POWERPC] Implement atomic{, 64}_{read, write}() without volatile") Cc: stable@vger.kernel.org # v2.6.24+ Reported-by: Stephen Rothwell Closes: https://lore.kernel.org/all/20240913125302.0a06b4c7@canb.auug.org.au Tested-by: Mina Almasry Reviewed-by: Segher Boessenkool Signed-off-by: Michael Ellerman Link: https://msgid.link/20240916120510.2017749-1-mpe@ellerman.id.au

powerpc/uaccess: Use YZ asm constraint for ld

2024-05-30T12:57:27+00:00

The 'ld' instruction requires a 4-byte aligned displacement because it is a DS-form instruction. But the "m" asm constraint doesn't enforce that. Add a special case of __get_user_asm2_goto() so that the "YZ" constraint can be used for "ld". The "Z" constraint is documented in the GCC manual PowerPC machine constraints, and specifies a "memory operand accessed with indexed or indirect addressing". "Y" is not documented in the manual but specifies a "memory operand for a DS-form instruction". Using both allows the compiler to generate a DS-form "ld" or X-form "ldx" as appropriate. The change has to be conditional on CONFIG_PPC_KERNEL_PREFIXED because the "Y" constraint does not guarantee 4-byte alignment when prefixed instructions are enabled. No build errors have been reported due to this, but the possibility is there depending on compiler code generation decisions. Fixes: c20beffeec3c ("powerpc/uaccess: Use flexible addressing with __put_user()/__get_user()") Signed-off-by: Michael Ellerman Link: https://msgid.link/20240529123029.146953-2-mpe@ellerman.id.au