kernel/linux.git/arch/s390/lib/uaccess.c, branch linux-7.0.y

KVM: s390: Switch to new gmap

2026-02-04T16:00:10+00:00

Switch KVM/s390 to use the new gmap code. Remove includes to and include "gmap.h" instead; fix all the existing users of the old gmap functions to use the new ones instead. Fix guest storage key access functions to work with the new gmap. Acked-by: Heiko Carstens Signed-off-by: Claudio Imbrenda

s390/uaccess: Merge cmpxchg_user_key() inline assemblies

2025-06-29T11:12:02+00:00

The inline assemblies for __cmpxchg_user_key1() and __cmpxchg_user_key2() are identical. Get rid of the duplication and provide a common helper function. Suggested-by: Mete Durlu Acked-by: Alexander Gordeev Signed-off-by: Heiko Carstens Signed-off-by: Alexander Gordeev

s390/uaccess: Prevent kprobes on cmpxchg_user_key() functions

2025-06-29T11:12:02+00:00

Code regions within cmpxchg_user_key() functions may be executed with a non-default access key, which may lead to a protection exception if the corresponding page has the fetch-protection bit enabled. There is code in place which initializes the storage keys of such pages when needed. However there is also the possibility of out-of-line execution of such code in case a kprobe is set within such a region. To avoid this problem prevent that any kprobe can be set within the cmpxchg_user_key() functions. Reviewed-by: Claudio Imbrenda Signed-off-by: Heiko Carstens Signed-off-by: Alexander Gordeev

s390/uaccess: Initialize code pages executed with non-default access key

2025-06-29T11:12:02+00:00

cmpxchg_user_key() may be executed with a non-zero key; if then the storage key of the page which belongs to the cmpxchg_user_key() code contains a key with fetch-protection enabled the result is a protection exception: Unable to handle kernel pointer dereference in virtual kernel address space Failing address: 0000000000000000 TEID: 000000000000080b Fault in home space mode while using kernel ASCE. AS:0000000002528007 R3:00000001ffffc007 S:00000001ffffb801 P:000000000000013d Oops: 0004 ilc:1 [#1]SMP Modules linked in: CPU: 3 UID: 0 PID: 791 Comm: memop Not tainted 6.16.0-rc1-00006-g3b568201d0a6-dirty #11 NONE Hardware name: IBM 3931 A01 704 (z/VM 7.4.0) Krnl PSW : 0794f00180000000 000003ffe0f4d91e (__cmpxchg_user_key1+0xbe/0x190) R:0 T:1 IO:1 EX:1 Key:9 M:1 W:0 P:0 AS:3 CC:3 PM:0 RI:0 EA:3 Krnl GPRS: 070003ffdfbf6af0 0000000000070000 0000000095b5a300 0000000000000000 00000000f1000000 0000000000000000 0000000000000090 0000000000000000 0000000000000040 0000000000000018 000003ff9b23d000 0000037fe0ef7bd8 000003ffdfbf7500 00000000962e4000 0000037f00ffffff 0000037fe0ef7aa0 Krnl Code: 000003ffe0f4d912: ad03f0a0 stosm 160(%r15),3 000003ffe0f4d916: a7780000 lhi %r7,0 #000003ffe0f4d91a: b20a6000 spka 0(%r6) >000003ffe0f4d91e: b2790100 sacf 256 000003ffe0f4d922: a56f0080 llill %r6,128 000003ffe0f4d926: 5810a000 l %r1,0(%r10) 000003ffe0f4d92a: 141e nr %r1,%r14 000003ffe0f4d92c: c0e7ffffffff xilf %r14,4294967295 Call Trace: [<000003ffe0f4d91e>] __cmpxchg_user_key1+0xbe/0x190 [<000003ffe0189c6e>] cmpxchg_guest_abs_with_key+0x2fe/0x370 [<000003ffe016d28e>] kvm_s390_vm_mem_op_cmpxchg+0x17e/0x350 [<000003ffe0173284>] kvm_arch_vm_ioctl+0x354/0x6f0 [<000003ffe015fedc>] kvm_vm_ioctl+0x2cc/0x6e0 [<000003ffe05348ae>] vfs_ioctl+0x2e/0x70 [<000003ffe0535e70>] __s390x_sys_ioctl+0xe0/0x100 [<000003ffe0f40f06>] __do_syscall+0x136/0x340 [<000003ffe0f4cb2e>] system_call+0x6e/0x90 Last Breaking-Event-Address: [<000003ffe0f4d896>] __cmpxchg_user_key1+0x36/0x190 Fix this by defining all code ranges within cmpxchg_user_key() functions which may be executed with a non-default key and explicitly initialize storage keys by calling skey_regions_initialize(). Reviewed-by: Claudio Imbrenda Signed-off-by: Heiko Carstens Signed-off-by: Alexander Gordeev

s390/uaccess: Make cmpxchg_user_key() library code

2025-06-29T11:12:01+00:00

Move cmpxchg_user_key() handling to uaccess library code. The generated code is large in any case so that there is hardly any benefit if it is inlined. Reviewed-by: Claudio Imbrenda Signed-off-by: Heiko Carstens Signed-off-by: Alexander Gordeev

s390/mm: Reimplement lazy ASCE handling

2025-04-14T09:23:21+00:00

Reduce system call overhead time (round trip time for invoking a non-existent system call) by 25%. With the removal of set_fs() [1] lazy control register handling was removed in order to keep kernel entry and exit simple. However this made system calls slower. With the conversion to generic entry [2] and numerous follow up changes which simplified the entry code significantly, adding support for lazy asce handling doesn't add much complexity to the entry code anymore. In particular this means: - On kernel entry the primary asce is not modified and contains the user asce - Kernel accesses which require secondary-space mode (for example futex operations) are surrounded by enable_sacf_uaccess() and disable_sacf_uaccess() calls. enable_sacf_uaccess() sets the primary asce to kernel asce so that the sacf instruction can be used to switch to secondary-space mode. The primary asce is changed back to user asce with disable_sacf_uaccess(). The state of the control register which contains the primary asce is reflected with a new TIF_ASCE_PRIMARY bit. This is required on context switch so that the correct asce is restored for the scheduled in process. In result address spaces are now setup like this: CPU running in | %cr1 ASCE | %cr7 ASCE | %cr13 ASCE -----------------------------|-----------|-----------|----------- user space | user | user | kernel kernel (no sacf) | user | user | kernel kernel (during sacf uaccess) | kernel | user | kernel kernel (kvm guest execution) | guest | user | kernel In result cr1 control register content is not changed except for: - futex system calls - legacy s390 PCI system calls - the kvm specific cmpxchg_user_key() uaccess helper This leads to faster system call execution. [1] 87d598634521 ("s390/mm: remove set_fs / rework address space handling") [2] 56e62a737028 ("s390: convert to generic entry") Reviewed-by: Alexander Gordeev Signed-off-by: Heiko Carstens

s390/uaccess: Inline __clear_user()

2025-03-04T16:18:04+00:00

Rework __clear_user() similar to raw_copy_from_user() / raw_copy_to_user() and inline the function saving the overhead of branches. Signed-off-by: Heiko Carstens Signed-off-by: Vasily Gorbik

s390/uaccess: Separate key uaccess functions

2025-03-04T16:18:03+00:00

Implement separate raw_copy_to_user_key() and raw_copy_from_user_key() functions, which allows to remove the open-coded operand access control handling from the normal raw_copy_to_user() / raw_copy_from_user() functions - they are simplified to use immediate instructions to load hard-coded operand access control values into register zero, which saves one instruction. Signed-off-by: Heiko Carstens Signed-off-by: Vasily Gorbik

s390/uaccess: Remove INLINE_COPY_FROM_USER and INLINE_COPY_TO_USER

2025-01-26T16:24:07+00:00

The s390 implementations of raw_copy_from_user() and raw_copy_to_user() are never inlined. However INLINE_COPY_FROM_USER and INLINE_COPY_TO_USER are still set. This leads to the odd situation that only the error handling (memset to zero of the not copied bytes) of copy_from_user() is inlined, while the actual fast path code is out-of-line. This would make sense if raw_copy_from_user() and raw_copy_to_user() were implemented in assembler files, where inlining is not possible. But the current s390 setup does not make any sense. Address this by moving the raw uaccess copy inline assemblies to the uaccess header file, and remove INLINE_COPY_FROM_USER and INLINE_COPY_TO_USER definitions. This way the uaccess code, but now including error handling, is still out-of-line with the common code _copy_from_user() and _copy_to_user() variants, which inline the raw uaccess functions via _inline_copy_from_user() and _inline_copy_to_user(). This reduces the size of the kernel image by ~17kb. (defconfig, gcc 14.2.0) Acked-by: Alexander Gordeev Signed-off-by: Heiko Carstens Signed-off-by: Alexander Gordeev

s390: Replace S390_lowcore by get_lowcore()

2024-06-18T15:01:33+00:00

Replace all S390_lowcore usages in arch/s390/ by get_lowcore(). Acked-by: Heiko Carstens Signed-off-by: Sven Schnelle Signed-off-by: Vasily Gorbik