kernel/linux.git/arch/arm64/include/asm/asm-extable.h, branch linux-7.0.yLinux kernel stable tree (mirror)https://git.radix-linux.su/kernel/linux.git/atom?h=linux-7.0.y2025-11-11T19:35:59+00:00arm64: Replace __ASSEMBLY__ with __ASSEMBLER__ in non-uapi headers2025-11-11T19:35:59+00:00Thomas Huththuth@redhat.com2025-10-10T13:01:16+00:00urn:sha1:287d163322b743a50adcad25c851600c004f59e3
While the GCC and Clang compilers already define __ASSEMBLER__
automatically when compiling assembly code, __ASSEMBLY__ is a
macro that only gets defined by the Makefiles in the kernel.
This can be very confusing when switching between userspace
and kernelspace coding, or when dealing with uapi headers that
rather should use __ASSEMBLER__ instead. So let's standardize now
on the __ASSEMBLER__ macro that is provided by the compilers.
This is a mostly mechanical patch (done with a simple "sed -i"
statement), except for the following files where comments with
mis-spelled macros were tweaked manually:
arch/arm64/include/asm/stacktrace/frame.h
arch/arm64/include/asm/kvm_ptrauth.h
arch/arm64/include/asm/debug-monitors.h
arch/arm64/include/asm/esr.h
arch/arm64/include/asm/scs.h
arch/arm64/include/asm/memory.h
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
arm64: extable: Add fixup handling for uaccess CPY* instructions2025-03-07T16:18:06+00:00Kristina Martšenkokristina.martsenko@arm.com2025-02-28T17:00:04+00:00urn:sha1:653884f88777b8858ef438cd8ee6ae4722fd5553
A subsequent patch will use CPY* instructions to copy between user and
kernel memory. Add a new exception fixup type to avoid fixing up faults
on kernel memory accesses, in order to make it easier to debug kernel
bugs and to keep the same behavior as with regular loads/stores.
Signed-off-by: Kristina Martšenko <kristina.martsenko@arm.com>
Reviewed-by: Robin Murphy <robin.murphy@arm.com>
Link: https://lore.kernel.org/r/20250228170006.390100-2-kristina.martsenko@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
arm64: start using 'asm goto' for put_user()2024-06-19T19:33:38+00:00Linus Torvaldstorvalds@linux-foundation.org2024-06-01T22:04:53+00:00urn:sha1:7fd298d4b39d8d5fe99d56811a7ed78c7a5377d5
This generates noticeably better code since we don't need to test the
error register etc, the exception just jumps to the error handling
directly.
Unlike get_user(), there's no need to worry about old compilers. All
supported compilers support the regular non-output 'asm goto', as
pointed out by Nathan Chancellor.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
arm64: extable: cleanup redundant extable type EX_TYPE_FIXUP2022-06-28T11:11:47+00:00Tong Tiangentongtiangen@huawei.com2022-06-21T07:26:34+00:00urn:sha1:bacac637025c2d73c2af5188e5a12008c5869df9
Currently, extable type EX_TYPE_FIXUP is no place to use, We can safely
remove it.
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20220621072638.1273594-7-tongtiangen@huawei.com
Signed-off-by: Will Deacon <will@kernel.org>
arm64: extable: move _cond_extable to _cond_uaccess_extable2022-06-28T11:11:47+00:00Tong Tiangentongtiangen@huawei.com2022-06-21T07:26:33+00:00urn:sha1:e4208e80a353b5ee1d2ae4681bd41f62a3391958
Currently, We use _cond_extable for cache maintenance uaccess helper
caches_clean_inval_user_pou(), so this should be moved over to
EX_TYPE_UACCESS_ERR_ZERO and rename _cond_extable to _cond_uaccess_extable
for clarity.
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20220621072638.1273594-6-tongtiangen@huawei.com
Signed-off-by: Will Deacon <will@kernel.org>
arm64: extable: make uaaccess helper use extable type EX_TYPE_UACCESS_ERR_ZERO2022-06-28T11:11:47+00:00Tong Tiangentongtiangen@huawei.com2022-06-21T07:26:32+00:00urn:sha1:c4ed0d73ede96f1f9a20b5af17aa793ca91f8c4c
Currnetly, the extable type used by __arch_copy_from/to_user() is
EX_TYPE_FIXUP. In fact, It is more clearly to use meaningful
EX_TYPE_UACCESS_*.
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Link: https://lore.kernel.org/r/20220621072638.1273594-5-tongtiangen@huawei.com
Signed-off-by: Will Deacon <will@kernel.org>
arm64: asm-extable: add asm uacess helpers2022-06-28T11:11:41+00:00Mark Rutlandmark.rutland@arm.com2022-06-21T07:26:31+00:00urn:sha1:59e8a1ce8f7b7dd3cf5979ded908158e7e8bd3bc
In subsequent patches we want to explciitly annotate uaccess fixups in
assembly files.
We have existing helpers for this for inline assembly, but due to
differing stringification requirements it's not possible to have a
single definition that we can use for both inline asm and plain asm
files. So as with other cases (e.g. gpr-regnum.h), we must prove
separate helprs for plain asm and inline asm.
So that we can do so, this patch adds helpers to define
EX_TYPE_UACCESS_ERR_ZERO fixups in plain assembly. These correspond 1-1
with the inline assembly versions except for the absence of
stringification. No plain assmebly heleprs are added for
EX_TYPE_LOAD_UNALIGNED_ZEROPAD fixups as these only exist for a single C
function.
For copy_{to,from}_user() we'll need fixups with regs and err, so I've
added _ASM_EXTABLE_UACCESS(insn, fixup), where both the error and zero
registers are WZR.
For clarity, the existing `_asm_extable` assemgbly maco is now defined
in terms of the _ASM_EXTABLE() CPP macro, making the CPP macros
canonical in all cases.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Link: https://lore.kernel.org/r/20220621072638.1273594-4-tongtiangen@huawei.com
Signed-off-by: Will Deacon <will@kernel.org>
arm64: asm-extable: move data fields2022-06-28T11:11:27+00:00Mark Rutlandmark.rutland@arm.com2022-06-21T07:26:30+00:00urn:sha1:5519d7de2f8e81c317034d2f08362285a0e7ab42
In subsequent patches we'll need to fill in extable data fields in
regular assembly files. In preparation for this, move the definitions of
the extable data fields earlier in asm-extable.h so that they are
defined for both assembly and C files.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Link: https://lore.kernel.org/r/20220621072638.1273594-3-tongtiangen@huawei.com
Signed-off-by: Will Deacon <will@kernel.org>
arm64: extable: add new extable type EX_TYPE_KACCESS_ERR_ZERO support2022-06-28T11:11:00+00:00Tong Tiangentongtiangen@huawei.com2022-06-21T07:26:29+00:00urn:sha1:4953fc3d3270b7466a3b334838b76308c75f7751
Currently, The extable type EX_TYPE_UACCESS_ERR_ZERO is used by
__get/put_kernel_nofault(), but those helpers are not uaccess type, so we
add a new extable type EX_TYPE_KACCESS_ERR_ZERO which can be used by
__get/put_kernel_no_fault().
This is also to prepare for distinguishing the two types in machine check
safe process.
Suggested-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Tong Tiangen <tongtiangen@huawei.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20220621072638.1273594-2-tongtiangen@huawei.com
Signed-off-by: Will Deacon <will@kernel.org>
arm64: extable: add load_unaligned_zeropad() handler2021-10-21T09:45:22+00:00Mark Rutlandmark.rutland@arm.com2021-10-19T16:02:18+00:00urn:sha1:753b32368705c396000f95f33c3b7018474e33ad
For inline assembly, we place exception fixups out-of-line in the
`.fixup` section such that these are out of the way of the fast path.
This has a few drawbacks:
* Since the fixup code is anonymous, backtraces will symbolize fixups as
offsets from the nearest prior symbol, currently
`__entry_tramp_text_end`. This is confusing, and painful to debug
without access to the relevant vmlinux.
* Since the exception handler adjusts the PC to execute the fixup, and
the fixup uses a direct branch back into the function it fixes,
backtraces of fixups miss the original function. This is confusing,
and violates requirements for RELIABLE_STACKTRACE (and therefore
LIVEPATCH).
* Inline assembly and associated fixups are generated from templates,
and we have many copies of logically identical fixups which only
differ in which specific registers are written to and which address is
branched to at the end of the fixup. This is potentially wasteful of
I-cache resources, and makes it hard to add additional logic to fixups
without significant bloat.
* In the case of load_unaligned_zeropad(), the logic in the fixup
requires a temporary register that we must allocate even in the
fast-path where it will not be used.
This patch address all four concerns for load_unaligned_zeropad() fixups
by adding a dedicated exception handler which performs the fixup logic
in exception context and subsequent returns back after the faulting
instruction. For the moment, the fixup logic is identical to the old
assembly fixup logic, but in future we could enhance this by taking the
ESR and FAR into account to constrain the faults we try to fix up, or to
specialize fixups for MTE tag check faults.
Other than backtracing, there should be no functional change as a result
of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20211019160219.5202-13-mark.rutland@arm.com
Signed-off-by: Will Deacon <will@kernel.org>