kernel/linux.git/arch/arm64/include, branch v6.6.132

arm64: mm: Batch dsb and isb when populating pgtables

2026-03-25T10:05:58+00:00

[ Upstream commit 1fcb7cea8a5f7747e02230f816c2c80b060d9517 ] After removing uneccessary TLBIs, the next bottleneck when creating the page tables for the linear map is DSB and ISB, which were previously issued per-pte in __set_pte(). Since we are writing multiple ptes in a given pte table, we can elide these barriers and insert them once we have finished writing to the table. Execution time of map_mem(), which creates the kernel linear map page tables, was measured on different machines with different RAM configs: | Apple M2 VM | Ampere Altra| Ampere Altra| Ampere Altra | VM, 16G | VM, 64G | VM, 256G | Metal, 512G ---------------|-------------|-------------|-------------|------------- | ms (%) | ms (%) | ms (%) | ms (%) ---------------|-------------|-------------|-------------|------------- before | 78 (0%) | 435 (0%) | 1723 (0%) | 3779 (0%) after | 11 (-86%) | 161 (-63%) | 656 (-62%) | 1654 (-56%) Signed-off-by: Ryan Roberts Tested-by: Itaru Kitayama Tested-by: Eric Chanudet Reviewed-by: Mark Rutland Reviewed-by: Ard Biesheuvel Link: https://lore.kernel.org/r/20240412131908.433043-3-ryan.roberts@arm.com Signed-off-by: Will Deacon [ Ryan: Trivial backport ] Signed-off-by: Ryan Roberts Signed-off-by: Greg Kroah-Hartman

arm64: mm: Add PTE_DIRTY back to PAGE_KERNEL* to fix kexec/hibernation

2026-03-25T10:05:52+00:00

commit c25c4aa3f79a488cc270507935a29c07dc6bddfc upstream. Commit 143937ca51cc ("arm64, mm: avoid always making PTE dirty in pte_mkwrite()") changed pte_mkwrite_novma() to only clear PTE_RDONLY when PTE_DIRTY is set. This was to allow writable-clean PTEs for swap pages that haven't actually been written. However, this broke kexec and hibernation for some platforms. Both go through trans_pgd_create_copy() -> _copy_pte(), which calls pte_mkwrite_novma() to make the temporary linear-map copy fully writable. With the updated pte_mkwrite_novma(), read-only kernel pages (without PTE_DIRTY) remain read-only in the temporary mapping. While such behaviour is fine for user pages where hardware DBM or trapping will make them writeable, subsequent in-kernel writes by the kexec relocation code will fault. Add PTE_DIRTY back to all _PAGE_KERNEL* protection definitions. This was the case prior to 5.4, commit aa57157be69f ("arm64: Ensure VM_WRITE|VM_SHARED ptes are clean by default"). With the kernel linear-map PTEs always having PTE_DIRTY set, pte_mkwrite_novma() correctly clears PTE_RDONLY. Fixes: 143937ca51cc ("arm64, mm: avoid always making PTE dirty in pte_mkwrite()") Signed-off-by: Catalin Marinas Cc: stable@vger.kernel.org Reported-by: Jianpeng Chang Link: https://lore.kernel.org/r/20251204062722.3367201-1-jianpeng.chang.cn@windriver.com Cc: Will Deacon Cc: Huang, Ying Cc: Guenter Roeck Reviewed-by: Huang Ying Signed-off-by: Will Deacon Signed-off-by: Greg Kroah-Hartman

arm64: Fix non-atomic __READ_ONCE() with CONFIG_LTO=y

2026-03-04T12:21:14+00:00

[ Upstream commit bb0c99e08ab9aa6d04b40cb63c72db9950d51749 ] The implementation of __READ_ONCE() under CONFIG_LTO=y incorrectly qualified the fallback "once" access for types larger than 8 bytes, which are not atomic but should still happen "once" and suppress common compiler optimizations. The cast `volatile typeof(__x)` applied the volatile qualifier to the pointer type itself rather than the pointee. This created a volatile pointer to a non-volatile type, which violated __READ_ONCE() semantics. Fix this by casting to `volatile typeof(*__x) *`. With a defconfig + LTO + debug options build, we see the following functions to be affected: xen_manage_runstate_time (884 -> 944 bytes) xen_steal_clock (248 -> 340 bytes) ^-- use __READ_ONCE() to load vcpu_runstate_info structs Fixes: e35123d83ee3 ("arm64: lto: Strengthen READ_ONCE() to acquire when CONFIG_LTO=y") Cc: stable@vger.kernel.org Reviewed-by: Boqun Feng Signed-off-by: Marco Elver Tested-by: David Laight Signed-off-by: Will Deacon Signed-off-by: Sasha Levin

hyperv-tlfs: Change prefix of generic HV_REGISTER_* MSRs to HV_MSR_*

2026-01-30T09:27:31+00:00

[ Upstream commit 0e3f7d120086c8b9d6e1ae0dd4917fc529daa1ca ] The HV_REGISTER_ are used as arguments to hv_set/get_register(), which delegate to arch-specific mechanisms for getting/setting synthetic Hyper-V MSRs. On arm64, HV_REGISTER_ defines are synthetic VP registers accessed via the get/set vp registers hypercalls. The naming matches the TLFS document, although these register names are not specific to arm64. However, on x86 the prefix HV_REGISTER_ indicates Hyper-V MSRs accessed via rdmsrl()/wrmsrl(). This is not consistent with the TLFS doc, where HV_REGISTER_ is *only* used for used for VP register names used by the get/set register hypercalls. To fix this inconsistency and prevent future confusion, change the arch-generic aliases used by callers of hv_set/get_register() to have the prefix HV_MSR_ instead of HV_REGISTER_. Use the prefix HV_X64_MSR_ for the x86-only Hyper-V MSRs. On x86, the generic HV_MSR_'s point to the corresponding HV_X64_MSR_. Move the arm64 HV_REGISTER_* defines to the asm-generic hyperv-tlfs.h, since these are not specific to arm64. On arm64, the generic HV_MSR_'s point to the corresponding HV_REGISTER_. While at it, rename hv_get/set_registers() and related functions to hv_get/set_msr(), hv_get/set_nested_msr(), etc. These are only used for Hyper-V MSRs and this naming makes that clear. Signed-off-by: Nuno Das Neves Reviewed-by: Wei Liu Reviewed-by: Michael Kelley Link: https://lore.kernel.org/r/1708440933-27125-1-git-send-email-nunodasneves@linux.microsoft.com Signed-off-by: Wei Liu Message-ID: <1708440933-27125-1-git-send-email-nunodasneves@linux.microsoft.com> Stable-dep-of: 49f49d47af67 ("Drivers: hv: Always do Hyper-V panic notification in hv_kmsg_dump()") Signed-off-by: Sasha Levin

arm64, mm: avoid always making PTE dirty in pte_mkwrite()

2025-10-29T13:07:02+00:00

[ Upstream commit 143937ca51cc6ae2fccc61a1cb916abb24cd34f5 ] Current pte_mkwrite_novma() makes PTE dirty unconditionally. This may mark some pages that are never written dirty wrongly. For example, do_swap_page() may map the exclusive pages with writable and clean PTEs if the VMA is writable and the page fault is for read access. However, current pte_mkwrite_novma() implementation always dirties the PTE. This may cause unnecessary disk writing if the pages are never written before being reclaimed. So, change pte_mkwrite_novma() to clear the PTE_RDONLY bit only if the PTE_DIRTY bit is set to make it possible to make the PTE writable and clean. The current behavior was introduced in commit 73e86cb03cf2 ("arm64: Move PTE_RDONLY bit handling out of set_pte_at()"). Before that, pte_mkwrite() only sets the PTE_WRITE bit, while set_pte_at() only clears the PTE_RDONLY bit if both the PTE_WRITE and the PTE_DIRTY bits are set. To test the performance impact of the patch, on an arm64 server machine, run 16 redis-server processes on socket 1 and 16 memtier_benchmark processes on socket 0 with mostly get transactions (that is, redis-server will mostly read memory only). The memory footprint of redis-server is larger than the available memory, so swap out/in will be triggered. Test results show that the patch can avoid most swapping out because the pages are mostly clean. And the benchmark throughput improves ~23.9% in the test. Fixes: 73e86cb03cf2 ("arm64: Move PTE_RDONLY bit handling out of set_pte_at()") Signed-off-by: Huang Ying Cc: Will Deacon Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Gavin Shan Cc: Ard Biesheuvel Cc: Matthew Wilcox (Oracle) Cc: Yicong Yang Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Reviewed-by: Catalin Marinas Signed-off-by: Catalin Marinas Signed-off-by: Sasha Levin

arm64: cputype: Add Neoverse-V3AE definitions

2025-10-23T14:16:37+00:00

commit 3bbf004c4808e2c3241e5c1ad6cc102f38a03c39 upstream. Add cputype definitions for Neoverse-V3AE. These will be used for errata detection in subsequent patches. These values can be found in the Neoverse-V3AE TRM: https://developer.arm.com/documentation/SDEN-2615521/9-0/ ... in section A.6.1 ("MIDR_EL1, Main ID Register"). Signed-off-by: Mark Rutland Cc: James Morse Cc: Will Deacon Cc: Catalin Marinas Signed-off-by: Ryan Roberts Signed-off-by: Will Deacon Signed-off-by: Ryan Roberts Signed-off-by: Greg Kroah-Hartman

arm64: ftrace: fix unreachable PLT for ftrace_caller in init_module with CONFIG_DYNAMIC_FTRACE

2025-09-09T16:56:26+00:00

commit a7ed7b9d0ebb038db9963d574da0311cab0b666a upstream. On arm64, it has been possible for a module's sections to be placed more than 128M away from each other since commit: commit 3e35d303ab7d ("arm64: module: rework module VA range selection") Due to this, an ftrace callsite in a module's .init.text section can be out of branch range for the module's ftrace PLT entry (in the module's .text section). Any attempt to enable tracing of that callsite will result in a BRK being patched into the callsite, resulting in a fatal exception when the callsite is later executed. Fix this by adding an additional trampoline for .init.text, which will be within range. No additional trampolines are necessary due to the way a given module's executable sections are packed together. Any executable section beginning with ".init" will be placed in MOD_INIT_TEXT, and any other executable section, including those beginning with ".exit", will be placed in MOD_TEXT. Fixes: 3e35d303ab7d ("arm64: module: rework module VA range selection") Cc: # 6.5.x Signed-off-by: panfan Acked-by: Mark Rutland Link: https://lore.kernel.org/r/20250905032236.3220885-1-panfan@qti.qualcomm.com Signed-off-by: Catalin Marinas Signed-off-by: Greg Kroah-Hartman

sched/topology: Add a new arch_scale_freq_ref() method

2025-08-28T14:28:42+00:00

commit 9942cb22ea458c34fa17b73d143ea32d4df1caca upstream. Create a new method to get a unique and fixed max frequency. Currently cpuinfo.max_freq or the highest (or last) state of performance domain are used as the max frequency when computing the frequency for a level of utilization, but: - cpuinfo_max_freq can change at runtime. boost is one example of such change. - cpuinfo.max_freq and last item of the PD can be different leading to different results between cpufreq and energy model. We need to save the reference frequency that has been used when computing the CPUs capacity and use this fixed and coherent value to convert between frequency and CPU's capacity. In fact, we already save the frequency that has been used when computing the capacity of each CPU. We extend the precision to save kHz instead of MHz currently and we modify the type to be aligned with other variables used when converting frequency to capacity and the other way. [ mingo: Minor edits. ] Signed-off-by: Vincent Guittot Signed-off-by: Ingo Molnar Tested-by: Lukasz Luba Reviewed-by: Lukasz Luba Acked-by: Sudeep Holla Link: https://lore.kernel.org/r/20231211104855.558096-2-vincent.guittot@linaro.org Stable-dep-of: e37617c8e53a ("sched/fair: Fix frequency selection for non-invariant case") Signed-off-by: Wentao Guan Signed-off-by: Greg Kroah-Hartman

arm64: Handle KCOV __init vs inline mismatches

2025-08-28T14:28:14+00:00

[ Upstream commit 65c430906efffee9bd7551d474f01a6b1197df90 ] GCC appears to have kind of fragile inlining heuristics, in the sense that it can change whether or not it inlines something based on optimizations. It looks like the kcov instrumentation being added (or in this case, removed) from a function changes the optimization results, and some functions marked "inline" are _not_ inlined. In that case, we end up with __init code calling a function not marked __init, and we get the build warnings I'm trying to eliminate in the coming patch that adds __no_sanitize_coverage to __init functions: WARNING: modpost: vmlinux: section mismatch in reference: acpi_get_enable_method+0x1c (section: .text.unlikely) -> acpi_psci_present (section: .init.text) This problem is somewhat fragile (though using either __always_inline or __init will deterministically solve it), but we've tripped over this before with GCC and the solution has usually been to just use __always_inline and move on. For arm64 this requires forcing one ACPI function to be inlined with __always_inline. Link: https://lore.kernel.org/r/20250724055029.3623499-1-kees@kernel.org Signed-off-by: Kees Cook Signed-off-by: Sasha Levin

arm64/entry: Mask DAIF in cpu_switch_to(), call_on_irq_stack()

2025-08-01T08:47:30+00:00

commit d42e6c20de6192f8e4ab4cf10be8c694ef27e8cb upstream. `cpu_switch_to()` and `call_on_irq_stack()` manipulate SP to change to different stacks along with the Shadow Call Stack if it is enabled. Those two stack changes cannot be done atomically and both functions can be interrupted by SErrors or Debug Exceptions which, though unlikely, is very much broken : if interrupted, we can end up with mismatched stacks and Shadow Call Stack leading to clobbered stacks. In `cpu_switch_to()`, it can happen when SP_EL0 points to the new task, but x18 stills points to the old task's SCS. When the interrupt handler tries to save the task's SCS pointer, it will save the old task SCS pointer (x18) into the new task struct (pointed to by SP_EL0), clobbering it. In `call_on_irq_stack()`, it can happen when switching from the task stack to the IRQ stack and when switching back. In both cases, we can be interrupted when the SCS pointer points to the IRQ SCS, but SP points to the task stack. The nested interrupt handler pushes its return addresses on the IRQ SCS. It then detects that SP points to the task stack, calls `call_on_irq_stack()` and clobbers the task SCS pointer with the IRQ SCS pointer, which it will also use ! This leads to tasks returning to addresses on the wrong SCS, or even on the IRQ SCS, triggering kernel panics via CONFIG_VMAP_STACK or FPAC if enabled. This is possible on a default config, but unlikely. However, when enabling CONFIG_ARM64_PSEUDO_NMI, DAIF is unmasked and instead the GIC is responsible for filtering what interrupts the CPU should receive based on priority. Given the goal of emulating NMIs, pseudo-NMIs can be received by the CPU even in `cpu_switch_to()` and `call_on_irq_stack()`, possibly *very* frequently depending on the system configuration and workload, leading to unpredictable kernel panics. Completely mask DAIF in `cpu_switch_to()` and restore it when returning. Do the same in `call_on_irq_stack()`, but restore and mask around the branch. Mask DAIF even if CONFIG_SHADOW_CALL_STACK is not enabled for consistency of behaviour between all configurations. Introduce and use an assembly macro for saving and masking DAIF, as the existing one saves but only masks IF. Cc: Signed-off-by: Ada Couprie Diaz Reported-by: Cristian Prundeanu Fixes: 59b37fe52f49 ("arm64: Stash shadow stack pointer in the task struct on interrupt") Tested-by: Cristian Prundeanu Acked-by: Will Deacon Link: https://lore.kernel.org/r/20250718142814.133329-1-ada.coupriediaz@arm.com Signed-off-by: Will Deacon Signed-off-by: Greg Kroah-Hartman