kernel/linux.git/arch/arm/mm/context.c, branch linux-7.1.y

ARM: mm: make vmalloc_seq handling SMP safe

2022-01-25T08:53:52+00:00

Rework the vmalloc_seq handling so it can be used safely under SMP, as we started using it to ensure that vmap'ed stacks are guaranteed to be mapped by the active mm before switching to a task, and here we need to ensure that changes to the page tables are visible to other CPUs when they observe a change in the sequence count. Since LPAE needs none of this, fold a check against it into the vmalloc_seq counter check after breaking it out into a separate static inline helper. Given that vmap'ed stacks are now also supported on !SMP configurations, let's drop the WARN() that could potentially now fire spuriously. Signed-off-by: Ard Biesheuvel

ARM: 9150/1: Fix PID_IN_CONTEXTIDR regression when THREAD_INFO_IN_TASK=y

2021-10-30T10:24:36+00:00

The code that implements the rarely used PID_IN_CONTEXTIDR feature dereferences the 'task' field of struct thread_info directly, and this is no longer possible when THREAD_INFO_IN_TASK=y, as the 'task' field is omitted from the struct definition in that case. Instead, we should just cast the thread_info pointer to a task_struct pointer, given that the former is now the first member of the latter. So use a helper that abstracts this, and provide implementations for both cases. Reported by: Arnd Bergmann Fixes: 18ed1c01a7dd ("ARM: smp: Enable THREAD_INFO_IN_TASK") Signed-off-by: Ard Biesheuvel Signed-off-by: Russell King (Oracle)

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 500

2019-06-19T15:09:55+00:00

Based on 2 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation this program is free software you can redistribute it and or modify it under the terms of the gnu general public license version 2 as published by the free software foundation # extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 4122 file(s). Signed-off-by: Thomas Gleixner Reviewed-by: Enrico Weigelt Reviewed-by: Kate Stewart Reviewed-by: Allison Randal Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190604081206.933168790@linutronix.de Signed-off-by: Greg Kroah-Hartman

ARM: 8465/1: mm: keep reserved ASIDs in sync with mm after multiple rollovers

2015-12-02T23:57:54+00:00

Under some unusual context-switching patterns, it is possible to end up with multiple threads from the same mm running concurrently with different ASIDs: 1. CPU x schedules task t with mm p containing ASID a and generation g This task doesn't block and the CPU doesn't context switch. So: * per_cpu(active_asid, x) = {g,a} * p->context.id = {g,a} 2. Some other CPU generates an ASID rollover. The global generation is now (g + 1). CPU x is still running t, with no context switch and so per_cpu(reserved_asid, x) = {g,a} 3. CPU y schedules task t', which shares mm p with t. The generation mismatches, so we take the slowpath and hit the reserved ASID from CPU x. p is then updated so that p->context.id = {g + 1,a} 4. CPU y schedules some other task u, which has an mm != p. 5. Some other CPU generates *another* CPU rollover. The global generation is now (g + 2). CPU x is still running t, with no context switch and so per_cpu(reserved_asid, x) = {g,a}. 6. CPU y once again schedules task t', but now *fails* to hit the reserved ASID from CPU x because of the generation mismatch. This results in a new ASID being allocated, despite the fact that t is still running on CPU x with the same mm. Consequently, TLBIs (e.g. as a result of CoW) will not be synchronised between the two threads. This patch fixes the problem by updating all of the matching reserved ASIDs when we hit on the slowpath (i.e. in step 3 above). This keeps the reserved ASIDs in-sync with the mm and avoids the problem. Cc: Reported-by: Tony Thompson Reviewed-by: Catalin Marinas Signed-off-by: Will Deacon Signed-off-by: Russell King

ARM: 8299/1: mm: ensure local active ASID is marked as allocated on rollover

2015-02-03T12:57:33+00:00

Commit e1a5848e3398 ("ARM: 7924/1: mm: don't bother with reserved ttbr0 when running with LPAE") removed the use of the reserved TTBR0 value for LPAE systems, since the ASID is held in the TTBR and can be updated atomicly with the pgd of the next mm. Unfortunately, this patch forgot to update flush_context, which deliberately avoids marking the local active ASID as allocated, since we used to switch via ASID zero and didn't need to allocate the ASID of the previous mm. The side-effect of this is that we can allocate the same ASID to the next mm and, between flushing the local TLB and updating TTBR0, we can perform speculative TLB fills for userspace nG mappings using the page table of the previous mm. The consequence of this is that the next mm can erroneously hit some mappings of the previous mm. Note that this was made significantly harder to hit by a391263cd84e ("ARM: 8203/1: mm: try to re-use old ASID assignments following a rollover") but is still theoretically possible. This patch fixes the problem by removing the code from flush_context that forces the allocated ASID to zero for the local CPU. Many thanks to the Broadcom guys for tracking this one down. Fixes: e1a5848e3398 ("ARM: 7924/1: mm: don't bother with reserved ttbr0 when running with LPAE") Cc: # v3.14+ Reported-by: Raymond Ngun Tested-by: Raymond Ngun Reviewed-by: Gregory Fong Signed-off-by: Will Deacon Signed-off-by: Russell King

ARM: 8203/1: mm: try to re-use old ASID assignments following a rollover

2014-11-21T15:24:46+00:00

Rather than unconditionally allocating a fresh ASID to an mm from an older generation, attempt to re-use the old assignment where possible. This can bring performance benefits on systems where the ASID is used to tag things other than the TLB (e.g. branch prediction resources). Acked-by: Catalin Marinas Signed-off-by: Will Deacon Signed-off-by: Russell King

ARM: 7926/1: mm: flesh out and fix the comments in the ASID allocator

2013-12-29T12:46:49+00:00

The ASID allocator has to deal with some pretty horrible behaviours by the CPU, so expand on some of the comments in there so I remember why we can never allocate ASID zero to a userspace task. Reviewed-by: Catalin Marinas Signed-off-by: Will Deacon Signed-off-by: Russell King

ARM: 7925/1: mm: keep track of last ASID allocation to improve bitmap searching

2013-12-29T12:46:48+00:00

Since we only clear entries in the ASID bitmap on a rollover event, the bitmap tends to consist of a block of consecutive set bits followed by a block of consecutive clear bits. The exception to this rule is for ASIDs which have been carried over from a previous generation, but these are bound by the number of CPUs. This patch optimises our bitmap searching strategy, so that we search from the last successful allocation, rather than search from index 1 each time we allocate a new ASID. Reviewed-by: Catalin Marinas Signed-off-by: Will Deacon Signed-off-by: Russell King

ARM: 7924/1: mm: don't bother with reserved ttbr0 when running with LPAE

2013-12-29T12:46:47+00:00

With the new ASID allocation algorithm, active ASIDs at the time of a rollover event will be marked as reserved, so active mm_structs can continue to operate with the same ASID as before. This in turn means that we don't need to worry about allocating a new ASID to an mm that is currently active (installed in TTBR0). Since updating the pgd and ASID is atomic on LPAE systems (by virtue of the two being fields in the same hardware register), we can dispose of the reserved TTBR0 and rely on whatever tables we currently have live. Reviewed-by: Catalin Marinas Signed-off-by: Will Deacon Signed-off-by: Russell King

ARM: tlb: don't perform inner-shareable invalidation for local TLB ops

2013-08-12T11:25:44+00:00

Inner-shareable TLB invalidation is typically more expensive than local (non-shareable) invalidation, so performing the broadcasting for local_flush_tlb_* operations is a waste of cycles and needlessly clobbers entries in the TLBs of other CPUs. This patch introduces __flush_tlb_* versions for many of the TLB invalidation functions, which only respect inner-shareable variants of the invalidation instructions when presented with the TLB_V7_UIS_FULL flag. The local version is also inlined to prevent SMP_ON_UP kernels from missing flushes, where the __flush variant would be called with the UP flags. This gains us around 0.5% in hackbench scores for a dual-core A15, but I would expect this to improve as more cores (and clusters) are added to the equation. Reviewed-by: Catalin Marinas Reported-by: Albin Tonnerre Signed-off-by: Will Deacon