kernel/linux.git/mm/vmalloc.c, branch v6.1.186

mm/vmalloc: acquire init_mm lock on huge vmap to avoid ptdump UAF

2026-08-23T12:18:34+00:00

[ Upstream commit 26444eb71465c9934d9d418ef69c43f61185329b ] Patch series "mm: fix UAF caused by race between ptdump and vmap pgtable freeing", v6. Kernel page table walkers fall into two broad categories - those ranges where no exclusion is required via walk_kernel_page_table_range_lockless() and those where exclusion is required via walk_kernel_page_table_range() or walk_page_range_debug(). The former category is used only by arm64 arch code operating on ranges it both wholly owns and does not concurrently write. The latter category consists of kernel page table walkers operating on ranges that are wholly owned (but which need exclusion against concurrent writers). The lock used for exclusion is the mmap lock, and for kernel ranges this is the mmap lock on init_mm. ptdump is a special case being both the only user of walk_page_range_debug(), and the only case in which it walks ranges it does not own. This presents a problem, as page tables may be freed under ptdump. And indeed there is a use-after-free bug in the kernel as a result, which this series addresses. vmap promotes page tables to huge leaf entries where possible, freeing the lower page table when it does. It does this with no meaningful locks held against concurrent ptdump walks. As a result, use-after-free can currently occur. This series addresses the issue by having the vmap huge promotion logic acquire the mmap read lock while both setting the huge page table entry and freeing the prior leaf page table. The ptdump code already acquires the mmap write lock, so by doing so we ensure that the ptdump walker only ever observes either the huge page table entry or the existing page table entry, and nothing is freed underneath it. A mitigation for this issue was already applied for arm64 in commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), which this series has to deal with carefully. This mitigation resolves the issue by acquiring the mmap read lock on init_mm on vmap page table free if a ptdump is in progress. However the fix in this series would cause a deadlock if we were to simply apply it for arm64 without also reverting the change. This is because vmap may acquire the read lock before ptdump attempts to acquire the write lock, which then gets queued, and rwsem starvation rules mean that the (unacknowledged) nested mmap read lock in the arm64 code would also block, meaning the original read lock is never released and thus deadlock. This series works around this by #ifndef CONFIG_ARM64'ing the mmap read lock in vmap logic, then partially reverting commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), keeping the enablement of huge vmap support, and removing the ifdeffery with the partial revert patch. There are related issues that are also addressed in this series: * x86 page attribute logic, specifically Change Page Attributes (CPA), implements a feature whereby huge ranges can be collapsed into huge leaf entries. This can similarly cause a UAF when done in parallel with a ptdump walk, so similarly acquire the init_mm mmap lock to avoid this. * The CPA logic allows concurrent page table manipulation and CPA collapse, meaning the former risks accessing a page table the latter frees. Fix this by acquiring mmap write lock on init_mm across the whole CPA collapse operation and read lock on the page table manipulation. * x86 and arm64 permit walks of non-kernel mm's (both allowing efi mm walks, and in x86's case arbitrary mm's), so we ensure kernel mappings remain stable by locking the init_mm as well as the mm being walked. The ordering of patches is established for both strict dependencies (the arm64 partial revert in particular has to be done after the vmap changes) and logical ones (the non-kernel mm fix only makes sense once the vmap/CPA fixes are in place). This patch (of 3): Currently there is a nasty race between ptdump and vmap when attempting to map a huge P4D, PUD or PMD entry: * ptdump walks kernel page table ranges it doesn't own. * When vmap maps ranges it tries to promotes existing ones to huge page tables in vmap_try_huge_[p4d,pud,pmd]() at P4D, PUD and PMD level, freeing the lower page table in [p4d,pud,pmd]_free_[pud,pmd,pte]_page() when it succeeds. Both of these things can happen at the same time and as a result ptdump can access a freed page table, resulting in a use-after-free and memory corruption. This is possible because while ptdump_walk_pgd() holds both the mem hotplug lock and the mmap write lock before invoking walk_page_range_debug(), vmap takes no relevant locks at all. Fix this by holding the mmap read lock in vmap_try_huge_*() when freeing page tables. The read lock is sufficient: ptdump is the only walker that must be excluded and it holds the mmap write lock. Other holders of the read lock may run concurrently, but each exclusively owns the range it operates on and cannot reach the page tables freed here. We also hold the lock while assigning the huge page table entry, which means page table walkers observe only the huge or non-huge page table entry. We use a trylock to prevent ptdump from blocking vmap making forward progress. This is fine because it's an optimisation in any case, and thus the vmap can safely proceed regardless. All other kernel page table walkers that touch vmalloc ranges either exclusively own the memory walked or acquire the mmap lock, so this correctly excludes those walkers. One wrinkle here is commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), which addresses the issue for arm64 only by explicitly acquiring the mmap read lock on kernel page table freeing should a concurrent ptdump be in progress. This is problematic as vmap may acquire the mmap read lock prior to ptdump attempting to acquire an mmap write lock, leading to a deadlock when the mmap read lock is slept upon on page table freeing due to rwsem anti-starvation. We work around this by predicating the mmap lock being taken on !CONFIG_ARM64 for the time being. With this patch applied, a follow up will partially revert commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump") and at that stage remove the arm64 ifdeffery. We also update walk_page_range_debug() to assert the mmap write lock unconditionally and update the comment here to reflect this change. The issue has existed as long as ptdump was available and vmap freed page tables when promoting to a huge leaf entry, that is, since commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table") for huge ioremap, and commit 121e6f3258fe ("mm/vmalloc: hugepage vmalloc mappings") for huge vmalloc. Since the former is the earlier of the two we choose that for our Fixes tag. We also define a guard class for mmap_read_trylock() so we can use cleanup.h to make the scope handling cleaner in the implementation. This patch is based on work by David Carlier (linked), with gratitude! Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-0-8cc77dcc0018@kernel.org Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-1-8cc77dcc0018@kernel.org Fixes: b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table") Signed-off-by: Lorenzo Stoakes (ARM) Reported-by: syzbot+fd95a72470f5a44e464c@syzkaller.appspotmail.com Closes: https://lore.kernel.org/all/6a287988.39669fcc.33b062.00a0.GAE@google.com/T/ Link: https://lore.kernel.org/linux-mm/20260706203128.162335-1-devnexen@gmail.com/ Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Dev Jain Acked-by: David Hildenbrand (Arm) Reviewed-by: Kiryl Shutsemau Cc: Cc: Andy Lutomirski Cc: "Borah, Chaitanya Kumar" Cc: "Borislav Petkov (AMD)" Cc: Catalin Marinas Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Liam R. Howlett Cc: Michal Hocko Cc: Peter Zijlstra Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Toshi Kani Cc: "Uladzislau Rezki (Sony)" Cc: Vlastimil Babka Cc: Will Deacon Signed-off-by: Andrew Morton Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman

kasan: refactor pcpu kasan vmalloc unpoison

2026-01-11T14:19:11+00:00

commit 6f13db031e27e88213381039032a9cc061578ea6 upstream. A KASAN tag mismatch, possibly causing a kernel panic, can be observed on systems with a tag-based KASAN enabled and with multiple NUMA nodes. It was reported on arm64 and reproduced on x86. It can be explained in the following points: 1. There can be more than one virtual memory chunk. 2. Chunk's base address has a tag. 3. The base address points at the first chunk and thus inherits the tag of the first chunk. 4. The subsequent chunks will be accessed with the tag from the first chunk. 5. Thus, the subsequent chunks need to have their tag set to match that of the first chunk. Refactor code by reusing __kasan_unpoison_vmalloc in a new helper in preparation for the actual fix. Link: https://lkml.kernel.org/r/eb61d93b907e262eefcaa130261a08bcb6c5ce51.1764874575.git.m.wieczorretman@pm.me Fixes: 1d96320f8d53 ("kasan, vmalloc: add vmalloc tagging for SW_TAGS") Signed-off-by: Maciej Wieczor-Retman Reviewed-by: Andrey Konovalov Cc: Alexander Potapenko Cc: Andrey Ryabinin Cc: Danilo Krummrich Cc: Dmitriy Vyukov Cc: Jiayuan Chen Cc: Kees Cook Cc: Marco Elver Cc: "Uladzislau Rezki (Sony)" Cc: Vincenzo Frascino Cc: [6.1+] Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/vmalloc: leave lazy MMU mode on PTE mapping error

2025-07-24T06:51:55+00:00

commit fea18c686320a53fce7ad62a87a3e1d10ad02f31 upstream. vmap_pages_pte_range() enters the lazy MMU mode, but fails to leave it in case an error is encountered. Link: https://lkml.kernel.org/r/20250623075721.2817094-1-agordeev@linux.ibm.com Fixes: 2ba3e6947aed ("mm/vmalloc: track which page-table levels were modified") Signed-off-by: Alexander Gordeev Reported-by: kernel test robot Reported-by: Dan Carpenter Closes: https://lore.kernel.org/r/202506132017.T1l1l6ME-lkp@intel.com/ Reviewed-by: Ryan Roberts Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm: don't skip arch_sync_kernel_mappings() in error paths

2025-03-13T11:53:15+00:00

commit 3685024edd270f7c791f993157d65d3c928f3d6e upstream. Fix callers that previously skipped calling arch_sync_kernel_mappings() if an error occurred during a pgtable update. The call is still required to sync any pgtable updates that may have occurred prior to hitting the error condition. These are theoretical bugs discovered during code review. Link: https://lkml.kernel.org/r/20250226121610.2401743-1-ryan.roberts@arm.com Fixes: 2ba3e6947aed ("mm/vmalloc: track which page-table levels were modified") Fixes: 0c95cba49255 ("mm: apply_to_pte_range warn and fail if a large pte is encountered") Signed-off-by: Ryan Roberts Reviewed-by: Anshuman Khandual Reviewed-by: Catalin Marinas Cc: Christop Hellwig Cc: "Uladzislau Rezki (Sony)" Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

vmalloc: fix accounting with i915

2025-01-02T09:30:53+00:00

[ Upstream commit a2e740e216f5bf49ccb83b6d490c72a340558a43 ] If the caller of vmap() specifies VM_MAP_PUT_PAGES (currently only the i915 driver), we will decrement nr_vmalloc_pages and MEMCG_VMALLOC in vfree(). These counters are incremented by vmalloc() but not by vmap() so this will cause an underflow. Check the VM_MAP_PUT_PAGES flag before decrementing either counter. Link: https://lkml.kernel.org/r/20241211202538.168311-1-willy@infradead.org Fixes: b944afc9d64d ("mm: add a VM_MAP_PUT_PAGES flag for vmap") Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Johannes Weiner Reviewed-by: Shakeel Butt Reviewed-by: Balbir Singh Acked-by: Michal Hocko Cc: Christoph Hellwig Cc: Muchun Song Cc: Roman Gushchin Cc: "Uladzislau Rezki (Sony)" Cc: Signed-off-by: Andrew Morton Signed-off-by: Matthew Wilcox (Oracle) Signed-off-by: Sasha Levin

mm/vmalloc: fix page mapping if vm_area_alloc_pages() with high order fallback to order 0

2024-08-29T15:30:52+00:00

commit 61ebe5a747da649057c37be1c37eb934b4af79ca upstream. The __vmap_pages_range_noflush() assumes its argument pages** contains pages with the same page shift. However, since commit e9c3cda4d86e ("mm, vmalloc: fix high order __GFP_NOFAIL allocations"), if gfp_flags includes __GFP_NOFAIL with high order in vm_area_alloc_pages() and page allocation failed for high order, the pages** may contain two different page shifts (high order and order-0). This could lead __vmap_pages_range_noflush() to perform incorrect mappings, potentially resulting in memory corruption. Users might encounter this as follows (vmap_allow_huge = true, 2M is for PMD_SIZE): kvmalloc(2M, __GFP_NOFAIL|GFP_X) __vmalloc_node_range_noprof(vm_flags=VM_ALLOW_HUGE_VMAP) vm_area_alloc_pages(order=9) ---> order-9 allocation failed and fallback to order-0 vmap_pages_range() vmap_pages_range_noflush() __vmap_pages_range_noflush(page_shift = 21) ----> wrong mapping happens We can remove the fallback code because if a high-order allocation fails, __vmalloc_node_range_noprof() will retry with order-0. Therefore, it is unnecessary to fallback to order-0 here. Therefore, fix this by removing the fallback code. Link: https://lkml.kernel.org/r/20240808122019.3361-1-hailong.liu@oppo.com Fixes: e9c3cda4d86e ("mm, vmalloc: fix high order __GFP_NOFAIL allocations") Signed-off-by: Hailong Liu Reported-by: Tangquan Zheng Reviewed-by: Baoquan He Reviewed-by: Uladzislau Rezki (Sony) Acked-by: Barry Song Acked-by: Michal Hocko Cc: Matthew Wilcox Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/vmalloc: fix vmalloc which may return null if called with __GFP_NOFAIL

2024-06-21T12:35:41+00:00

[ Upstream commit 8e0545c83d672750632f46e3f9ad95c48c91a0fc ] commit a421ef303008 ("mm: allow !GFP_KERNEL allocations for kvmalloc") includes support for __GFP_NOFAIL, but it presents a conflict with commit dd544141b9eb ("vmalloc: back off when the current task is OOM-killed"). A possible scenario is as follows: process-a __vmalloc_node_range(GFP_KERNEL | __GFP_NOFAIL) __vmalloc_area_node() vm_area_alloc_pages() --> oom-killer send SIGKILL to process-a if (fatal_signal_pending(current)) break; --> return NULL; To fix this, do not check fatal_signal_pending() in vm_area_alloc_pages() if __GFP_NOFAIL set. This issue occurred during OPLUS KASAN TEST. Below is part of the log -> oom-killer sends signal to process [65731.222840] [ T1308] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/apps/uid_10198,task=gs.intelligence,pid=32454,uid=10198 [65731.259685] [T32454] Call trace: [65731.259698] [T32454] dump_backtrace+0xf4/0x118 [65731.259734] [T32454] show_stack+0x18/0x24 [65731.259756] [T32454] dump_stack_lvl+0x60/0x7c [65731.259781] [T32454] dump_stack+0x18/0x38 [65731.259800] [T32454] mrdump_common_die+0x250/0x39c [mrdump] [65731.259936] [T32454] ipanic_die+0x20/0x34 [mrdump] [65731.260019] [T32454] atomic_notifier_call_chain+0xb4/0xfc [65731.260047] [T32454] notify_die+0x114/0x198 [65731.260073] [T32454] die+0xf4/0x5b4 [65731.260098] [T32454] die_kernel_fault+0x80/0x98 [65731.260124] [T32454] __do_kernel_fault+0x160/0x2a8 [65731.260146] [T32454] do_bad_area+0x68/0x148 [65731.260174] [T32454] do_mem_abort+0x151c/0x1b34 [65731.260204] [T32454] el1_abort+0x3c/0x5c [65731.260227] [T32454] el1h_64_sync_handler+0x54/0x90 [65731.260248] [T32454] el1h_64_sync+0x68/0x6c [65731.260269] [T32454] z_erofs_decompress_queue+0x7f0/0x2258 --> be->decompressed_pages = kvcalloc(be->nr_pages, sizeof(struct page *), GFP_KERNEL | __GFP_NOFAIL); kernel panic by NULL pointer dereference. erofs assume kvmalloc with __GFP_NOFAIL never return NULL. [65731.260293] [T32454] z_erofs_runqueue+0xf30/0x104c [65731.260314] [T32454] z_erofs_readahead+0x4f0/0x968 [65731.260339] [T32454] read_pages+0x170/0xadc [65731.260364] [T32454] page_cache_ra_unbounded+0x874/0xf30 [65731.260388] [T32454] page_cache_ra_order+0x24c/0x714 [65731.260411] [T32454] filemap_fault+0xbf0/0x1a74 [65731.260437] [T32454] __do_fault+0xd0/0x33c [65731.260462] [T32454] handle_mm_fault+0xf74/0x3fe0 [65731.260486] [T32454] do_mem_abort+0x54c/0x1b34 [65731.260509] [T32454] el0_da+0x44/0x94 [65731.260531] [T32454] el0t_64_sync_handler+0x98/0xb4 [65731.260553] [T32454] el0t_64_sync+0x198/0x19c Link: https://lkml.kernel.org/r/20240510100131.1865-1-hailong.liu@oppo.com Fixes: 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL") Signed-off-by: Hailong.Liu Acked-by: Michal Hocko Suggested-by: Barry Song <21cnbao@gmail.com> Reported-by: Oven Reviewed-by: Barry Song Reviewed-by: Uladzislau Rezki (Sony) Cc: Chao Yu Cc: Christoph Hellwig Cc: Gao Xiang Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Signed-off-by: Andrew Morton Signed-off-by: Sasha Levin

mm, vmalloc: fix high order __GFP_NOFAIL allocations

2024-06-21T12:35:41+00:00

[ Upstream commit e9c3cda4d86e56bf7fe403729f38c4f0f65d3860 ] Gao Xiang has reported that the page allocator complains about high order __GFP_NOFAIL request coming from the vmalloc core: __alloc_pages+0x1cb/0x5b0 mm/page_alloc.c:5549 alloc_pages+0x1aa/0x270 mm/mempolicy.c:2286 vm_area_alloc_pages mm/vmalloc.c:2989 [inline] __vmalloc_area_node mm/vmalloc.c:3057 [inline] __vmalloc_node_range+0x978/0x13c0 mm/vmalloc.c:3227 kvmalloc_node+0x156/0x1a0 mm/util.c:606 kvmalloc include/linux/slab.h:737 [inline] kvmalloc_array include/linux/slab.h:755 [inline] kvcalloc include/linux/slab.h:760 [inline] it seems that I have completely missed high order allocation backing vmalloc areas case when implementing __GFP_NOFAIL support. This means that [k]vmalloc at al. can allocate higher order allocations with __GFP_NOFAIL which can trigger OOM killer for non-costly orders easily or cause a lot of reclaim/compaction activity if those requests cannot be satisfied. Fix the issue by falling back to zero order allocations for __GFP_NOFAIL requests if the high order request fails. Link: https://lkml.kernel.org/r/ZAXynvdNqcI0f6Us@dhcp22.suse.cz Fixes: 9376130c390a ("mm/vmalloc: add support for __GFP_NOFAIL") Reported-by: Gao Xiang Link: https://lkml.kernel.org/r/20230305053035.1911-1-hsiangkao@linux.alibaba.com Signed-off-by: Michal Hocko Reviewed-by: Uladzislau Rezki (Sony) Acked-by: Vlastimil Babka Cc: Baoquan He Cc: Christoph Hellwig Cc: Mel Gorman Signed-off-by: Andrew Morton Stable-dep-of: 8e0545c83d67 ("mm/vmalloc: fix vmalloc which may return null if called with __GFP_NOFAIL") Signed-off-by: Sasha Levin

mm/vmalloc: add a safer version of find_vm_area() for debug

2023-09-13T07:43:00+00:00

commit 0818e739b5c061b0251c30152380600fb9b84c0c upstream. It is unsafe to dump vmalloc area information when trying to do so from some contexts. Add a safer trylock version of the same function to do a best-effort VMA finding and use it from vmalloc_dump_obj(). [applied test robot feedback on unused function fix.] [applied Uladzislau feedback on locking.] Link: https://lkml.kernel.org/r/20230904180806.1002832-1-joel@joelfernandes.org Fixes: 98f180837a89 ("mm: Make mem_dump_obj() handle vmalloc() memory") Signed-off-by: Joel Fernandes (Google) Reviewed-by: Uladzislau Rezki (Sony) Reported-by: Zhen Lei Cc: Paul E. McKenney Cc: Zqiang Cc: Cc: Matthew Wilcox (Oracle) Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm: add a call to flush_cache_vmap() in vmap_pfn()

2023-08-30T14:11:06+00:00

commit a50420c79731fc5cf27ad43719c1091e842a2606 upstream. flush_cache_vmap() must be called after new vmalloc mappings are installed in the page table in order to allow architectures to make sure the new mapping is visible. It could lead to a panic since on some architectures (like powerpc), the page table walker could see the wrong pte value and trigger a spurious page fault that can not be resolved (see commit f1cb8f9beba8 ("powerpc/64s/radix: avoid ptesync after set_pte and ptep_set_access_flags")). But actually the patch is aiming at riscv: the riscv specification allows the caching of invalid entries in the TLB, and since we recently removed the vmalloc page fault handling, we now need to emit a tlb shootdown whenever a new vmalloc mapping is emitted (https://lore.kernel.org/linux-riscv/20230725132246.817726-1-alexghiti@rivosinc.com/). That's a temporary solution, there are ways to avoid that :) Link: https://lkml.kernel.org/r/20230809164633.1556126-1-alexghiti@rivosinc.com Fixes: 3e9a9e256b1e ("mm: add a vmap_pfn function") Reported-by: Dylan Jhong Closes: https://lore.kernel.org/linux-riscv/ZMytNY2J8iyjbPPy@atctrx.andestech.com/ Signed-off-by: Alexandre Ghiti Reviewed-by: Christoph Hellwig Reviewed-by: Palmer Dabbelt Acked-by: Palmer Dabbelt Reviewed-by: Dylan Jhong Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman