kernel/linux.git/mm/sparse-vmemmap.c, branch v7.2-rc1

mm/sparse-vmemmap: provide generic vmemmap_set_pmd() and vmemmap_check_pmd()

2026-06-09T01:21:30+00:00

Patch series "mm/sparse-vmemmap: Provide generic vmemmap_set_pmd() and vmemmap_check_pmd()", v3. The weak vmemmap_set_pmd() and vmemmap_check_pmd() hooks are currently no-ops in the generic code, which leaves architectures that need PMD-level handling to open-code the same logic locally. This series provides generic implementations for both helpers in mm/sparse-vmemmap.c. vmemmap_set_pmd() installs a huge PMD with PAGE_KERNEL protection, and vmemmap_check_pmd() verifies a present leaf PMD before reusing the existing vmemmap_verify() helper. With those generic helpers in place, patches 2-5 remove the now redundant arch-specific implementations from arm64, riscv, loongarch, and sparc. This patch (of 5): The two weak functions are currently no-ops on every architecture, forcing each platform that needs them to duplicate the same handful of lines. Provide a generic implementation: - vmemmap_set_pmd() simply sets a huge PMD with PAGE_KERNEL protection. - vmemmap_check_pmd() verifies that the PMD is present and leaf, then calls the existing vmemmap_verify() helper. Architectures that need special handling can continue to override the weak symbols; everyone else gets the standard version for free. Link: https://lore.kernel.org/20260601084845.3792171-1-songmuchun@bytedance.com Link: https://lore.kernel.org/20260601084845.3792171-2-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: David Hildenbrand (Arm) Acked-by: Oscar Salvador (SUSE) Cc: Albert Ou Cc: Alexandre Ghiti Cc: Andreas Larsson Cc: Catalin Marinas Cc: David S. Miller Cc: Huacai Chen Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Palmer Dabbelt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: WANG Xuerui Cc: Will Deacon Signed-off-by: Andrew Morton

mm/sparse-vmemmap: fix DAX vmemmap accounting with optimization

2026-05-29T04:04:54+00:00

When vmemmap optimization is enabled for DAX, the nr_memmap_pages counter in /proc/vmstat is incorrect. The current code always accounts for the full, non-optimized vmemmap size, but vmemmap optimization reduces the actual number of vmemmap pages by reusing tail pages. This causes the system to overcount vmemmap usage, leading to inaccurate page statistics in /proc/vmstat. Fix this by introducing section_nr_vmemmap_pages(), which returns the exact vmemmap page count for a given pfn range based on whether optimization is in effect. Link: https://lore.kernel.org/20260428081855.1249045-5-songmuchun@bytedance.com Fixes: 15995a352474 ("mm: report per-page metadata information") Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Oscar Salvador Acked-by: David Hildenbrand (Arm) Acked-by: Liam R. Howlett Cc: "Aneesh Kumar K.V" Cc: Joao Martins Cc: Lorenzo Stoakes Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: Michal Hocko Cc: Nicholas Piggin Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Signed-off-by: Andrew Morton

mm/sparse-vmemmap: pass @pgmap argument to memory deactivation paths

2026-05-29T04:04:54+00:00

Currently, the memory hot-remove call chain -- arch_remove_memory(), __remove_pages(), sparse_remove_section() and section_deactivate() -- does not carry the struct dev_pagemap pointer. This prevents the lower levels from knowing whether the section was originally populated with vmemmap optimizations (e.g., DAX with vmemmap optimization enabled). Without this information, we cannot call vmemmap_can_optimize() to determine if the vmemmap pages were optimized. As a result, the vmemmap page accounting during teardown will mistakenly assume a non-optimized allocation, leading to incorrect memmap statistics. To lay the groundwork for fixing the vmemmap page accounting, we need to pass the @pgmap pointer down to the deactivation location. Plumb the @pgmap argument through the APIs of arch_remove_memory(), __remove_pages() and sparse_remove_section(), mirroring the corresponding *_activate() paths. Link: https://lore.kernel.org/20260428081855.1249045-4-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Reviewed-by: Oscar Salvador Acked-by: David Hildenbrand (Arm) Acked-by: Liam R. Howlett Cc: "Aneesh Kumar K.V" Cc: Joao Martins Cc: Lorenzo Stoakes Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: Michal Hocko Cc: Nicholas Piggin Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton

mm/sparse-vmemmap: fix vmemmap accounting underflow

2026-05-29T04:04:53+00:00

Patch series "mm: Fix vmemmap optimization accounting and initialization", v8. The series fixes several bugs in vmemmap optimization, mainly around incorrect page accounting and memmap initialization in DAX and memory hotplug paths. It also fixes pageblock migratetype initialization and struct page initialization for ZONE_DEVICE compound pages. Patches 1-4 fix vmemmap accounting issues. Patch 1 fixes an accounting underflow in the section activation failure path by moving vmemmap page accounting into the lower-level allocation and freeing helpers. Patch 2 fixes incorrect altmap passing in the memory hotplug error path. Patch 3 passes pgmap through memory deactivation paths so the teardown side can determine whether vmemmap optimization was in effect. Patch 4 uses that information to account the optimized DAX vmemmap size correctly. Patches 5-6 fix initialization issues in mm/mm_init. One makes sure all pageblocks in ZONE_DEVICE compound pages get their migratetype initialized. The other fixes a case where DAX memory hotplug reuses an unoptimized early-section memmap while compound_nr_pages() still assumes vmemmap optimization, leaving tail struct pages uninitialized. This patch (of 6): In section_activate(), if populate_section_memmap() fails, the error handling path calls section_deactivate() to roll back the state. This causes a vmemmap accounting imbalance. Since commit c3576889d87b ("mm: fix accounting of memmap pages"), memmap pages are accounted for only after populate_section_memmap() succeeds. However, the failure path unconditionally calls section_deactivate(), which decreases the vmemmap count. Consequently, a failure in populate_section_memmap() leads to an accounting underflow, incorrectly reducing the system's tracked vmemmap usage. Fix this more thoroughly by moving all accounting calls into the lower level functions that actually perform the vmemmap allocation and freeing: - populate_section_memmap() accounts for newly allocated vmemmap pages - depopulate_section_memmap() unaccounts when vmemmap is freed This ensures proper accounting in all code paths, including error handling and early section cases. Link: https://lore.kernel.org/20260428081855.1249045-1-songmuchun@bytedance.com Link: https://lore.kernel.org/20260428081855.1249045-2-songmuchun@bytedance.com Fixes: c3576889d87b ("mm: fix accounting of memmap pages") Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: Oscar Salvador Acked-by: David Hildenbrand (Arm) Acked-by: Liam R. Howlett Cc: "Aneesh Kumar K.V" Cc: Joao Martins Cc: Lorenzo Stoakes Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: Michal Hocko Cc: Nicholas Piggin Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Signed-off-by: Andrew Morton

mm/sparse: remove sparse buffer pre-allocation mechanism

2026-05-29T04:04:49+00:00

Commit 9bdac9142407 ("sparsemem: Put mem map for one node together.") introduced a mechanism to pre-allocate a large memory block to hold all memmaps for a NUMA node upfront. However, the original commit message did not clearly state the actual benefits or the necessity of explicitly pre-allocating a single chunk for all memmap areas of a given node. One of the concerns about removing this pre-allocation is that the subsequent per-section memmap allocations could become scattered around, and might turn too many memory blocks/sections into an "un-offlinable" state. However, tests show that even without the explicit node-wide pre-allocation, memblock still allocates memory closely and back-to-back. When tracing vmemmap_set_pmd allocations, the physical chunks allocated by memblock are strictly adjacent to each other in a single contiguous physical range (mapped top-down). Because they are packed tightly together naturally, they will at most consume or pollute the exact same number of memory blocks as the explicit pre-allocation did. Another concern is the boot performance impact of calling memmap_alloc() multiple times compared to one large node-wide allocation. Tests on a 256GB VM showed that memmap allocation time increased from 199,555 ns to 741,292 ns. Even though it is 3.7x slower, on a 1TB machine, the entire memory allocation time would only take a few milliseconds. This boot performance difference is completely negligible. Since no negative impact on memory offlining behavior or noticeable boot performance regression was found, this patch proposes removing the explicit node-wide memmap pre-allocation mechanism to reduce the maintenance burden. Link: https://lore.kernel.org/20260410092419.2446420-1-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton

mm: convert vmemmap_p?d_populate() to static functions

2026-05-29T04:04:40+00:00

Since the vmemmap_p?d_populate functions are unused outside the mm subsystem, we can remove their external declarations and convert them to static functions. Link: https://lore.kernel.org/20260423101441.7089-1-kaitao.cheng@linux.dev Signed-off-by: Chengkaitao Acked-by: David Hildenbrand (arm) Acked-by: Mike Rapoport (Microsoft) Acked-by: Oscar Salvador Cc: David Hildenbrand Cc: Liam Howlett Cc: Lorenzo Stoakes (Oracle) Cc: Michal Hocko Cc: Suren Baghdasaryan Signed-off-by: Andrew Morton

mm: mark early-init static variables with __meminitdata

2026-04-05T20:53:34+00:00

Static variables defined inside __meminit functions should also be marked with __meminitdata, so that their storage is placed in the .init.data section and reclaimed with free_initmem(), thereby reducing permanent .bss memory usage when CONFIG_MEMORY_HOTPLUG is disabled. Link: https://lkml.kernel.org/r/20260321120847.8159-1-pilgrimtao@gmail.com Signed-off-by: Kaitao Cheng Reviewed-by: Andrew Morton Cc: David Hildenbrand Cc: Liam Howlett Cc: Lorenzo Stoakes (Oracle) Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Signed-off-by: Andrew Morton

mm/sparse: move memory hotplug bits to sparse-vmemmap.c

2026-04-05T20:53:33+00:00

Let's move all memory hoptplug related code to sparse-vmemmap.c. We only have to expose sparse_index_init(). While at it, drop the definition of sparse_index_init() for !CONFIG_SPARSEMEM, which is unused, and place the declaration in internal.h. Link: https://lkml.kernel.org/r/20260320-sparsemem_cleanups-v2-15-096addc8800d@kernel.org Signed-off-by: David Hildenbrand (Arm) Reviewed-by: Lorenzo Stoakes (Oracle) Reviewed-by: Mike Rapoport (Microsoft) Cc: Axel Rasmussen Cc: Liam Howlett Cc: Michal Hocko Cc: Oscar Salvador Cc: Sidhartha Kumar Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm/hugetlb: remove fake head pages

2026-04-05T20:53:09+00:00

HugeTLB Vmemmap Optimization (HVO) reduces memory usage by freeing most vmemmap pages for huge pages and remapping the freed range to a single page containing the struct page metadata. With the new mask-based compound_info encoding (for power-of-2 struct page sizes), all tail pages of the same order are now identical regardless of which compound page they belong to. This means the tail pages can be truly shared without fake heads. Allocate a single page of initialized tail struct pages per zone per order in the vmemmap_tails[] array in struct zone. All huge pages of that order in the zone share this tail page, mapped read-only into their vmemmap. The head page remains unique per huge page. Redefine MAX_FOLIO_ORDER using ilog2(). The define has to produce a compile-constant as it is used to specify vmemmap_tail array size. For some reason, compiler is not able to solve get_order() at compile-time, but ilog2() works. Avoid PUD_ORDER to define MAX_FOLIO_ORDER as it adds dependency to which generates hard-to-break include loop. This eliminates fake heads while maintaining the same memory savings, and simplifies compound_head() by removing fake head detection. Link: https://lkml.kernel.org/r/20260227194302.274384-13-kas@kernel.org Signed-off-by: Kiryl Shutsemau Reviewed-by: Vlastimil Babka (SUSE) Acked-by: David Hildenbrand (Arm) Cc: Albert Ou Cc: Alexandre Ghiti Cc: Baoquan He Cc: Christoph Lameter Cc: David Rientjes Cc: Frank van der Linden Cc: Harry Yoo Cc: Huacai Chen Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Oscar Salvador Cc: Palmer Dabbelt Cc: Paul Walmsley Cc: Roman Gushchin Cc: Usama Arif Cc: WANG Xuerui Cc: Zi Yan Signed-off-by: Andrew Morton

mm/hugetlb: defer vmemmap population for bootmem hugepages

2026-04-05T20:53:08+00:00

Currently, the vmemmap for bootmem-allocated gigantic pages is populated early in hugetlb_vmemmap_init_early(). However, the zone information is only available after zones are initialized. If it is later discovered that a page spans multiple zones, the HVO mapping must be undone and replaced with a normal mapping using vmemmap_undo_hvo(). Defer the actual vmemmap population to hugetlb_vmemmap_init_late(). At this stage, zones are already initialized, so it can be checked if the page is valid for HVO before deciding how to populate the vmemmap. This allows us to remove vmemmap_undo_hvo() and the complex logic required to rollback HVO mappings. In hugetlb_vmemmap_init_late(), if HVO population fails or if the zones are invalid, fall back to a normal vmemmap population. Postponing population until hugetlb_vmemmap_init_late() also makes zone information available from within vmemmap_populate_hvo(). Link: https://lkml.kernel.org/r/20260227194302.274384-10-kas@kernel.org Signed-off-by: Kiryl Shutsemau (Meta) Acked-by: David Hildenbrand (Arm) Cc: Albert Ou Cc: Alexandre Ghiti Cc: Baoquan He Cc: Christoph Lameter Cc: David Rientjes Cc: Frank van der Linden Cc: Harry Yoo Cc: Huacai Chen Cc: Johannes Weiner Cc: Jonathan Corbet Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Oscar Salvador Cc: Palmer Dabbelt Cc: Paul Walmsley Cc: Roman Gushchin Cc: Usama Arif Cc: Vlastimil Babka Cc: WANG Xuerui Cc: Zi Yan Signed-off-by: Andrew Morton