kernel/linux.git/include/linux/mmzone.h, branch v7.3-rc1

mm/sparse: keep mem_section_usage_size() internal

2026-08-25T01:43:06+00:00

mem_section_usage_size() is only needed by sparsemem implementation code after commit ae751d567baa ("mm/bootmem_info: stop marking mem_section_usage as MIX_SECTION_INFO"), so keeping the declaration in mmzone.h now exposes the helper to all mmzone.h users for no reason. Move the helper to sparse.h so sparse.c and sparse-vmemmap.c can share it through the internal header. While doing so, calculate the allocation size with struct_size_t(), which ties the expression to the pageblock_flags trailing array instead of open-coding the struct header plus bitmap size. Link: https://lore.kernel.org/20260805022536.1206575-1-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton

mm/vmscan: reduce lru_lock contention via vmstat-derived scan-balance cost

2026-08-25T01:42:56+00:00

The anon/file scan balance in get_scan_count() is driven by two scalars in struct lruvec, anon_cost and file_cost, accumulated by every reclaim producer under lruvec->lru_lock. The acquisition sites for cost work specifically are: - shrink_inactive_list() re-takes lru_lock at function exit purely to call lru_note_cost_unlock_irq() with (nr_pageout, nr_scanned - nr_reclaimed). One acquisition per inactive shrink. - shrink_active_list() does the same with (0, nr_rotated). One acquisition per active shrink. - workingset_refault() takes the lock via folio_lruvec_lock_irq() purely to record the refault cost. One acquisition per refault. - prepare_scan_control() takes lru_lock just to snapshot the two scalars into sc->{anon,file}_cost. - lru_note_cost_unlock_irq() itself walks parent_lruvec and re-acquires lru_lock on each ancestor to propagate the update, adding O(memcg-depth) acquisitions per producer call. This hurts because lru_lock is already a heavy contention point on memory-heavy workloads: every isolate_lru_folios(), move_folios_to_lru() and folio_add_lru() takes it. The cost work itself is trivial (two scalar bumps and one comparison), but it contends with and causes contention for actual LRU manipulation. The parent_lruvec() walk also multiplies cost-update overhead by memcg hierarchy depth. The balance formula for anon and file, respectively, is this: cost = nr_io * SWAP_CLUSTER_MAX + nr_rotated Instead of recording cost and running averaging logic directly when these events occur, snapshot running vmstat counters once per reclaim cycle and derive the balance from event deltas since the last run. Use PGROTATE_* from the preceding patch for the rotation input. WORKINGSET_RESTORE_* and NR_VMSCAN_WRITE provide the remaining event counters. Charge NR_VMSCAN_WRITE through lruvec stats so all inputs can be sampled per lruvec and aggregated through the memcg hierarchy. This is overall cheaper and has fewer lock acquisition sites. Moving accumulation and decay to the reclaim side also improves the cost model across reclaim gaps. With producer-side decay, events that happen while reclaim is idle still age each other before reclaim ever samples the costs. If a workload refaults a large anon set and then a smaller file set before reclaim runs again, the later file activity can age the earlier anon activity out of the cost model. The new scheme observes the whole between-reclaim delta and decays anon and file proportionally, so the scan-balance history better represents what happened since the last reclaim pass. A dedicated per-lruvec spinlock, cost_lock, serialises the delta extraction, the cost->count update and the halving loop against concurrent reclaimers in the same memcg+node. NR_VMSCAN_WRITE is accounted at writeout(), so reclaim_stat.nr_pageout is no longer needed and is removed. memcg-v1's memory.stat anon_cost/file_cost is now sourced from cost[].count instead of the removed lruvec anon_cost/file_cost fields. The reported values only refresh when prepare_scan_control() runs and are bounded at ~lrusize/4 by the halving loop; the scan-balance signal they express is unchanged. Under pure MGLRU the scan-balance signal itself is not consumed (both prepare_scan_control() and get_scan_count() are short-circuited on the MGLRU paths, and MGLRU's own type/tier selection comes from read_ctrl_pos() on lrugen->{avg_refaulted,avg_total,refaulted,evicted}, not from anon_cost/file_cost). NR_VMSCAN_WRITE naturally covers writeout from either reclaim implementation. The preceding patch also bumps PGROTATE_{ANON,FILE} from evict_folios(), so rotation-driven reclaim work is accounted consistently across both implementations. Link: https://lore.kernel.org/20260727162550.2032-4-usama.arif@linux.dev Signed-off-by: Usama Arif Acked-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm/vmscan: add pgrotate_anon and pgrotate_file vmstat counters

2026-08-25T01:42:56+00:00

Reclaim can spend substantial work on an LRU type without immediately reclaiming or demoting a corresponding amount of memory. Record this work in PGROTATE_ANON and PGROTATE_FILE. For classic LRU reclaim: - Inactive-list reclaim adds nr_scanned - nr_reclaimed to the corresponding anon/file counter when isolation succeeds. - Active-list reclaim adds referenced executable file folios that are retained on the active list to PGROTATE_FILE. Active anon reclaim does not contribute this component. For MGLRU, add the number of initially isolated pages that remain unreclaimed after both the initial and retry passes to the counter for the selected anon/file type. These counters are distinct from the existing pgrotated vm event. pgrotated records an actual move to the inactive-list tail, primarily after reclaim-marked writeback completes or failed invalidation leaves a folio for accelerated reclaim. PGROTATE_ANON and PGROTATE_FILE measure reclaim cost and do not imply that a folio moved to an LRU tail. A subsequent patch will consume these counters for anon/file scan balancing. Link: https://lore.kernel.org/20260727162550.2032-3-usama.arif@linux.dev Signed-off-by: Usama Arif Acked-by: Shakeel Butt Acked-by: Johannes Weiner Reviewed-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: David Hildenbrand Cc: David Rientjes Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm: fix ASSERT_EXCLUSIVE_BITS by passing memdesc_flags_t by pointer

2026-07-31T02:40:31+00:00

KCSAN reports a data race between page_to_nid()/folio_pgdat() reading page->flags and folio_trylock()/folio_lock() concurrently doing test_and_set_bit_lock(PG_locked, ...) on the same word, e.g.: BUG: KCSAN: data-race in __lruvec_stat_mod_folio / shmem_get_folio_gfp The race is benign: nid/zone bits are set once at page init and never overlap with PG_locked. However, ASSERT_EXCLUSIVE_BITS() inside memdesc_nid/zonenum() was checking a by-value copy of the flags word, not the live page->flags, so it failed to annotate the real access. Change memdesc_nid(), memdesc_zonenum(), memdesc_section(), and memdesc_is_zone_device() to take a const memdesc_flags_t * and update all callers to pass &page->flags / &folio->flags, so ASSERT_EXCLUSIVE_BITS() operates on the actual shared word. Guard the ASSERT_EXCLUSIVE_BITS() call in memdesc_zonenum() under ZONES_WIDTH != 0 to avoid a zero-mask check on configs where the zone field is absent. memdesc_section() needs no such guard, since SECTIONS_WIDTH is never 0 wherever SECTION_IN_PAGE_FLAGS is defined. Under CONFIG_NUMA=n, memdesc_nid() itself is stubbed to "return 0" instead of reading page->flags, since NODES_MASK is 0 and the check can never fire; page_to_nid()/folio_nid() now just call memdesc_nid() unconditionally and rely on that stub, instead of duplicating the CONFIG_NUMA split at each call site. [zhuhui@kylinos.cn: v11] Link: https://lore.kernel.org/20260708083308.747930-1-hui.zhu@linux.dev [ziy@nvidia.com: build fix] Link: https://lore.kernel.org/DJSGLP9VFLRI.355AAWR2VHIEL@nvidia.com Link: https://lore.kernel.org/20260630070810.470763-1-hui.zhu@linux.dev Signed-off-by: Hui Zhu Co-developed-by: David Hildenbrand (Arm) Signed-off-by: David Hildenbrand (Arm) Acked-by: David Hildenbrand (Arm) Cc: Axel Rasmussen Cc: Barry Song Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm: constify oom_control, scan_control, and alloc_context nodemask

2026-07-29T04:12:04+00:00

The nodemasks in these structures may come from a variety of sources, including tasks and cpusets - and should never be modified by any code when being passed around inside another context. Link: https://lore.kernel.org/20260609002919.3967782-1-gourry@gourry.net Signed-off-by: Gregory Price Reviewed-by: Lorenzo Stoakes Acked-by: David Hildenbrand (Arm) Reviewed-by: Barry Song Acked-by: Vlastimil Babka (SUSE) Tested-by: SeongJae Park Acked-by: SeongJae Park Acked-by: Waiman Long Acked-by: Zi Yan Cc: Axel Rasmussen Cc: Baoquan He Cc: Brendan Jackman Cc: Chris Li Cc: David Rientjes Cc: Johannes Weiner Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Michal Hocko Cc: Michal Koutný Cc: Mike Rapoport Cc: Nhat Pham Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Tejun Heo Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm/sparse-vmemmap: remove sparse_vmemmap_init_nid_late()

2026-07-29T04:12:00+00:00

hugetlb_vmemmap_init_late() no longer has any users, so the remaining late-init path in sparse_vmemmap_init_nid_late() is dead code. Remove sparse_vmemmap_init_nid_late() and its declarations. Link: https://lore.kernel.org/20260612035903.2468601-18-songmuchun@bytedance.com Signed-off-by: Muchun Song Acked-by: Mike Rapoport (Microsoft) Cc: "Aneesh Kumar K.V" Cc: David Hildenbrand Cc: Frank van der Linden Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Oscar Salvador Cc: Oscar Salvador (SUSE) Cc: "Ritesh Harjani (IBM)" Cc: Usama Arif Cc: Vlastimil Babka Signed-off-by: Andrew Morton

mm: replace ASSEMBLY with ASSEMBLER in memory management header files

2026-07-29T04:11:46+00:00

While the GCC and Clang compilers already define __ASSEMBLER__ automatically when compiling assembly code, __ASSEMBLY__ is a macro that only gets defined by the Makefiles in the kernel. This can be very confusing when switching between userspace and kernelspace coding, or when dealing with uapi headers that rather should use __ASSEMBLER__ instead. So let's standardize now on the __ASSEMBLER__ macro that is provided by the compilers. This is a completely mechanical patch (done with a simple "sed -i" statement). Link: https://lore.kernel.org/20260619131830.229804-1-thuth@redhat.com Signed-off-by: Thomas Huth Cc: Arnd Bergmann Cc: Axel Rasmussen Cc: Barry Song Cc: David Hildenbrand Cc: Kairui Song Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm: delete stale comment about cachelines

2026-06-09T01:21:27+00:00

These comments have been wrong since commit a211c6550efc ("mm: page_alloc: defrag_mode kswapd/kcompactd watermarks") added NR_FREE_PAGES_BLOCKS. Since nobody has complained about it in the last year, it seems unlikely these comments were particularly useful anyway, so delete them. Link: https://lore.kernel.org/20260601-zone_stat_item-comment-v1-1-f452dd91d5eb@google.com Signed-off-by: Brendan Jackman Acked-by: David Hildenbrand (Arm) Reviewed-by: Lorenzo Stoakes Acked-by: Vlastimil Babka (SUSE) Signed-off-by: Andrew Morton

mm: switch deferred split shrinker to list_lru

2026-06-09T01:21:25+00:00

The deferred split queue handles cgroups in a suboptimal fashion. The queue is per-NUMA node or per-cgroup, not the intersection. That means on a cgrouped system, a node-restricted allocation entering reclaim can end up splitting large pages on other nodes: alloc/unmap deferred_split_folio() list_add_tail(memcg->split_queue) set_shrinker_bit(memcg, node, deferred_shrinker_id) for_each_zone_zonelist_nodemask(restricted_nodes) mem_cgroup_iter() shrink_slab(node, memcg) shrink_slab_memcg(node, memcg) if test_shrinker_bit(memcg, node, deferred_shrinker_id) deferred_split_scan() walks memcg->split_queue The shrinker bit adds an imperfect guard rail. As soon as the cgroup has a single large page on the node of interest, all large pages owned by that memcg, including those on other nodes, will be split. list_lru properly sets up per-node, per-cgroup lists. As a bonus, it streamlines a lot of the list operations and reclaim walks. It's used widely by other major shrinkers already. Convert the deferred split queue as well. The list_lru per-memcg heads are instantiated on demand when the first object of interest is allocated for a cgroup, by calling folio_memcg_alloc_deferred(). Add calls to where splittable pages are created: anon faults, swapin faults, khugepaged collapse. These calls create all possible node heads for the cgroup at once, so the migration code (between nodes) doesn't need any special care. [akpm@linux-foundation.org: fix build with CONFIG_TRANSPARENT_HUGEPAGE=n] Link: https://lore.kernel.org/202605281620.lc3rtkBm-lkp@intel.com [hannes@cmpxchg.org: fix cgroup.memory=nokmem handling] Link: https://lore.kernel.org/ah9PGv12mqai84ES@cmpxchg.org Link: https://lore.kernel.org/20260527204757.2544958-10-hannes@cmpxchg.org Signed-off-by: Johannes Weiner Reported-by: Mikhail Zaslonko Tested-by: Mikhail Zaslonko Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes (Oracle) Acked-by: Usama Arif Reviewed-by: Kairui Song Cc: Baolin Wang Cc: Barry Song Cc: Dave Chinner Cc: David Hildenbrand (Arm) Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Michal Hocko Cc: Muchun Song Cc: Nico Pache Cc: Roman Gushchin Cc: Ryan Roberts Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: Zi Yan Cc: kernel test robot Signed-off-by: Andrew Morton

mm: introduce for_each_free_list()

2026-06-02T22:22:19+00:00

Patch series "mm: misc cleanups from __GFP_UNMAPPED series". In v2 of the __GFP_UNMAPPED series [0], we realised that some of the patches could potentially be merged as independent cleanups. These are all independent of one another, if you think some are useful cleanups and others are pointless churn, it should be fine to just pick whatever subset you prefer. No functional change intended. This patch (of 4): There are a couple of places that iterate over the freelists with awareness of the data structures' layout. It seems ideally, code outside of mm should not be aware of the page allocator's freelists at all. But, this patch just doesn't hide them completely, it's just a meek incremental step in that direction: provide a macro to iterate over it without needing to be aware of the actual struct fields. Link: https://lore.kernel.org/20260513-page_alloc-unmapped-prep-v1-0-dacdf5402be8@google.com Link: https://lore.kernel.org/20260513-page_alloc-unmapped-prep-v1-1-dacdf5402be8@google.com Link: https://lore.kernel.org/all/20260320-page_alloc-unmapped-v2-0-28bf1bd54f41@google.com/ [0] Signed-off-by: Brendan Jackman Acked-by: Mike Rapoport (Microsoft) Reviewed-by: Vlastimil Babka (SUSE) Cc: Axel Rasmussen Cc: Barry Song Cc: David Hildenbrand Cc: Johannes Weiner Cc: Kairui Song Cc: Len Brown Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: "Rafael J. Wysocki" Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Wei Xu Cc: Yuanchu Xie Cc: Zi Yan Signed-off-by: Andrew Morton