kernel/linux.git/mm/huge_memory.c, branch v7.2-rc7

mm/huge_memory: initialise workingset state before folio split

2026-08-05T03:02:03+00:00

xas_try_split() adds __GFP_ACCOUNT for page-cache xa_nodes, but __folio_split() leaves the xa_state's xa_lru unset. That lets a live, memcg-charged xa_node exist without being linked into the mapping's shadow_nodes list_lru; when reclaim later walks the list_lru it trips VM_WARN_ON(!css_is_dying()). Use mapping_set_update() to install both the workingset update callback and the shadow_nodes list_lru on the xa_state. Link: https://lore.kernel.org/20260725101419.3938406-1-matt@readmodwrite.com Fixes: 58729c04cf10 ("mm/huge_memory: add buddy allocator like (non-uniform) folio_split()") Signed-off-by: Matt Fleming Reported-by: syzbot+c5b060ce82921a2fd500@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=c5b060ce82921a2fd500 Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Barry Song Cc: Dave Chinner Cc: Dev Jain Cc: Kairui Song Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Muchun Song Cc: Nico Pache Cc: Roman Gushchin Cc: Ryan Roberts Cc: Shakeel Butt Cc: Signed-off-by: Andrew Morton

mm/huge_memory: separate out CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic

2026-08-05T03:02:02+00:00

Rather than mixing the refcounted and non-refcounted CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic, separate the two out cleanly so it is clear what happens when this configuration option is set and what happens when it is not. Introduce HUGE_ZERO_UNSET_PFN to abstract the ~0UL assignment, only introduce the refcount, lock and shrinker if !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, abstract initialisation and teardown, abstract the huge zero folio allocation from refcounting. Also change a BUG_ON() to WARN_ON_ONCE() while we're at it. No functional change intended. Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-2-c5d8a41b317f@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Fixes: 3b77e8c8cde5 ("mm/thp: make is_huge_zero_pmd() safe and quicker") Cc: Baolin Wang Cc: Barry Song Cc: David Hildenbrand (Arm) Cc: Dev Jain Cc: Hannes Reinecke Cc: Hengbin Zhang Cc: Hugh Dickins Cc: Kiryl Shutsemau Cc: Lance Yang Cc: Liam R. Howlett Cc: Nico Pache Cc: Pankaj Raghav Cc: Ryan Roberts Cc: Yang Shi Cc: Zi Yan Cc: Signed-off-by: Andrew Morton

mm/huge_memory: fix huge_zero_pfn race

2026-08-05T03:02:01+00:00

Patch series "mm/huge_memory: fix huge_zero_pfn race", v2. There is a subtle race in the reference-counted huge_zero_folio implementation. The fast path atomic logic fails to account for the fact that the shrinker (which drops the final huge_zero_refcount pin) can overwrite huge_zero_pfn with the ~0UL sentinel value in shrink_huge_zero_folio_scan() after a racing get_huge_zero_folio() installed a valid value there. This results in huge_zero_folio being correctly set but huge_zero_pfn being set incorrectly and thus is_huge_zero_pfn() and consequently is_huge_zero_pmd() will misidentify the huge zero folio as being an ordinary THP folio. This can result in the huge zero folio being split and otherwise treated incorrectly. The solution to this is very subtle as there is an atomic fast path, and thus ordering in weakly ordered architectures has to be treated very carefully. The first commit fixes the issue by introducing a spinlock around huge_zero_[pfn, folio, refcount] write, with careful consideration paid to load/store ordering in the fast path. It is placed first and kept as small as possible so that it can be backported on its own. The second commit is a pure cleanup which reworks the CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic to better separate the persistent logic from the dynamically allocated one. This patch (of 2): If !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, the huge_zero_folio is refcounted by huge_zero_refcount and returned by mm_get_huge_zero_folio(). When the caller is done with the huge zero page, its reference count is decremented. Only a shrinker can set the reference count to zero. A race can unfortunately occur between a shrinker decrementing the reference count to zero and a concurrent page fault. This is because shrink_huge_zero_folio_scan() might, if very unlucky, be preempted between setting huge_zero_refcount to zero and writing an invalid value. During this time get_huge_zero_folio() could write to huge_zero_pfn before shrink_huge_zero_folio_scan() resumes. In this event the huge zero folio will be persistently misidentified causing the THP code path to be entered inappropriately for the huge zero folio: CPU 0 CPU 1 =======================================|================================= shrink_huge_zero_folio_scan() | atomic_cmpxchg() sets refcount to 0 | xchg() sets huge_zero_folio to NULL | get_huge_zero_folio() | | atomic_inc_not_zero() -> zero preempted for a long time | Allocate new huge zero folio | | Write valid huge_zero_folio v | Write valid huge_zero_pfn Overwrite huge_zero_pfn with ~0UL <--- Invalid overwrite! This results in is_huge_zero_pfn() and is_huge_zero_pmd() incorrectly returning false for a huge zero page which could result in issues like the huge zero folio being incorrectly split. Note that the issue is with huge_zero_pfn not huge_zero_folio, as get_huge_zero_folio() uses cmpxchg() gated on huge_zero_folio being NULL with a retry loop and shrink_huge_zero_folio_scan() uses xchg() to set huge_zero_folio. Fix the issue by introducing a spinlock, huge_zero_lock, to prevent concurrent write of huge_zero_folio, huge_zero_pfn and huge_zero_refcount. There needs to be significant care taken here to ensure correctness: The fast path in get_huge_zero_folio() uses atomic_inc_not_zero(), which is outside of the critical section, and means huge zero allocation is gated on zero huge_zero_refcount. The fast path doesn't use huge_zero_lock, so the critical section is irrelevant to it. So invariants are required - huge_zero_refcount MUST: * Only be set in the huge_zero_lock critical section to ensure serialisation of huge_zero_pfn, huge_zero_folio and huge_zero_refcount writes. * Be set non-zero only AFTER huge_zero_[pfn, folio] are set to valid values so installation of the huge zero folio on read page fault ensures concurrent is_huge_zero_*() calls correctly identify the huge zero folio. * Be set zero only BEFORE huge_zero_[pfn, folio] are set to NULL and ~0UL respectively, and atomically. Establish these by: * Only setting huge_zero_refcount to zero or an absolute value in the huge_zero_lock critical section in get_huge_zero_folio() and shrink_huge_zero_folio_scan(), and always updating atomically there and elsewhere. * Using atomic_set_release(&huge_zero_refcount) in get_huge_zero_folio() after huge_zero_[pfn, folio] are set. This is paired with atomic_inc_not_zero() to ensure atomic_inc_not_zero() only observes a non-zero value if huge_zero_[pfn, folio] are set. * Using atomic_cmpxchg() in shrink_huge_zero_folio_scan() (as before) to ensure that it is set zero only when equal to 1 and set atomically. * atomic_cmpxchg() being fully ordered ensures this is done prior to huge_zero_[folio, pfn] being set to NULL and ~0UL respectively. Eliminate the retry loop in get_huge_zero_folio() as the atomic_cmpxchg() in shrink_huge_zero_folio_scan() is now performed under the lock, and replace with an equally locked atomic_inc() to set the reference count should the caller be raced on huge zero folio installation. folio_put() naturally implies a full memory barrier so its ordering is maintained correctly. The huge zero folio also cannot be released except when the shrinker does so as it is non-LRU and non-rmappable. Note that only the huge zero shrinker (via shrink_huge_zero_folio_scan()) can actually set huge_zero_refcount to zero, which is the count of mm's which have at least one huge zero folio installed plus one shrinker pin. Additionally convert a BUG_ON() to a VM_WARN_ON_ONCE(). Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-0-c5d8a41b317f@kernel.org Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-1-c5d8a41b317f@kernel.org Fixes: 3b77e8c8cde5 ("mm/thp: make is_huge_zero_pmd() safe and quicker") Signed-off-by: Lorenzo Stoakes (ARM) Reported-by: Hengbin Zhang Closes: https://lore.kernel.org/linux-mm/20260727154001.4102341-1-uqbarz@gmail.com/ Suggested-by: David Hildenbrand (Arm) Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Hannes Reinecke Cc: Hugh Dickins Cc: Kiryl Shutsemau Cc: Lance Yang Cc: Liam R. Howlett Cc: Nico Pache Cc: Pankaj Raghav Cc: Ryan Roberts Cc: Yang Shi Cc: Zi Yan Cc: Signed-off-by: Andrew Morton

mm/huge_memory: unlock i_mmap_rwsem before releasing after-split folios

2026-07-29T00:37:32+00:00

__folio_split() keeps dereferencing the mapping after the split: shmem_uncharge(mapping->host) and remap_page() while the folios are still frozen/locked, and i_mmap_unlock_read(mapping) at the very end, after the after-split folios have been unlocked and freed. Nothing holds an inode reference across that. The split relies on @folio -- which the beyond-EOF drop loop never removes, as it starts at folio_next(folio) -- staying locked and in the page cache to hold off eviction. But the unlock loop unlocks @folio before i_mmap_unlock_read() runs. If the caller's @lock_at is a tail beyond EOF, as memory_failure() passes when splitting a poisoned tail of a shmem THP that reaches past i_size during truncation, it too is gone from the page cache; so once @folio is unlocked no locked, in-cache folio pins the inode, and a concurrent final iput() can evict and RCU-free it before i_mmap_unlock_read() touches i_mmap_rwsem: BUG: KASAN: slab-use-after-free in __up_read+0x634/0x790 i_mmap_unlock_read include/linux/fs.h:537 [inline] __folio_split+0x732/0x1640 mm/huge_memory.c:4100 try_to_split_thp_page+0xab/0x390 mm/memory-failure.c:1675 memory_failure+0x1394/0x26e0 mm/memory-failure.c:2470 Freed by task 4601: shmem_free_in_core_inode+0x54/0xb0 mm/shmem.c:5177 evict+0x57f/0xac0 fs/inode.c:870 Do every mapping dereference while @folio still pins the inode: drop i_mmap_rwsem right after remap_page(), before the loop that unlocks and frees the after-split folios, and clear @mapping so the exit path does not unlock it again. shmem_uncharge() and remap_page() already run before that point, so after this nothing past the unlock loop touches the inode or the mapping. This is now a rule the split depends on, alongside keeping @folio frozen until the page cache is updated: no inode or mapping dereference once the after-split folios start being unlocked. Link: https://lore.kernel.org/20260716095424.471052-1-kirill@shutemov.name Fixes: baa355fd3314 ("thp: file pages support for split_huge_page()") Signed-off-by: Kiryl Shutsemau (Meta) Reported-by: Hao Zhang Closes: https://lore.kernel.org/linux-mm/20260710071344.GA106129@zh-pc Co-developed-by: Hao Zhang Signed-off-by: Hao Zhang Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Miaohe Lin Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Naoya Horiguchi Cc: Nico Pache Cc: Ryan Roberts Cc: Signed-off-by: Andrew Morton

userfaultfd: wait on source PMD during UFFDIO_MOVE

2026-07-21T00:41:26+00:00

move_pages_huge_pmd() snapshots src_pmdval under src_ptl, drops the lock, and, for migration entries, waits with pmd_migration_entry_wait(). Passing &src_pmdval is wrong. pmd_migration_entry_wait() must lock and re-read the real page-table PMD; on split-PMD-lock kernels, a stack address also resolves to the wrong lock. softleaf_entry_wait_on_locked() then waits without a folio reference, which is safe only while serialized against migration-entry removal by the real PT lock. Pass src_pmd, matching __handle_mm_fault() and hmm_vma_walk_pmd(). Link: https://lore.kernel.org/20260705131231.1499198-1-usama.arif@linux.dev Fixes: adef440691ba ("userfaultfd: UFFDIO_MOVE uABI") Reported-by: sashiko-bot Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=8 Signed-off-by: Usama Arif Reviewed-by: Rik van Riel Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Acked-by: David Hildenbrand (Arm) Reviewed-by: Lance Yang Reviewed-by: Lorenzo Stoakes Cc: Andrea Arcangeli Cc: Barry Song Cc: Dev Jain Cc: Johannes Weiner Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Shakeel Butt Cc: Zi Yan Cc: Signed-off-by: Andrew Morton

mm/huge_memory: set PG_has_hwpoisoned only after new folio head is established

2026-07-09T22:48:54+00:00

__split_folio_to_order() copies the hwpoison state onto each new sub-folio while splitting a folio to a non-zero order. It does so via if (handle_hwpoison && page_range_has_hwpoisoned(new_head, new_nr_pages)) folio_set_has_hwpoisoned(new_folio); *before* clear_compound_head(new_head)/prep_compound_page(new_head, ...) turns @new_head from a tail page into a proper folio head. PG_has_hwpoisoned is a FOLIO_SECOND_PAGE flag, so folio_set_has_hwpoisoned() resolves to folio_flags(folio, 1). With the new compound_info-based page-flags layout, folio_flags() asserts the page is not a tail: VM_BUG_ON_PGFLAGS(page->compound_info & 1, page); VM_BUG_ON_PGFLAGS(n > 0 && !test_bit(PG_head, &page->flags.f), page); At the current call site @new_head still has the tail marker (compound_info bit 0 set, PG_head clear), so on CONFIG_DEBUG_VM kernels this hits: kernel BUG at include/linux/page-flags.h:354 folio_flags+0x82 folio_set_has_hwpoisoned __split_folio_to_order __split_unmapped_folio __folio_split truncate_inode_partial_folio (shmem hole-punch / MADV_REMOVE) Reproduced by syzkaller: hwpoison-inject a few subpages of a large shmem folio, then MADV_REMOVE (fallocate punch hole) on the same range, which splits the partial folio to a non-zero order. memory_failure() tries to split the poisoned folio to order 0 first, but that split is best-effort; when it fails the folio is left large with PG_has_hwpoisoned set, the case fa5a06170036 added this hwpoison copying for. Move the folio_set_has_hwpoisoned() call to after clear_compound_head()/prep_compound_page(), where @new_folio is a real order-new_order head folio (handle_hwpoison implies new_order != 0, so a second page always exists). The flag still lands on the same struct page (page[1] of the new folio); only the ordering relative to compound-head setup changes, satisfying the FOLIO_SECOND_PAGE precondition. Link: https://lore.kernel.org/20260701174235.3173401-1-riel@surriel.com Fixes: fa5a06170036 ("mm/huge_memory: preserve PG_has_hwpoisoned if a folio is split to >0 order") Signed-off-by: Rik van Riel Assisted-by: Claude:claude-opus-4-8 Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Tested-by: Lance Yang Reviewed-by: Lorenzo Stoakes Reviewed-by: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Nico Pache Cc: Ryan Roberts Cc: Yang Shi Cc: Signed-off-by: Andrew Morton

Merge tag 'mm-stable-2026-06-23-08-55' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

2026-06-23T19:03:44+00:00

Pull more MM updates from Andrew Morton: - "khugepaged: add mTHP collapse support" (Nico Pache) Provide khugepaged with the capability to collapse anonymous memory regions to mTHPs - "Remove CONFIG_READ_ONLY_THP_FOR_FS and enable file THP for writable files" (Zi Yan) Remove the READ_ONLY_THP_FOR_FS check in file_thp_enabled(), so that khugepaged and MADV_COLLAPSE can run on filesystems with PMD THP pagecache support even without READ_ONLY_THP_FOR_FS enabled - "make MM selftests more CI friendly" (Mike Rapoport) General fixes and cleanups to the MM selftests. Also move more MM selftests under the kselftest framework, making them more amenable to ongoing CI testing - "selftests/mm: fix failures and robustness improvements" and "selftests/mm: assorted fixes for hmm-tests" (Sayali Patil) Fix several issues in MM selftests which were revealed by powerpc 64k pagesize * tag 'mm-stable-2026-06-23-08-55' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (118 commits) Revert "mm: limit filemap_fault readahead to VMA boundaries" mm/vmscan: pass NULL to trace vmscan node reclaim mm: use mapping_mapped to simplify the code selftests/mm: fix exclusive_cow test fork() handling selftests/mm: remove hardcoded THP sizing assumptions in hmm tests selftests/mm: allow PUD-level entries in compound testcase of hmm tests mm/gup_test: reject wrapped user ranges mm/page_frag: reject invalid CPUs in page_frag_test mm/damon/core: always put unsuccessfully committed target pids mm: page_isolation: avoid unsafe folio reads while scanning compound pages mm/shrinker: do not hold RCU lock in shrinker_debugfs_count_show() selftests: mm: fix and speedup "droppable" test mm: merge writeout into pageout MAINTAINERS: add Hao Ge as reviewer for codetag and alloc_tag selftests/mm: clarify alternate unmapping in compaction_test selftests/mm: move hwpoison setup into run_test() and silence modprobe output for memory-failure category selftests/mm: skip uffd-stress test when nr_pages_per_cpu is zero selftests/mm: skip uffd-wp-mremap if UFFD write-protect is unsupported selftests/mm: ensure destination is hugetlb-backed in hugetlb-mremap selftest/mm: register existing mapping with userfaultfd in hugetlb-mremap ...

mm/khugepaged: enable clean pagecache folio collapse for writable files

2026-06-21T18:37:17+00:00

collapse_file() is capable of collapsing pagecache folios from writable files to PMD folios. Now enable clean pagecache folio collapse in addition to read-only pagecache folio collapse by removing the inode_is_open_for_write() from file_thp_enabled() and only performing filemap_flush() if the file is read-only. This means userspace needs to explicitly flush the content of pagecache folios before khugepaged can collapse the folios, or use madvise(MADV_COLLAPSE), which does the flush in the retry. The reason is that blindly enabling dirty pagecache folio from writable files collapse makes khugepaged flush these folios all the time. It is undesirable to cause system level pagecache flushes. To properly support dirty pagecache folio collapse, filemap_flush() needs to be avoided. Potentially, merging associated buffer instead of dropping it with filemap_release_folio() might be needed. NOTE: this breaks khugepaged selftests for writable file pagecache collapse, which is set to fail all the time. The next commit fixes it. Link: https://lore.kernel.org/20260517135416.1434539-14-ziy@nvidia.com Signed-off-by: Zi Yan Reviewed-by: Lance Yang Cc: Al Viro Cc: Baolin Wang Cc: Barry Song Cc: Chris Mason Cc: Christian Brauner Cc: David Hildenbrand (Arm) Cc: David Sterba Cc: Dev Jain Cc: Jan Kara Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Mike Rapoport Cc: Nico Pache Cc: Ryan Roberts Cc: Shuah Khan Cc: Song Liu Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton

mm/huge_memory: remove folio split check for READ_ONLY_THP_FOR_FS

2026-06-21T18:37:15+00:00

Without READ_ONLY_THP_FOR_FS, large file-backed folios cannot be created by a FS without large folio support. The check is no longer needed. Link: https://lore.kernel.org/20260517135416.1434539-9-ziy@nvidia.com Signed-off-by: Zi Yan Reviewed-by: Lance Yang Reviewed-by: Lorenzo Stoakes (Oracle) Reviewed-by: Baolin Wang Acked-by: David Hildenbrand (Arm) Cc: Al Viro Cc: Barry Song Cc: Chris Mason Cc: Christian Brauner Cc: David Sterba Cc: Dev Jain Cc: Jan Kara Cc: Liam Howlett Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Mike Rapoport Cc: Nico Pache Cc: Ryan Roberts Cc: Shuah Khan Cc: Song Liu Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton

mm: fs: remove filemap_nr_thps*() functions and their users

2026-06-21T18:37:15+00:00

They are used by READ_ONLY_THP_FOR_FS to handle writes to FSes without large folio support, so that read-only THPs created in these FSes are not seen by the FSes when the underlying fd becomes writable. Now read-only PMD THPs only appear in a FS with large folio support and the supported orders include PMD_ORDER. READ_ONLY_THP_FOR_FS was using mapping->nr_thps, inode->i_writecount, and smp_mb() to prevent writes to a read-only THP and collapsing writable folios into a THP. In collapse_file(), mapping->nr_thps is increased, then smp_mb(), and if inode->i_writecount > 0, collapse is stopped, while do_dentry_open() first increases inode->i_writecount, then a full memory fence, and if mapping->nr_thps > 0, all read-only THPs are truncated. Now this mechanism can be removed along with READ_ONLY_THP_FOR_FS code, since a dirty folio check has been added after try_to_unmap() in collapse_file() to prevent dirty folios from being collapsed as clean. Link: https://lore.kernel.org/20260517135416.1434539-7-ziy@nvidia.com Signed-off-by: Zi Yan Reviewed-by: Matthew Wilcox (Oracle) Acked-by: David Hildenbrand (Arm) Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Cc: Al Viro Cc: Barry Song Cc: Chris Mason Cc: Christian Brauner Cc: David Sterba Cc: Dev Jain Cc: Jan Kara Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Nico Pache Cc: Ryan Roberts Cc: Shuah Khan Cc: Song Liu Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton