kernel/linux.git/mm/interval_tree.c, branch v7.3-rc1

mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios

2026-08-25T01:42:53+00:00

Currently anonymous folios belonging to CoW'd MAP_PRIVATE file-backed mappings are indexed by their page offset within the file in which they were originally mapped. This differs from anonymous folios belonging to pure anon mappings which are indexed by their anonymous page offset (the address at which they'd belong in the VMA when first faulted). This change fixes this inconsistency, always indexing anonymous folios by their anonymous page offset regardless of the VMA to which they belong. The foundations have been laid such that we need only switch this functionality on such by: * Using linear_anon_page_index() in __folio_set_anon() to assign the folio's index to the anonymous linear index rather than the file-backed one. * Otherwise using linear_anon_page_index() in all instances where anonymous folios are being referenced or manipulated. * Replacing vma_address() with vma_filebacked_address() or vma_anon_address() as appropriate. * Updating the merging logic to check that anonymous page offsets are aligned as well as filebacked ones for MAP_PRIVATE file-backed VMAs, introducing needs_adjacent_anon_pgoff() to figure out when this is required. * Updating linear_folio_page_index() to invoke linear_anon_page_index() if the folio is anonymous. * Updating vma_address_end() to use the VMA's anonymous page offset when pvmw->pgoff is anonymous. * Correcting folio_within_range() to use anonymous page offset for anonymous folios. This will have no impact on merging of anonymous VMAs, whose page offset and anonymous page offset are identical, nor will it impact shared file-backed VMAs, which will continue to be merged based on the file-backed page offset. However, MAP_PRIVATE file-backed mappings must now be aligned on anonymous page offset as well. In most instances this should have no impact on merging of file-backed mappings, which are usually not merged all that often, let alone MAP_PRIVATE mapped ones, and rarely remapped and faulted before being moved back in place (the case in which a merge may now fail). One subtle impact of this change is in NUMA interleaving - since commit 88c91dc58582 ("mempolicy: migration attempt to match interleave nodes"), migration heuristically tries to maintain interleaving behaviour matching the policy using folio indices. When doing migration of CoW'd MAP_PRIVATE-file backed ranges, the 'base' upon which the interleaving behaviour is performed will vary for these ranges. However the commit notes that ranges spanning multiple VMAs will already cause varying bases, and that this is an acceptable approximation. It is very unlikely real world use-cases will be impacted by this (MAP_PRIVATE file-backed mappings are already an edge case), and all that will happen is that such ranges will cause interleaving to be rotated over the CoW'd range, with little to no impact. This commit lays the foundations for future scalable CoW work which needs to track some remaps, meaning that most remap tracking can be avoided, and in nearly all cases the anonymous page offset will be able to be used to quickly find the VMA in an mm. Note that the need_rmap_locks check doesn't need to be updated, as any remapping will offset both the anonymous and file-backed page offset, so it suffices to check only one. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-14-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton

mm/rmap: rename anon_vma_interval_tree_() to anon_rmap_tree_()

2026-08-05T02:18:58+00:00

To be consistent with the newly renamed mapping_rmap_tree_*(), rename the anon_vma_interval_tree_*() helpers to anon_rmap_tree_*(). No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-11-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes Reviewed-by: Gregory Price Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Zi Yan Cc: Ackerley Tng Cc: David Hildenbrand (Arm) Cc: Kai Huang Cc: Marek Szyprowski Cc: Pedro Falcato Cc: SJ Park Cc: Thomas Zimmermann Cc: Liam R. Howlett (Oracle) Signed-off-by: Andrew Morton

mm/rmap: rename anon_vma_interval_tree_*() params and use pgoff_t

2026-08-05T02:18:58+00:00

Rename parameters used by anon_vma_interval_tree_*() functions: 'node' to 'avc', 'start/first' to 'pgoff_start', and 'last' to 'pgoff_last' to make clear what is being passed. Also, express page offsets in terms of pgoff_t to be consistent. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-10-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes Reviewed-by: Gregory Price Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Zi Yan Cc: Ackerley Tng Cc: David Hildenbrand (Arm) Cc: Kai Huang Cc: Marek Szyprowski Cc: Pedro Falcato Cc: SJ Park Cc: Thomas Zimmermann Cc: Liam R. Howlett (Oracle) Signed-off-by: Andrew Morton

mm/rmap: parameterise anon_vma_interval_tree_*() by anon_vma

2026-08-05T02:18:58+00:00

Similar to what we did with mapping_rmap_tree*(), let's declare anon_vma_interval_tree*() in terms of anon_vma rather than rb_root_cached. In each case the rb tree referenced is &anon_vma->rb_root, so just pass anon_vma and the functions can figure this out themselves. Also update the VMA userland tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-9-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes Reviewed-by: Pedro Falcato Reviewed-by: Gregory Price Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Zi Yan Cc: Ackerley Tng Cc: David Hildenbrand (Arm) Cc: Kai Huang Cc: Marek Szyprowski Cc: SJ Park Cc: Thomas Zimmermann Cc: Liam R. Howlett (Oracle) Signed-off-by: Andrew Morton

mm/rmap: rename vma_interval_tree_() to mapping_rmap_tree_()

2026-08-05T02:18:58+00:00

The family of vma_interval_tree_() functions manipulate the address_space (which, of course, is generally referred to as 'mapping') reverse mapping, but are named the 'VMA' interval tree. VMAs may be mapped by an anon_vma, an address_space, or both. Therefore calling the mapping interval tree a 'VMA' interval tree is rather confusing. This is also inconsistent with the anon_vma_interval_tree_*() functions which explicitly reference the rmap object to which they pertain. Rename the vma_interval_tree_*() functions to mapping_rmap_tree_*() to correct this. We will rename the anon rmap functions similarly in a subsequent patch. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-8-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes Reviewed-by: Gregory Price Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Zi Yan Cc: Ackerley Tng Cc: David Hildenbrand (Arm) Cc: Kai Huang Cc: Marek Szyprowski Cc: Pedro Falcato Cc: SJ Park Cc: Thomas Zimmermann Cc: Liam R. Howlett (Oracle) Signed-off-by: Andrew Morton

mm/rmap: elide unnecessary static inline's in interval_tree.c

2026-08-05T02:18:57+00:00

It's not necessary to declare these functions static inline as they are contained within a single compilation unit. This makes the anonymous interval tree code consistent with the newly updated file-backed interval tree code. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-7-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes Reviewed-by: Pedro Falcato Reviewed-by: Gregory Price Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Zi Yan Cc: Ackerley Tng Cc: David Hildenbrand (Arm) Cc: Kai Huang Cc: Marek Szyprowski Cc: SJ Park Cc: Thomas Zimmermann Cc: Liam R. Howlett (Oracle) Signed-off-by: Andrew Morton

mm/rmap: parameterise vma_interval_tree_*() by address_space

2026-08-05T02:18:57+00:00

The file-backed mapping interval tree functions vma_interval_tree_*() accept a raw rb_root_cached pointer to determine the tree in which they are operating. However, in each case, this is always associated with an address_space data type. So simply pass a pointer to that instead to simplify the code, and more clearly differentiate between these operations and those concerning anonymous mappings. While we're here, make the generated interval tree functions static as they do not need to be used externally (any previously existing external users have now been removed). We also rename VMA parameters from 'node' to 'vma' as calling this a node is simply confusing, update the input index types to pgoff_t since they reference page offsets and rename the parameters to pgoff_start and pgoff_last. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-6-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes Reviewed-by: Pedro Falcato Reviewed-by: Gregory Price Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Zi Yan Cc: Ackerley Tng Cc: David Hildenbrand (Arm) Cc: Kai Huang Cc: Marek Szyprowski Cc: SJ Park Cc: Thomas Zimmermann Cc: Liam R. Howlett (Oracle) Signed-off-by: Andrew Morton

mm/rmap: update mm/interval_tree.c comments

2026-08-05T02:18:57+00:00

Update the file comment to clarify that both file-backed and anonymous interval trees are provided, referencing the relevant data types for clarity. Also add comments to indicate which parts of the file apply to each. While we're here, convert the VM_BUG_ON_VMA() to VM_WARN_ON_ONCE_VMA(). Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-5-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes Reviewed-by: Pedro Falcato Reviewed-by: Gregory Price Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Zi Yan Cc: Ackerley Tng Cc: David Hildenbrand (Arm) Cc: Kai Huang Cc: Marek Szyprowski Cc: SJ Park Cc: Thomas Zimmermann Cc: Liam R. Howlett (Oracle) Signed-off-by: Andrew Morton

mm: move vma_start_pgoff() into mm.h and clean up

2026-08-05T02:18:56+00:00

Patch series "mm: make VMA page offset handling more consistent", v2. This series performs a series of cleanups and improvements around how the vma->vm_pgoff field is used. Folios belonging to file-backed mappings are simply indexed by the page offset within the file they map (excepting pfnmap and driver mappings which sometimes do things different). However, anonymous folios belonging to pure anonymous mappings are indexed by their "virtual" page offset, which is equal to addr >> PAGE_SHIFT at the time at which the VMA was first faulted in. The page offset of a VMA is stored in vma->vm_pgoff and indicates the page offset of the start of the VMA range, whether it be file-backed or anonymous. The work here both cleans up how we reference this field, as well as laying the foundations for a future series which addresses the inconsistency of CoW'd folios in MAP_PRIVATE-file backed mappings, which are indexed as if they were file-backed but behave as if they were anonymous. This future series will make it such that all anonymous folios are indexed by virtual page offset whether belonging to VMAs who satisfy vma_is_anonymous() or MAP_PRIVATE-mapped file-backed mappings. This series: * Exposes vma_start_pgoff() and updates the kernel to use it consistently. * Adds and uses the useful vma_end_pgoff() helper. * Parameterises the file-backed mapping helpers vma_interval_tree_*() by address_space rather than rb_root_cached. * Renames: the misleadingly-named vma_interval_tree_*() helpers to mapping_rmap_tree_*() to be consistent with the renamed anon_rmap_tree_*(). * Parameterises anon_rmap_tree_*() by anon_vma. * Moves mm/interval_tree.c to the rmap section. * Adds vmg_*() helpers for page offset. * Clarifies the confusing vmg_adjust_set_range() function. * Introduces linear_page_delta() to provide relative pgoff within a VMA. * Replaces open-coded versions of linear_page_delta() and linear_page_index() with invocations of these functions. * Introduces and uses vma_assert_can_modify() to account for whether a VMA can be modified (detached or write locked). * Adds and uses vma_[add,sub]_pgoff() to adjust VMA page offset. * Moves __install_special_mapping() to vma.c. * Makes vma_set_range() static and internal to vma.c. * Introduces and makes use of vma_set_pgoff(). * Fixes incorrect vma.h header inclusion. * Defaults VMA userland tests to 64-bit vma flags size. * Updates VMA userland tests to give better output on failure. * Various smaller cleanups. * Updated the vma_set_pgoff() assert to account for MAP_PRIVATE /dev/zero correctly. This patch (of 33): vma_last_pgoff() already lives there, so it's a bit odd to keep vma_start_pgoff() in mm/interval_tree.c. Move them together. These each return unsigned long, which pgoff_t is typedef'd to. Make this consistent and have these functions return pgoff_t instead. Additionally, express vma_last_pgoff() in terms of vma_start_pgoff(), since we wrap the vma->vm_pgoff access, we may as well use it here. Also while we're here, const-ify the VMA and cleanup a bit. Also update the VMA userland tests to reflect the change. No functional change intended. Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-0-2a5aa403d977@kernel.org Link: https://lore.kernel.org/20260710-b4-pre-scalable-cow-v2-1-2a5aa403d977@kernel.org Signed-off-by: Lorenzo Stoakes Reviewed-by: Gregory Price Acked-by: David Hildenbrand (Arm) Reviewed-by: Pedro Falcato Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Liam R. Howlett (Oracle) Reviewed-by: Zi Yan Cc: Ackerley Tng Cc: Kai Huang Cc: Marek Szyprowski Cc: SJ Park Cc: Thomas Zimmermann Signed-off-by: Andrew Morton

mm/memory: simplify calculation in unmap_mapping_range_tree()

2026-04-05T20:53:13+00:00

Let's simplify the calculation a bit further to make it easier to get, reusing vma_last_pgoff() which we move from interval_tree.c to mm.h. Link: https://lkml.kernel.org/r/20260227200848.114019-5-david@kernel.org Signed-off-by: David Hildenbrand (Arm) Reviewed-by: Lorenzo Stoakes (Oracle) Cc: Alexander Gordeev Cc: Alexei Starovoitov Cc: Alice Ryhl Cc: Andrii Nakryiko Cc: Andy Lutomirski Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Arve Cc: "Borislav Petkov (AMD)" Cc: Carlos Llamas Cc: Christian Borntraeger Cc: Christian Brauner Cc: Claudio Imbrenda Cc: Daniel Borkman Cc: Dave Airlie Cc: David Ahern Cc: David Rientjes Cc: David S. Miller Cc: Dimitri Sivanich Cc: Eric Dumazet Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Hartley Sweeten Cc: Heiko Carstens Cc: Ian Abbott Cc: Ingo Molnar Cc: Jakub Kacinski Cc: Jani Nikula Cc: Jann Horn Cc: Janosch Frank Cc: Jarkko Sakkinen Cc: Jason Gunthorpe Cc: Jonas Lahtinen Cc: Leon Romanovsky Cc: Liam Howlett Cc: Madhavan Srinivasan Cc: Matthew Wilcox (Oracle) Cc: Michael Ellerman Cc: Michal Hocko Cc: Miguel Ojeda Cc: Mike Rapoport Cc: Namhyung kim Cc: Neal Cardwell Cc: Paolo Abeni Cc: Pedro Falcato Cc: Peter Zijlstra Cc: Rodrigo Vivi Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Todd Kjos Cc: Tvrtko Ursulin Cc: Vasily Gorbik Cc: Vincenzo Frascino Signed-off-by: Andrew Morton

kernel/linux.git/mm/interval_tree.c, branch v7.3-rc1

mm/rmap: use anon pgoff to track MAP_PRIVATE file-backed anon folios

mm/rmap: rename anon_vma_interval_tree_*() to anon_rmap_tree_*()

mm/rmap: rename anon_vma_interval_tree_*() params and use pgoff_t

mm/rmap: parameterise anon_vma_interval_tree_*() by anon_vma

mm/rmap: rename vma_interval_tree_*() to mapping_rmap_tree_*()

mm/rmap: elide unnecessary static inline's in interval_tree.c

mm/rmap: parameterise vma_interval_tree_*() by address_space

mm/rmap: update mm/interval_tree.c comments

mm: move vma_start_pgoff() into mm.h and clean up

mm/memory: simplify calculation in unmap_mapping_range_tree()

mm/rmap: rename anon_vma_interval_tree_() to anon_rmap_tree_()

mm/rmap: rename vma_interval_tree_() to mapping_rmap_tree_()