kernel/linux.git/mm/slab_common.c, branch v7.0.10

Merge branch 'slab/for-7.0/sheaves' into slab/for-next

2026-02-10T08:10:00+00:00

Merge series "slab: replace cpu (partial) slabs with sheaves". The percpu sheaves caching layer was introduced as opt-in but the goal was to eventually move all caches to them. This is the next step, enabling sheaves for all caches (except the two bootstrap ones) and then removing the per cpu (partial) slabs and lots of associated code. Besides the lower locking overhead and much more likely fastpath when freeing, this removes the rather complicated code related to the cpu slab lockless fastpaths (using this_cpu_try_cmpxchg128/64) and all its complications for PREEMPT_RT or kmalloc_nolock(). The lockless slab freelist+counters update operation using try_cmpxchg128/64 remains and is crucial for freeing remote NUMA objects and to allow flushing objects from sheaves to slabs mostly without the node list_lock. Link: https://lore.kernel.org/all/20260123-sheaves-for-all-v4-0-041323d506f7@suse.cz/

mm/slab: only allow SLAB_OBJ_EXT_IN_OBJ for unmergeable caches

2026-02-04T09:05:36+00:00

While SLAB_OBJ_EXT_IN_OBJ allows to reduce memory overhead to account slab objects, it prevents slab merging because merging can change the metadata layout. As pointed out Vlastimil Babka, disabling merging solely for this memory optimization may not be a net win, because disabling slab merging tends to increase overall memory usage. Restrict SLAB_OBJ_EXT_IN_OBJ to caches that are already unmergeable for other reasons (e.g., those with constructors or SLAB_TYPESAFE_BY_RCU). Suggested-by: Vlastimil Babka Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260127103151.21883-3-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

mm/slab: place slabobj_ext metadata in unused space within s->size

2026-02-04T09:05:36+00:00

When a cache has high s->align value and s->object_size is not aligned to it, each object ends up with some unused space because of alignment. If this wasted space is big enough, we can use it to store the slabobj_ext metadata instead of wasting it. On my system, this happens with caches like kmem_cache, mm_struct, pid, task_struct, sighand_cache, xfs_inode, and others. To place the slabobj_ext metadata within each object, the existing slab_obj_ext() logic can still be used by setting: - slab->obj_exts = slab_address(slab) + (slabobj_ext offset) - stride = s->size slab_obj_ext() doesn't need know where the metadata is stored, so this method works without adding extra overhead to slab_obj_ext(). A good example benefiting from this optimization is xfs_inode (object_size: 992, align: 64). To measure memory savings, 2 millions of files were created on XFS. [ MEMCG=y, MEM_ALLOC_PROFILING=n ] Before patch (creating ~2.64M directories on xfs): Slab: 5175976 kB SReclaimable: 3837524 kB SUnreclaim: 1338452 kB After patch (creating ~2.64M directories on xfs): Slab: 5152912 kB SReclaimable: 3838568 kB SUnreclaim: 1314344 kB (-23.54 MiB) Enjoy the memory savings! Suggested-by: Vlastimil Babka Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260113061845.159790-10-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

mm/slab: move [__]ksize and slab_ksize() to mm/slub.c

2026-02-04T09:05:35+00:00

To access SLUB's internal implementation details beyond cache flags in ksize(), move __ksize(), ksize(), and slab_ksize() to mm/slub.c. [vbabka@suse.cz: also make __ksize() static and move its kerneldoc to ksize() ] Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260113061845.159790-9-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

mm/slab: allow specifying free pointer offset when using constructor

2026-02-04T09:05:35+00:00

When a slab cache has a constructor, the free pointer is placed after the object because certain fields must not be overwritten even after the object is freed. However, some fields that the constructor does not initialize can safely be overwritten after free. Allow specifying the free pointer offset within the object, reducing the overall object size when some fields can be reused for the free pointer. Adjust the document accordingly. Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260113061845.159790-3-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

slab: introduce percpu sheaves bootstrap

2026-01-29T08:21:38+00:00

Until now, kmem_cache->cpu_sheaves was !NULL only for caches with sheaves enabled. Since we want to enable them for almost all caches, it's suboptimal to test the pointer in the fast paths, so instead allocate it for all caches in do_kmem_cache_create(). Instead of testing the cpu_sheaves pointer to recognize caches (yet) without sheaves, test kmem_cache->sheaf_capacity for being 0, where needed, using a new cache_has_sheaves() helper. However, for the fast paths sake we also assume that the main sheaf always exists (pcs->main is !NULL), and during bootstrap we cannot allocate sheaves yet. Solve this by introducing a single static bootstrap_sheaf that's assigned as pcs->main during bootstrap. It has a size of 0, so during allocations, the fast path will find it's empty. Since the size of 0 matches sheaf_capacity of 0, the freeing fast paths will find it's "full". In the slow path handlers, we use cache_has_sheaves() to recognize that the cache doesn't (yet) have real sheaves, and fall back. Thus sharing the single bootstrap sheaf like this for multiple caches and cpus is safe. Reviewed-by: Harry Yoo Reviewed-by: Hao Li Reviewed-by: Liam R. Howlett Signed-off-by: Vlastimil Babka

mm/slab: factor out slab_args_unmergeable()

2026-01-27T16:59:38+00:00

slab_mergeable() determines whether a slab cache can be merged, but it should not be used when the cache is not fully created yet. Extract the pre-cache-creation mergeability checks into slab_args_unmergeable(), which evaluates kmem_cache_args, slab flags, and slab_nomerge to determine if a cache will be mergeable before it is created. Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260127103151.21883-2-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

mm/slab: make caches with sheaves mergeable

2026-01-27T16:59:31+00:00

Before enabling sheaves for all caches (with automatically determined capacity), their enablement should no longer prevent merging of caches. Limit this merge prevention only to caches that were created with a specific sheaf capacity, by adding the SLAB_NO_MERGE flag to them. Reviewed-by: Harry Yoo Reviewed-by: Liam R. Howlett Signed-off-by: Vlastimil Babka

mm/slab: move and refactor __kmem_cache_alias()

2026-01-27T16:58:06+00:00

Move __kmem_cache_alias() to slab_common.c since it's called by __kmem_cache_create_args() and calls find_mergeable() that both are in this file. We can remove two slab.h declarations and make them static. Instead declare sysfs_slab_alias() from slub.c so that __kmem_cache_alias() can keep calling it. Add args parameter to __kmem_cache_alias() and find_mergeable() instead of align and ctor. With that we can also move the checks for usersize and sheaf_capacity there from __kmem_cache_create_args() and make the result more symmetric with slab_unmergeable(). No functional changes intended. Reviewed-by: Harry Yoo Reviewed-by: Suren Baghdasaryan Reviewed-by: Liam R. Howlett Signed-off-by: Vlastimil Babka

slab: add SLAB_CONSISTENCY_CHECKS to SLAB_NEVER_MERGE

2026-01-27T16:55:54+00:00

All the debug flags prevent merging, except SLAB_CONSISTENCY_CHECKS. This is suboptimal because this flag (like any debug flags) prevents the usage of any fastpaths, and thus affect performance of any aliased cache. Also the objects from an aliased cache than the one specified for debugging could also interfere with the debugging efforts. Fix this by adding the whole SLAB_DEBUG_FLAGS collection to SLAB_NEVER_MERGE instead of individual debug flags, so it now also includes SLAB_CONSISTENCY_CHECKS. Reviewed-by: Suren Baghdasaryan Reviewed-by: Harry Yoo Signed-off-by: Vlastimil Babka