kernel/linux.git/mm/slab.h, branch v7.0.10

mm/slab: change stride type from unsigned short to unsigned int

2026-03-04T10:05:57+00:00

Commit 7a8e71bc619d ("mm/slab: use stride to access slabobj_ext") defined the type of slab->stride as unsigned short, because the author initially planned to store stride within the lower 16 bits of the page_type field, but later stored it in unused bits in the counters field instead. However, the idea of having only 2-byte stride turned out to be a serious mistake. On systems with 64k pages, order-1 pages are 128k, which is larger than USHRT_MAX. It triggers a debug warning because s->size is 128k while stride, truncated to 2 bytes, becomes zero: ------------[ cut here ]------------ Warning! stride (0) != s->size (131072) WARNING: mm/slub.c:2231 at alloc_slab_obj_exts_early.constprop.0+0x524/0x534, CPU#6: systemd-sysctl/307 Modules linked in: CPU: 6 UID: 0 PID: 307 Comm: systemd-sysctl Not tainted 7.0.0-rc1+ #6 PREEMPTLAZY Hardware name: IBM,9009-22A POWER9 (architected) 0x4e0202 0xf000005 of:IBM,FW950.E0 (VL950_179) hv:phyp pSeries NIP: c0000000008a9ac0 LR: c0000000008a9abc CTR: 0000000000000000 REGS: c0000000141f7390 TRAP: 0700 Not tainted (7.0.0-rc1+) MSR: 8000000000029033 CR: 28004400 XER: 00000005 CFAR: c000000000279318 IRQMASK: 0 GPR00: c0000000008a9abc c0000000141f7630 c00000000252a300 c00000001427b200 GPR04: 0000000000000004 0000000000000000 c000000000278fd0 0000000000000000 GPR08: fffffffffffe0000 0000000000000000 0000000000000000 0000000022004400 GPR12: c000000000f644b0 c000000017ff8f00 0000000000000000 0000000000000000 GPR16: 0000000000000000 c0000000141f7aa0 0000000000000000 c0000000141f7a88 GPR20: 0000000000000000 0000000000400cc0 ffffffffffffffff c00000001427b180 GPR24: 0000000000000004 00000000000c0cc0 c000000004e89a20 c00000005de90011 GPR28: 0000000000010010 c00000005df00000 c000000006017f80 c00c000000177a00 NIP [c0000000008a9ac0] alloc_slab_obj_exts_early.constprop.0+0x524/0x534 LR [c0000000008a9abc] alloc_slab_obj_exts_early.constprop.0+0x520/0x534 Call Trace: [c0000000141f7630] [c0000000008a9abc] alloc_slab_obj_exts_early.constprop.0+0x520/0x534 (unreliable) [c0000000141f76c0] [c0000000008aafbc] allocate_slab+0x154/0x94c [c0000000141f7760] [c0000000008b41c0] refill_objects+0x124/0x16c [c0000000141f77c0] [c0000000008b4be0] __pcs_replace_empty_main+0x2b0/0x444 [c0000000141f7810] [c0000000008b9600] __kvmalloc_node_noprof+0x840/0x914 [c0000000141f7900] [c000000000a3dd40] seq_read_iter+0x60c/0xb00 [c0000000141f7a10] [c000000000b36b24] proc_reg_read_iter+0x154/0x1fc [c0000000141f7a50] [c0000000009cee7c] vfs_read+0x39c/0x4e4 [c0000000141f7b30] [c0000000009d0214] ksys_read+0x9c/0x180 [c0000000141f7b90] [c00000000003a8d0] system_call_exception+0x1e0/0x4b0 [c0000000141f7e50] [c00000000000d05c] system_call_vectored_common+0x15c/0x2ec This leads to slab_obj_ext() returning the first slabobj_ext or all objects and confuses the reference counting of object cgroups [1] and memory (un)charging for memory cgroups [2]. Fortunately, the counters field has 32 unused bits instead of 16 on 64-bit CPUs, which is wide enough to hold any value of s->size. Change the type to unsigned int. Reported-by: Venkat Rao Bagalkote Closes: https://lore.kernel.org/lkml/ca241daa-e7e7-4604-a48d-de91ec9184a5@linux.ibm.com [1] Closes: https://lore.kernel.org/all/ddff7c7d-c0c3-4780-808f-9a83268bbf0c@linux.ibm.com [2] Fixes: 7a8e71bc619d ("mm/slab: use stride to access slabobj_ext") Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260303135722.2680521-1-harry.yoo@oracle.com Reviewed-by: Hao Li Tested-by: Venkat Rao Bagalkote Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: mark alloc tags empty for sheaves allocated with __GFP_NO_OBJ_EXT

2026-02-26T16:30:32+00:00

alloc_empty_sheaf() allocates sheaves from SLAB_KMALLOC caches using __GFP_NO_OBJ_EXT to avoid recursion, however it does not mark their allocation tags empty before freeing, which results in a warning when CONFIG_MEM_ALLOC_PROFILING_DEBUG is set. Fix this by marking allocation tags for such sheaves as empty. The problem was technically introduced in commit 4c0a17e28340 but only becomes possible to hit with commit 913ffd3a1bf5. Fixes: 4c0a17e28340 ("slab: prevent recursive kmalloc() in alloc_empty_sheaf()") Fixes: 913ffd3a1bf5 ("slab: handle kmalloc sheaves bootstrap") Reported-by: David Wang <00107082@163.com> Closes: https://lore.kernel.org/all/20260223155128.3849-1-00107082@163.com/ Analyzed-by: Harry Yoo Signed-off-by: Suren Baghdasaryan Reviewed-by: Harry Yoo Tested-by: Harry Yoo Tested-by: David Wang <00107082@163.com> Link: https://patch.msgid.link/20260225163407.2218712-1-surenb@google.com Signed-off-by: Vlastimil Babka (SUSE)

Merge branch 'slab/for-7.0/sheaves' into slab/for-next

2026-02-10T08:10:00+00:00

Merge series "slab: replace cpu (partial) slabs with sheaves". The percpu sheaves caching layer was introduced as opt-in but the goal was to eventually move all caches to them. This is the next step, enabling sheaves for all caches (except the two bootstrap ones) and then removing the per cpu (partial) slabs and lots of associated code. Besides the lower locking overhead and much more likely fastpath when freeing, this removes the rather complicated code related to the cpu slab lockless fastpaths (using this_cpu_try_cmpxchg128/64) and all its complications for PREEMPT_RT or kmalloc_nolock(). The lockless slab freelist+counters update operation using try_cmpxchg128/64 remains and is crucial for freeing remote NUMA objects and to allow flushing objects from sheaves to slabs mostly without the node list_lock. Link: https://lore.kernel.org/all/20260123-sheaves-for-all-v4-0-041323d506f7@suse.cz/

mm/slab: only allow SLAB_OBJ_EXT_IN_OBJ for unmergeable caches

2026-02-04T09:05:36+00:00

While SLAB_OBJ_EXT_IN_OBJ allows to reduce memory overhead to account slab objects, it prevents slab merging because merging can change the metadata layout. As pointed out Vlastimil Babka, disabling merging solely for this memory optimization may not be a net win, because disabling slab merging tends to increase overall memory usage. Restrict SLAB_OBJ_EXT_IN_OBJ to caches that are already unmergeable for other reasons (e.g., those with constructors or SLAB_TYPESAFE_BY_RCU). Suggested-by: Vlastimil Babka Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260127103151.21883-3-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

mm/slab: move [__]ksize and slab_ksize() to mm/slub.c

2026-02-04T09:05:35+00:00

To access SLUB's internal implementation details beyond cache flags in ksize(), move __ksize(), ksize(), and slab_ksize() to mm/slub.c. [vbabka@suse.cz: also make __ksize() static and move its kerneldoc to ksize() ] Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260113061845.159790-9-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

mm/memcontrol,alloc_tag: handle slabobj_ext access under KASAN poison

2026-02-04T09:05:35+00:00

In the near future, slabobj_ext may reside outside the allocated slab object range within a slab, which could be reported as an out-of-bounds access by KASAN. As suggested by Andrey Konovalov [1], explicitly disable KASAN and KMSAN checks when accessing slabobj_ext within slab allocator, memory profiling, and memory cgroup code. While an alternative approach could be to unpoison slabobj_ext, out-of-bounds accesses outside the slab allocator are generally more common. Move metadata_access_enable()/disable() helpers to mm/slab.h so that it can be used outside mm/slub.c. However, as suggested by Suren Baghdasaryan [2], instead of calling them directly from mm code (which is more prone to errors), change users to access slabobj_ext via get/put APIs: - Users should call get_slab_obj_exts() to access slabobj_metadata and call put_slab_obj_exts() when it's done. - From now on, accessing it outside the section covered by get_slab_obj_exts() ~ put_slab_obj_exts() is illegal. This ensures that accesses to slabobj_ext metadata won't be reported as access violations. Call kasan_reset_tag() in slab_obj_ext() before returning the address to prevent SW or HW tag-based KASAN from reporting false positives. Suggested-by: Andrey Konovalov Suggested-by: Suren Baghdasaryan Link: https://lore.kernel.org/linux-mm/CA+fCnZezoWn40BaS3cgmCeLwjT+5AndzcQLc=wH3BjMCu6_YCw@mail.gmail.com [1] Link: https://lore.kernel.org/linux-mm/CAJuCfpG=Lb4WhYuPkSpdNO4Ehtjm1YcEEK0OM=3g9i=LxmpHSQ@mail.gmail.com [2] Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260113061845.159790-7-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

mm/slab: use stride to access slabobj_ext

2026-02-04T09:05:35+00:00

Use a configurable stride value when accessing slab object extension metadata instead of assuming a fixed sizeof(struct slabobj_ext). Store stride value in free bits of slab->counters field. This allows for flexibility in cases where the extension is embedded within slab objects. Since these free bits exist only on 64-bit, any future optimizations that need to change stride value cannot be enabled on 32-bit architectures. Suggested-by: Vlastimil Babka Reviewed-by: Suren Baghdasaryan Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260113061845.159790-6-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

mm/slab: abstract slabobj_ext access via new slab_obj_ext() helper

2026-02-04T09:05:35+00:00

Currently, the slab allocator assumes that slab->obj_exts is a pointer to an array of struct slabobj_ext objects. However, to support storage methods where struct slabobj_ext is embedded within objects, the slab allocator should not make this assumption. Instead of directly dereferencing the slabobj_exts array, abstract access to struct slabobj_ext via helper functions. Introduce a new API slabobj_ext metadata access: slab_obj_ext(slab, obj_exts, index) - returns the pointer to struct slabobj_ext element at the given index. Directly dereferencing the return value of slab_obj_exts() is no longer allowed. Instead, slab_obj_ext() must always be used to access individual struct slabobj_ext objects. Convert all users to use these APIs. No functional changes intended. Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260113061845.159790-5-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka

slab: remove struct kmem_cache_cpu

2026-01-29T08:29:27+00:00

The cpu slab is not used anymore for allocation or freeing, the remaining code is for flushing, but it's effectively dead. Remove the whole struct kmem_cache_cpu, the flushing code and other orphaned functions. The remaining used field of kmem_cache_cpu is the stat array with CONFIG_SLUB_STATS. Put it instead in a new struct kmem_cache_stats. In struct kmem_cache, the field is cpu_stats and placed near the end of the struct. Reviewed-by: Hao Li Reviewed-by: Suren Baghdasaryan Reviewed-by: Harry Yoo Signed-off-by: Vlastimil Babka

slab: simplify kmalloc_nolock()

2026-01-29T08:29:27+00:00

The kmalloc_nolock() implementation has several complications and restrictions due to SLUB's cpu slab locking, lockless fastpath and PREEMPT_RT differences. With cpu slab usage removed, we can simplify things: - relax the PREEMPT_RT context checks as they were before commit 99a3e3a1cfc9 ("slab: fix kmalloc_nolock() context check for PREEMPT_RT") and also reference the explanation comment in the page allocator - the local_lock_cpu_slab() macros became unused, remove them - we no longer need to set up lockdep classes on PREEMPT_RT - we no longer need to annotate ___slab_alloc as NOKPROBE_SYMBOL since there's no lockless cpu freelist manipulation anymore - __slab_alloc_node() can be called from kmalloc_nolock_noprof() unconditionally. It can also no longer return EBUSY. But trylock failures can still happen so retry with the larger bucket if the allocation fails for any reason. Note that we still need __CMPXCHG_DOUBLE, because while it was removed we don't use cmpxchg16b on cpu freelist anymore, we still use it on slab freelist, and the alternative is slab_lock() which can be interrupted by a nmi. Clarify the comment to mention it specifically. Acked-by: Alexei Starovoitov Reviewed-by: Hao Li Reviewed-by: Suren Baghdasaryan Reviewed-by: Harry Yoo Signed-off-by: Vlastimil Babka