kernel/linux.git/include/linux/slab.h, branch v7.3-rc1

Merge branch 'slab/for-7.3/kfree_rcu_nolock' into slab/for-next

2026-08-24T13:01:27+00:00

Merge series "mm/slab: introduce kfree_rcu_nolock() and improve slub_kunit coverage" from Harry Yoo. From the cover letter [1]: This series improves kmalloc_nolock() and kfree_nolock() coverage in slub_kunit and introduces kfree_rcu_nolock() for unknown context as suggested by Alexei Starovoitov. Unknown context means the caller does not know whether spinning on a lock is safe (e.g., a BPF program attached to an arbitrary kernel function or in NMI context). The slab allocator already supports unknown context via kmalloc_nolock() and kfree_nolock(), but te slab allocator does not support freeing objects by RCU in unknown context. It is not ideal to have completely separate batching for unknown context because the worst scenario where spinning on a lock would lead to deadlock is very rare, and in most cases, it is safe to use the existing mechanism (kfree_rcu_sheaf()). Since most part of the slab allocator already supports unknown context and sheaves support batching kvfree_rcu() calls for slab objects, implement kfree_rcu_nolock() with minimal changes by teaching kfree_rcu_sheaf() how to support unknown context and making it a little bit harder to allocate an empty sheaf, instead of making intrusive changes to the existing kvfree_rcu batching logic. kfree_rcu_nolock() tries to free the object to the rcu sheaf if trylock succeeds. Once the rcu sheaf becomes full, it is submitted to RCU via call_rcu() if spinning is allowed or IRQs are enabled (to avoid calling call_rcu() in the middle of call_rcu()). Otherwise, call_rcu() is deferred via irq work. When there is no sheaf available, kfree_rcu_sheaf() falls back to defer_kfree_rcu(). It submits the object to kvfree_rcu batching via irq work. To do this, patch 6 converts kvfree_rcu to use kvfree_rcu_head without visible changes to the API for now. Unlike kfree_rcu(), only the 2-argument variant is supported. This is because the last resort of the 1-arg variant is synchronize_rcu(), which cannot be used in an unknown context. As suggested by Alexei Starovoitov, kfree_rcu_nolock() can be used with struct kvfree_rcu_head (8 bytes), which is smaller than struct rcu_head (16 bytes). Link: https://lore.kernel.org/all/20260729-kfree_rcu_nolock-v5-0-a28cdcda9673@kernel.org/ [1]

mm/slab: add cache_ and slab_needs_objcg() helpers

2026-08-04T10:23:22+00:00

Slabs of some caches never need the objcg part of struct slabobj_ext. Introduce helpers to query this for a cache or a slab. Introduce SLAB_MAY_ACCOUNT flag that is currently only internal and all caches have it set except: - KMALLOC_NORMAL caches, as long as KMALLOC_RECLAIM caches are separate - KMALLOC_NO_OBJ_EXT caches, if they exist For named caches we currently can't derive SLAB_MAY_ACCOUNT from SLAB_ACCOUNT because some caches might be created without SLAB_ACCOUNT and then used both with and without __GFP_ACCOUNT concurrently, allocating obj_ext arrays on demand. So just add the SLAB_MAY_ACCOUNT to all kmem caches, unless kmem accounting is disabled. This can be improved later by finding out all caches used with __GFP_ACCOUNT, creating them with the SLAB_MAY_ACCOUNT flag explicitly, and then ignoring __GFP_ACCOUNT for all other caches (possibly with a warning). To make the evaluation of slab_needs_objcg() faster in the allocation and free fast paths, add a obj_exts_needs_objcg flag into slab itself. This optimization is only available on 64bit architectures where free bits are available for the flag. Reviewed-by: Hao Li Link: https://patch.msgid.link/20260727-b4-objext_split-v3-11-c29ef0f1f257@kernel.org Reviewed-by: Harry Yoo Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: introduce kfree_rcu_nolock()

2026-08-04T10:18:29+00:00

Currently, k[v]free_rcu() cannot be called in unknown context since it could lead to a deadlock when called in the middle of k[v]free_rcu(). Make users' lives easier by introducing kfree_rcu_nolock() variant, now that kfree_rcu_sheaf() is available on PREEMPT_RT and __kfree_rcu_sheaf() handles unknown context. When sheaves path fails, kfree_rcu_nolock() falls back to defer_kfree_rcu() that uses an irq work to free the object via kvfree_call_rcu(). In most cases, the sheaves path is expected to succeed and therefore it's unnecessary to introduce additional complexity to the existing kvfree_rcu batching by teaching it how to handle unknown context. Since defer_kfree_rcu() can be called on caches without sheaves, move deferred_work_barrier() and rcu_barrier() outside the branch in kvfree_rcu_barrier_on_cache(). Now that deferred kvfree_rcu objects are submitted to kvfree_call_rcu() after deferred_work_barrier() and may end up in RCU sheaves, deferred_work_barrier() must be invoked before flush_rcu_sheaves_on_cache(). Since the RCU sheaf path has not been used on !KVFREE_RCU_BATCHED kernels, always fall back when kvfree_rcu() is not batched, for consistency. kvfree_rcu_barrier{,_on_cache()}() on !KVFREE_RCU_BATCHED are moved to mm/slab_common.c to invoke deferred_work_barrier() before rcu_barrier(). Signed-off-by: Harry Yoo (Oracle) Link: https://patch.msgid.link/20260729-kfree_rcu_nolock-v5-7-a28cdcda9673@kernel.org Signed-off-by: Vlastimil Babka (SUSE)

slab: silence sparse warning with type-based partitioning

2026-07-21T09:28:21+00:00

Sparse does not know __builtin_infer_alloc_token() and complains: sparse: sparse: undefined identifier '__builtin_infer_alloc_token' Fix it by using a dummy variant of __kmalloc_token() if __CHECKER__ is defined. Fixes: feb662d9168b ("slab: support for compiler-assisted type-based slab cache partitioning") Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202607110912.nZTqfCrH-lkp@intel.com/ Signed-off-by: Marco Elver Link: https://patch.msgid.link/20260721092005.1986693-1-elver@google.com Acked-by: Harry Yoo (Oracle) Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: prevent unbounded recursion in free path with new kmalloc type

2026-07-14T15:22:40+00:00

Commit 280ea9c3154b ("mm/slab: avoid allocating slabobj_ext array from its own slab") avoided recursive allocation of obj_exts from kmalloc caches of the same size, by bumping the obj_exts array's allocation size whenever the array size equals the size of the object being allocated. However, as reported by Danielle Costantino and Shakeel Butt, even slabs from kmalloc caches of different sizes can form a cycle by allocating obj_exts arrays from each other [1]: What happened: a KMALLOC_NORMAL slab's obj_exts array (used by allocation profiling / memcg accounting) is itself kmalloc()'d from a KMALLOC_NORMAL cache, so the "slab holds another slab's obj_exts array" relation can form cycles. With sizeof(struct slabobj_ext) == 16 and the host's geometry: - kmalloc-512 has 64 objects/slab -> array is 64*16 == 1024 bytes, served from kmalloc-1k; - kmalloc-1k has 32 objects/slab -> array is 32*16 == 512 bytes, served from kmalloc-512. A kmalloc-512 slab and a kmalloc-1k slab therefore hold each other's obj_exts array. Discarding one frees the other's array, which empties and discards that slab, which frees the first's array, and so on: __free_slab() -> free_slab_obj_exts() -> kfree() -> discard_slab() -> __free_slab() recurses along the cycle until the stack is exhausted. With memory allocation profiling, this allows unbounded recursion in the free path and led to a stack overflow on a production host in the Meta fleet [1]: BUG: TASK stack guard page was hit Oops: stack guard page RIP: 0010:kfree+0x8/0x5d0 Call Trace: __free_slab+0x66/0xc0 kfree+0x3f0/0x5d0 ... ( ~125x __free_slab <-> kfree ) ... do_syscall_64 It is proposed [1] to resolve this issue by always serving the obj_exts array allocation from kmalloc caches (or large kmalloc) of sizes larger than the object size. However, as pointed out by Vlastimil Babka [2], this can waste an excessive amount of memory as slabs from large kmalloc sizes (e.g. kmalloc-8k) generally need obj_exts arrays much smaller than the object size. Therefore, rather than bumping the size, let us take a different approach; disallow formation of cycles between kmalloc types when allocating obj_exts arrays. Currently, all obj_exts arrays are served from normal kmalloc caches. Cycles cannot be created if obj_exts arrays of normal kmalloc caches are served from a special kmalloc type that can never have obj_exts arrays. To achieve this, create a new kmalloc type called KMALLOC_NO_OBJ_EXT. KMALLOC_NO_OBJ_EXT caches are created with SLAB_NO_OBJ_EXT flag when either 1) memory allocation profiling is not permanently disabled, or 2) kmalloc types with a priority higher than KMALLOC_CGROUP are aliased with KMALLOC_NORMAL. Sheaf bootstrapping for KMALLOC_NO_OBJ_EXT caches now must be deferred because allocation of a barn can trigger obj_exts array allocation of normal kmalloc caches when the KMALLOC_NO_OBJ_EXT cache for that size is not ready yet. For simplicity, perform bootstrapping of sheaves for all kmalloc caches later. Introduce a new slab alloc flag, SLAB_ALLOC_NO_OBJ_EXT, to prevent allocation of obj_exts arrays, and let kmalloc_slab() override the type to KMALLOC_NO_OBJ_EXT when specified. Note that kmalloc_type() remains unchanged because kmalloc_flags() bypasses the kmalloc fastpath. Do not pass SLAB_ALLOC_NO_RECURSE to kmalloc_flags() in alloc_slab_obj_exts() and instead use SLAB_ALLOC_NO_OBJ_EXT only when the objects are allocated from normal kmalloc caches. While this prevents unbounded recursive allocation of obj_exts, it allows KMALLOC_NO_OBJ_EXT caches to have sheaves. Since sheaf allocations specify SLAB_ALLOC_NO_RECURSE that prevents allocation of both sheaves and obj_exts arrays, the recursion depth is bounded. obj_exts arrays for non-kmalloc-normal caches can now have a valid tag. Do not call mark_obj_codetag_empty() when freeing an obj_exts array to avoid false warnings. KMALLOC_NO_OBJ_EXT don't need this as they never allocate those arrays. Reported-by: Danielle Costantino Reported-by: Shakeel Butt Closes: https://lore.kernel.org/linux-mm/20260625230029.703750-1-shakeel.butt@linux.dev [1] Fixes: 4b8736964640 ("mm/slab: add allocation accounting into slab allocation and free paths") Cc: stable@vger.kernel.org Link: https://lore.kernel.org/linux-mm/c5c4208d-a6f0-413e-bad9-49be12f12d55@kernel.org [2] Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Suren Baghdasaryan Link: https://patch.msgid.link/20260713-kmalloc-no-objext-v3-4-47c7bd138de7@kernel.org Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: decouple SLAB_NO_SHEAVES from SLAB_NO_OBJ_EXT

2026-07-14T14:43:34+00:00

Bootstrap caches are created with SLAB_NO_OBJ_EXT to disallow sheaves and obj_exts. To allow disabling obj_exts while allowing sheaves, decouple SLAB_NO_SHEAVES from SLAB_NO_OBJ_EXT. Bootstrap caches now have both SLAB_NO_SHEAVES and SLAB_NO_OBJ_EXT. No functional change intended. Reviewed-by: Vlastimil Babka (SUSE) Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Suren Baghdasaryan Link: https://patch.msgid.link/20260713-kmalloc-no-objext-v3-2-47c7bd138de7@kernel.org Signed-off-by: Vlastimil Babka (SUSE)

Merge tag 'slab-for-7.2-part2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab

2026-06-22T15:28:48+00:00

Pull more slab updates from Vlastimil Babka: - Introduce and wire up a new alloc_flags parameter for modifying slab-specific behavior without adding or reusing gfp flags. Also introduce slab_alloc_context to keep function parameter bloat in check. Both are similar to what the page allocator does. kmalloc_flags() exposes alloc_flags for mm-internal users. - SLAB_ALLOC_NOLOCK flag is used to implement kmalloc_nolock() behavior without relying on lack of __GFP_RECLAIM, which caused false positives with workarounds like fd3634312a04 ("debugobject: Make it work with deferred page initialization - again"). - SLAB_ALLOC_NO_RECURSE replaces __GFP_NO_OBJ_EXT, which could have been removed, but pending memory allocation profiling changes in mm tree have grown a new user - there is however a work ongoing to replace that too, so __GFP_NO_OBJ_EXT should eventually be removed. (Vlastimil Babka) - Add kmem_buckets_alloc_track_caller() with a user to be added in the net tree (Pedro Falcato) - Fixes for kernel-doc and slabinfo (Randy Dunlap, Yichong Chen) * tag 'slab-for-7.2-part2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab: tools/mm/slabinfo: fix total_objects attribute name slab: recognize @GFP parameter as optional in kernel-doc mm/slab: add a node-track-caller variant for kmem buckets allocation mm/slab: replace __GFP_NO_OBJ_EXT with SLAB_ALLOC_NO_RECURSE for sheaves mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts() mm/slab: introduce kmalloc_flags() mm/slab: allow __GFP_NOMEMALLOC and __GFP_NOWARN for kmalloc_nolock() mm/slab: pass slab_alloc_context to __do_kmalloc_node() mm/slab: allow kmem_cache_alloc_bulk() with any gfp flags mm/slab: replace slab_alloc_node() parameters with slab_alloc_context mm/slab: pass alloc_flags through slab_post_alloc_hook() chain mm/slab: pass alloc_flags to new slab allocation mm/slab: add alloc_flags to slab_alloc_context mm/slab: replace struct partial_context with slab_alloc_context mm/slab: introduce alloc_flags and SLAB_ALLOC_NOLOCK mm/slab: introduce slab_alloc_context mm/slab: stop inlining __slab_alloc_node() mm/slab: do not init any kfence objects on allocation

slab: recognize @GFP parameter as optional in kernel-doc

2026-06-18T08:40:53+00:00

Since the @GFP parameter in kmalloc_obj() etc. is now optional, change the kernel-doc to indicate that it is optional. This avoids kernel-doc warnings: WARNING: include/linux/slab.h:1101 Excess function parameter 'GFP' description in 'kmalloc_obj' WARNING: include/linux/slab.h:1113 Excess function parameter 'GFP' description in 'kmalloc_objs' WARNING: include/linux/slab.h:1128 Excess function parameter 'GFP' description in 'kmalloc_flex' Fixes: e19e1b480ac7 ("add default_gfp() helper macro and use it in the new *alloc_obj() helpers") Signed-off-by: Randy Dunlap Acked-by: Harry Yoo (Oracle) Link: https://patch.msgid.link/20260617163125.2716279-1-rdunlap@infradead.org Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: add a node-track-caller variant for kmem buckets allocation

2026-06-18T08:40:53+00:00

This is required by users that want to use kmem buckets, but still desire specifying the NUMA node. Reviewed-by: Kees Cook Acked-by: Vlastimil Babka (SUSE) Acked-by: Harry Yoo (Oracle) Signed-off-by: Pedro Falcato Link: https://patch.msgid.link/20260611124642.345400-2-pfalcato@suse.de Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: replace __GFP_NO_OBJ_EXT with SLAB_ALLOC_NO_RECURSE for sheaves

2026-06-18T08:40:53+00:00

Finish the switch away from __GFP_NO_OBJ_EXT by replacing it with SLAB_ALLOC_NO_RECURSE when allocating empty sheaves. Pass alloc_flags to [__]alloc_empty_sheaf(). Callers that can't be part of a recursive kmalloc() chain simply pass SLAB_ALLOC_DEFAULT. Use kmalloc_flags() instead of kzalloc() for allocating the sheaf. With that we can finalize the removal the __GFP_NO_OBJ_EXT handling from obj_ext allocations as well, leaving only SLAB_ALLOC_NO_RECURSE in place. This leaves __GFP_NO_OBJ_EXT with no users in slab, so stop allowing the flag in kmalloc_nolock(). Link: https://patch.msgid.link/20260610-slab_alloc_flags-v2-16-7190909db118@kernel.org Reviewed-by: Hao Li Reviewed-by: Harry Yoo (Oracle) Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE)