kernel/linux.git/mm/slub.c, branch v7.2-rc6

mm/slab: prevent unbounded recursion in free path with new kmalloc type

2026-07-14T15:22:40+00:00

Commit 280ea9c3154b ("mm/slab: avoid allocating slabobj_ext array from its own slab") avoided recursive allocation of obj_exts from kmalloc caches of the same size, by bumping the obj_exts array's allocation size whenever the array size equals the size of the object being allocated. However, as reported by Danielle Costantino and Shakeel Butt, even slabs from kmalloc caches of different sizes can form a cycle by allocating obj_exts arrays from each other [1]: What happened: a KMALLOC_NORMAL slab's obj_exts array (used by allocation profiling / memcg accounting) is itself kmalloc()'d from a KMALLOC_NORMAL cache, so the "slab holds another slab's obj_exts array" relation can form cycles. With sizeof(struct slabobj_ext) == 16 and the host's geometry: - kmalloc-512 has 64 objects/slab -> array is 64*16 == 1024 bytes, served from kmalloc-1k; - kmalloc-1k has 32 objects/slab -> array is 32*16 == 512 bytes, served from kmalloc-512. A kmalloc-512 slab and a kmalloc-1k slab therefore hold each other's obj_exts array. Discarding one frees the other's array, which empties and discards that slab, which frees the first's array, and so on: __free_slab() -> free_slab_obj_exts() -> kfree() -> discard_slab() -> __free_slab() recurses along the cycle until the stack is exhausted. With memory allocation profiling, this allows unbounded recursion in the free path and led to a stack overflow on a production host in the Meta fleet [1]: BUG: TASK stack guard page was hit Oops: stack guard page RIP: 0010:kfree+0x8/0x5d0 Call Trace: __free_slab+0x66/0xc0 kfree+0x3f0/0x5d0 ... ( ~125x __free_slab <-> kfree ) ... do_syscall_64 It is proposed [1] to resolve this issue by always serving the obj_exts array allocation from kmalloc caches (or large kmalloc) of sizes larger than the object size. However, as pointed out by Vlastimil Babka [2], this can waste an excessive amount of memory as slabs from large kmalloc sizes (e.g. kmalloc-8k) generally need obj_exts arrays much smaller than the object size. Therefore, rather than bumping the size, let us take a different approach; disallow formation of cycles between kmalloc types when allocating obj_exts arrays. Currently, all obj_exts arrays are served from normal kmalloc caches. Cycles cannot be created if obj_exts arrays of normal kmalloc caches are served from a special kmalloc type that can never have obj_exts arrays. To achieve this, create a new kmalloc type called KMALLOC_NO_OBJ_EXT. KMALLOC_NO_OBJ_EXT caches are created with SLAB_NO_OBJ_EXT flag when either 1) memory allocation profiling is not permanently disabled, or 2) kmalloc types with a priority higher than KMALLOC_CGROUP are aliased with KMALLOC_NORMAL. Sheaf bootstrapping for KMALLOC_NO_OBJ_EXT caches now must be deferred because allocation of a barn can trigger obj_exts array allocation of normal kmalloc caches when the KMALLOC_NO_OBJ_EXT cache for that size is not ready yet. For simplicity, perform bootstrapping of sheaves for all kmalloc caches later. Introduce a new slab alloc flag, SLAB_ALLOC_NO_OBJ_EXT, to prevent allocation of obj_exts arrays, and let kmalloc_slab() override the type to KMALLOC_NO_OBJ_EXT when specified. Note that kmalloc_type() remains unchanged because kmalloc_flags() bypasses the kmalloc fastpath. Do not pass SLAB_ALLOC_NO_RECURSE to kmalloc_flags() in alloc_slab_obj_exts() and instead use SLAB_ALLOC_NO_OBJ_EXT only when the objects are allocated from normal kmalloc caches. While this prevents unbounded recursive allocation of obj_exts, it allows KMALLOC_NO_OBJ_EXT caches to have sheaves. Since sheaf allocations specify SLAB_ALLOC_NO_RECURSE that prevents allocation of both sheaves and obj_exts arrays, the recursion depth is bounded. obj_exts arrays for non-kmalloc-normal caches can now have a valid tag. Do not call mark_obj_codetag_empty() when freeing an obj_exts array to avoid false warnings. KMALLOC_NO_OBJ_EXT don't need this as they never allocate those arrays. Reported-by: Danielle Costantino Reported-by: Shakeel Butt Closes: https://lore.kernel.org/linux-mm/20260625230029.703750-1-shakeel.butt@linux.dev [1] Fixes: 4b8736964640 ("mm/slab: add allocation accounting into slab allocation and free paths") Cc: stable@vger.kernel.org Link: https://lore.kernel.org/linux-mm/c5c4208d-a6f0-413e-bad9-49be12f12d55@kernel.org [2] Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Suren Baghdasaryan Link: https://patch.msgid.link/20260713-kmalloc-no-objext-v3-4-47c7bd138de7@kernel.org Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: decouple SLAB_NO_SHEAVES from SLAB_NO_OBJ_EXT

2026-07-14T14:43:34+00:00

Bootstrap caches are created with SLAB_NO_OBJ_EXT to disallow sheaves and obj_exts. To allow disabling obj_exts while allowing sheaves, decouple SLAB_NO_SHEAVES from SLAB_NO_OBJ_EXT. Bootstrap caches now have both SLAB_NO_SHEAVES and SLAB_NO_OBJ_EXT. No functional change intended. Reviewed-by: Vlastimil Babka (SUSE) Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Suren Baghdasaryan Link: https://patch.msgid.link/20260713-kmalloc-no-objext-v3-2-47c7bd138de7@kernel.org Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: fix a memory leak due to bootstrapping sheaves twice

2026-07-14T14:40:07+00:00

When kmalloc caches are aliased, multiple cache pointers reference the same kmem_cache. As a result, iterating over kmalloc indices and bootstrapping sheaves can bootstrap the same cache more than once and leak memory. Currently, this could happen when the architecture specifies minimum alignment for slab caches that is larger than ARCH_KMALLOC_MINALIGN. Bootstrap sheaves only when the cache does not have them already. Add a warning when bootstrap_cache_sheaves() is called for a cache that already has sheaves enabled. Fixes: 913ffd3a1bf5 ("slab: handle kmalloc sheaves bootstrap") Cc: stable@vger.kernel.org Signed-off-by: Harry Yoo (Oracle) Reviewed-by: Suren Baghdasaryan Link: https://patch.msgid.link/20260713-kmalloc-no-objext-v3-1-47c7bd138de7@kernel.org Signed-off-by: Vlastimil Babka (SUSE)

mm/slub: fix lost local objects when bulk remote free batch fills

2026-07-08T16:14:56+00:00

In free_to_pcs_bulk(), when remote_objects[] fills to PCS_BATCH_MAX, the code jumps to flush_remote to free the batch. If all remote entries have already been compacted out of p[] via tail swaps while local objects remain, the flush_remote path returns early since `i < size` no longer holds. The leftover local objects are then neither cached in the sheaf nor returned to the slab freelist, causing a memory leak. For illustration: size = 64, local objects at p[0..31], remote objects at p[32..63] After scanning all remotes: i = 32, size = 32 p[0..31] local objects are dropped. Harry pointed out that, although the logic contains a real leak, it does not appear to be triggerable with the current in-tree users. To hit this path, at least PCS_BATCH_MAX objects, currently hardcoded to 32, need to be collected in remote_objects[]. Looking at current kmem_cache_free_bulk() users: * maple_node has sheaf_capacity = 32 * skbuff_head_cache has sheaf_capacity = 28 * panthor and msm drivers have sheaf_capacity = 4 The sheaf capacity is, at least for now, derived purely from the object size, with the user-requested capacity used as a minimum. Therefore, among the current users, only maple_node has a sheaf_capacity large enough to reach PCS_BATCH_MAX. However, for the bug to trigger in maple_node, all objects in the sheaf would have to be from remote nodes. In that case, there would be no local objects left to leak. So this issue was found by code review rather than from a runtime report, and it does not seem to be triggerable by current users. Still, the bug could become reachable with future users, a different sheaf capacity, or a change to PCS_BATCH_MAX. Fix the logic by freeing a full remote batch in place during the scan and then continuing to process the compacted array. This keeps all local objects on the normal fast path, while the tail path only handles any leftover partial remote batch. The redundant next_remote_batch jump label is removed as well. Fixes: 989b09b73978 ("slab: skip percpu sheaves for remote object freeing") Signed-off-by: Shengming Hu Link: https://patch.msgid.link/202607062139095043SOsLi6TIf403tcjPf8fm@zte.com.cn Cc: stable@vger.kernel.org Reviewed-by: Harry Yoo (Oracle) Signed-off-by: Vlastimil Babka (SUSE)

Merge tag 'slab-for-7.2-part2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab

2026-06-22T15:28:48+00:00

Pull more slab updates from Vlastimil Babka: - Introduce and wire up a new alloc_flags parameter for modifying slab-specific behavior without adding or reusing gfp flags. Also introduce slab_alloc_context to keep function parameter bloat in check. Both are similar to what the page allocator does. kmalloc_flags() exposes alloc_flags for mm-internal users. - SLAB_ALLOC_NOLOCK flag is used to implement kmalloc_nolock() behavior without relying on lack of __GFP_RECLAIM, which caused false positives with workarounds like fd3634312a04 ("debugobject: Make it work with deferred page initialization - again"). - SLAB_ALLOC_NO_RECURSE replaces __GFP_NO_OBJ_EXT, which could have been removed, but pending memory allocation profiling changes in mm tree have grown a new user - there is however a work ongoing to replace that too, so __GFP_NO_OBJ_EXT should eventually be removed. (Vlastimil Babka) - Add kmem_buckets_alloc_track_caller() with a user to be added in the net tree (Pedro Falcato) - Fixes for kernel-doc and slabinfo (Randy Dunlap, Yichong Chen) * tag 'slab-for-7.2-part2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab: tools/mm/slabinfo: fix total_objects attribute name slab: recognize @GFP parameter as optional in kernel-doc mm/slab: add a node-track-caller variant for kmem buckets allocation mm/slab: replace __GFP_NO_OBJ_EXT with SLAB_ALLOC_NO_RECURSE for sheaves mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts() mm/slab: introduce kmalloc_flags() mm/slab: allow __GFP_NOMEMALLOC and __GFP_NOWARN for kmalloc_nolock() mm/slab: pass slab_alloc_context to __do_kmalloc_node() mm/slab: allow kmem_cache_alloc_bulk() with any gfp flags mm/slab: replace slab_alloc_node() parameters with slab_alloc_context mm/slab: pass alloc_flags through slab_post_alloc_hook() chain mm/slab: pass alloc_flags to new slab allocation mm/slab: add alloc_flags to slab_alloc_context mm/slab: replace struct partial_context with slab_alloc_context mm/slab: introduce alloc_flags and SLAB_ALLOC_NOLOCK mm/slab: introduce slab_alloc_context mm/slab: stop inlining __slab_alloc_node() mm/slab: do not init any kfence objects on allocation

mm/slab: replace __GFP_NO_OBJ_EXT with SLAB_ALLOC_NO_RECURSE for sheaves

2026-06-18T08:40:53+00:00

Finish the switch away from __GFP_NO_OBJ_EXT by replacing it with SLAB_ALLOC_NO_RECURSE when allocating empty sheaves. Pass alloc_flags to [__]alloc_empty_sheaf(). Callers that can't be part of a recursive kmalloc() chain simply pass SLAB_ALLOC_DEFAULT. Use kmalloc_flags() instead of kzalloc() for allocating the sheaf. With that we can finalize the removal the __GFP_NO_OBJ_EXT handling from obj_ext allocations as well, leaving only SLAB_ALLOC_NO_RECURSE in place. This leaves __GFP_NO_OBJ_EXT with no users in slab, so stop allowing the flag in kmalloc_nolock(). Link: https://patch.msgid.link/20260610-slab_alloc_flags-v2-16-7190909db118@kernel.org Reviewed-by: Hao Li Reviewed-by: Harry Yoo (Oracle) Reviewed-by: Suren Baghdasaryan Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: remove __GFP_NO_OBJ_EXT usage from alloc_slab_obj_exts()

2026-06-18T08:40:32+00:00

__GFP_NO_OBJ_EXT has limited scope within the slab allocator itself and gfp flags are a scarce resource, unlike slab's alloc_flags. Introduce SLAB_ALLOC_NO_RECURSE alloc flag that has the same intent as __GFP_NO_OBJ_EXT but a more generic name, meaning that a kmalloc() family function should not recurse into another kmalloc*() for the purposes of allocating auxiliary structures (obj_ext arrays or sheaves). First, replace the __GFP_NO_OBJ_EXT for allocating obj_ext arrays in alloc_slab_obj_exts(). Make use of the newly added kmalloc_flags() function, where we can pass alloc_flags with SLAB_ALLOC_NO_RECURSE added. This will also pass through SLAB_ALLOC_NOLOCK so we don't need to special case kmalloc_nolock() anymore. Note that until now the kmalloc_nolock() ignored the incoming gfp flags and hardcoded __GFP_ZERO | __GFP_NO_OBJ_EXT. But it's correct to pass on the incoming gfp flags (only augmented with __GFP_ZERO), because if alloc_flags contain SLAB_ALLOC_NOLOCK, the incoming gfp flags have to be also compatible with it. However, we might have added __GFP_THISNODE for opportunistic slab allocation, as pointed out by Hao Li, and __GFP_COMP by allocate_slab() as pointed out by Shengming Hu. Solve this by adding both flags to OBJCGS_CLEAR_MASK as it makes sense to strip them anyway for non-kmalloc_nolock() allocations of sheaves or obj_ext arrays as well. To avoid recursion of sheaf -> obj_ext -> sheaf -> ... allocations at this patch, until the next patch converts sheaves to SLAB_ALLOC_NO_RECURSE, use both gfp and alloc_flags for obj_ext. The next patch will remove the gfp part. Link: https://patch.msgid.link/20260610-slab_alloc_flags-v2-15-7190909db118@kernel.org Reviewed-by: Harry Yoo (Oracle) Reviewed-by: Suren Baghdasaryan Reviewed-by: Hao Li Signed-off-by: Vlastimil Babka (SUSE)

Merge tag 'slab-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab

2026-06-16T03:14:43+00:00

Pull slab updates from Vlastimil Babka: - Support for "allocation tokens" (currently available in Clang 22+) for smarter partitioning of kmalloc caches based on the allocated object type, which can be enabled instead of the "random" per-caller-address-hash partitioning. It should be able to deterministically separate types containing a pointer from those that do not (Marco Elver) - Improvements and simplification of the kmem_cache_alloc_bulk() and mempool_alloc_bulk() API. This includes adaptation of callers (Christoph Hellwig) - Performance improvements and cleanups related mostly to sheaves refill (Hao Li, Shengming Hu, Vlastimil Babka) - Several fixups for the slabinfo tool (Xuewen Wang) * tag 'slab-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab: mm/slab: do not limit zeroing to orig_size when only red zoning is enabled mm/slub: preserve original size in _kmalloc_nolock_noprof retry path mm: simplify the mempool_alloc_bulk API mm/slab: improve kmem_cache_alloc_bulk mm/slub: detach and reattach partial slabs in batch mm/slub: introduce helpers for node partial slab state mm/slub: use empty sheaf helpers for oversized sheaves tools/mm/slabinfo: remove redundant slab->partial assignment tools/mm/slabinfo: remove dead assignment in get_obj_and_str() tools/mm/slabinfo: Fix trace disable logic inversion MAINTAINERS: add slab-related scripts and tools to SLAB ALLOCATOR mm/slub: fix typo in sheaves comment mm, slab: simplify returning slab in __refill_objects_node() mm, slab: add an optimistic __slab_try_return_freelist() slab: fix kernel-docs for mm-api slab: improve KMALLOC_PARTITION_RANDOM randomness slab: support for compiler-assisted type-based slab cache partitioning mm/slub: defer freelist construction until after bulk allocation from a new slab

mm/slab: introduce kmalloc_flags()

2026-06-15T11:29:23+00:00

With alloc_flags usage in slab, we can replace __GFP_NO_OBJ_EXT with an alloc flag that prevents kmalloc recursion. For that we need a version of kmalloc() that takes alloc_flags and use it in places that perform these potentially recursive kmalloc allocations (of sheaves or obj_ext arrays). Add this function, named kmalloc_flags(). Right now it's only useful for these nested allocations, so it doesn't need to optimize build-time constant sizes like kmalloc() or kmalloc_buckets. Since we need it to support both normal and non-spinning kmalloc_nolock() context through the SLAB_ALLOC_NOLOCK flag, split out most of the special _kmalloc_nolock_noprof() implementation to __kmalloc_nolock_noprof() that takes a slab_alloc_context, and make _kmalloc_nolock_noprof() a simple tail calling wrapper with the proper context. kmalloc_flags() can thus determine whether to call __kmalloc_nolock_noprof() or __do_kmalloc_node(), based on the given alloc_flags. Link: https://patch.msgid.link/20260610-slab_alloc_flags-v2-14-7190909db118@kernel.org Reviewed-by: Hao Li Reviewed-by: Suren Baghdasaryan Reviewed-by: Harry Yoo (Oracle) Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: allow __GFP_NOMEMALLOC and __GFP_NOWARN for kmalloc_nolock()

2026-06-15T11:26:29+00:00

The two flags are added internally so there's no point for warning if they are passed by the caller as well, so allow them. This will allow simplifying obj_ext allocation under kmalloc_nolock(). Also it's not necessary to have the extra alloc_gfp variable for adding the two flags. The original gfp_flags parameter is not used anywhere except for the warning. So remove alloc_gfp and directly modify and use gfp_flags everywhere. Link: https://patch.msgid.link/20260610-slab_alloc_flags-v2-13-7190909db118@kernel.org Reviewed-by: Hao Li Reviewed-by: Suren Baghdasaryan Reviewed-by: Harry Yoo (Oracle) Signed-off-by: Vlastimil Babka (SUSE)