kernel/linux.git/mm/slub.c, branch v6.18.42

mm/slab: do not limit zeroing to orig_size when only red zoning is enabled

2026-07-18T14:53:21+00:00

commit 648927ceb84021a25a0fbd5673740956f318d534 upstream. When init (zeroing) on allocation is requested, for kmalloc() we generally have to zero the full object size even if a smaller size is requested, in order to provide krealloc()'s __GFP_ZERO guarantees. But if we track the requested size, krealloc() uses that information to do the right thing, so we can zero only the requested size. With red zoning also enabled, any extra size became part of the red zone, so it must not be zeroed and thus we must zero only the requested size. However the current check is imprecise, and will trigger also when only SLAB_RED_ZONE is enabled without SLAB_STORE_USER (which enables tracking the requested size). This means enabling red zoning alone can compromise krealloc()'s __GFP_ZERO contract. Fix this by using slub_debug_orig_size() instead, which is the exact check for whether the requested size is tracked. We don't need to care if red zoning is also enabled or not. Also update and expand the comment accordingly. Fixes: 9ce67395f5a0 ("mm/slub: only zero requested size of buffer for kzalloc when debug enabled") Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260610-slab_alloc_flags-v2-1-7190909db118@kernel.org Reviewed-by: Harry Yoo (Oracle) Reviewed-by: Hao Li Signed-off-by: Vlastimil Babka (SUSE) Signed-off-by: Greg Kroah-Hartman

mm/slub: hold cpus_read_lock around flush_rcu_sheaves_on_cache()

2026-06-09T10:28:52+00:00

[ Upstream commit 67ea9d353d0ba12bdbc9183ff568dead9e949b80 ] flush_rcu_sheaves_on_cache() calls queue_work_on() in a for_each_online_cpu() loop, which requires the cpu to stay online. But cpus_read_lock() is not held in kvfree_rcu_barrier_on_cache() and the set of "online cpus" is subject to change. There are two paths that call flush_rcu_sheaves_on_cache(): // has cpus_read_lock() flush_all_rcu_sheaves() -> flush_rcu_sheaves_on_cache() // no cpus_read_lock() kvfree_rcu_barrier_on_cache() -> flush_rcu_sheaves_on_cache() Fix this by holding cpus_read_lock() in kvfree_rcu_barrier_on_cache(). Why not move cpus_read_lock() from flush_all_rcu_sheaves() into flush_rcu_sheaves_on_cache()? The reason is it would introduce a new lock order (slab_mutex -> cpu_hotplug_lock). The reverse order (cpu_hotplug_lock -> slab_mutex) is established by - cpuhp_setup_state_nocalls(..., slub_cpu_setup, ...) - kmem_cache_destroy() The two orders together would form an AB-BA deadlock. Finally, add lockdep_assert_cpus_held() in flush_rcu_sheaves_on_cache() to catch the same problem in the future. Fixes: 0f35040de593 ("mm/slab: introduce kvfree_rcu_barrier_on_cache() for cache destruction") Cc: Signed-off-by: Qing Wang Link: https://patch.msgid.link/20260512035035.762317-1-wangqing7171@gmail.com Signed-off-by: Vlastimil Babka (SUSE) Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman

mm/slab: return NULL early from kmalloc_nolock() in NMI on UP

2026-05-07T04:12:02+00:00

commit 5b31044e649e3e54c2caef135c09b371c2fbcd08 upstream. On UP kernels (!CONFIG_SMP), spin_trylock() is a no-op that unconditionally succeeds even when the lock is already held. As a result, kmalloc_nolock() called from NMI context can re-enter the slab allocator and acquire n->list_lock that the interrupted context is already holding, corrupting slab state. With CONFIG_DEBUG_SPINLOCK on UP, the following BUG is triggered with the slub_kunit test module: BUG: spinlock trylock failure on UP on CPU#0, kunit_try_catch/243 [...] Call Trace: dump_stack_lvl+0x3f/0x60 do_raw_spin_trylock+0x41/0x50 _raw_spin_trylock+0x24/0x50 get_from_partial_node+0x120/0x4d0 ___slab_alloc+0x8a/0x4c0 kmalloc_nolock_noprof+0x164/0x310 [...] Fix this by returning NULL early when invoked from NMI on a UP kernel. Link: https://lore.kernel.org/linux-mm/ad_cqe51pvr1WaDg@hyeyoo Cc: stable@vger.kernel.org Fixes: af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().") Signed-off-by: Harry Yoo (Oracle) Link: https://patch.msgid.link/20260427-nolock-api-fix-v2-2-a6b83a92d9a4@kernel.org Signed-off-by: Vlastimil Babka (SUSE) Signed-off-by: Greg Kroah-Hartman

slub: fix data loss and overflow in krealloc()

2026-05-07T04:11:46+00:00

commit 082a6d03a2d685a83a332666b500ad3966349588 upstream. Commit 2cd8231796b5 ("mm/slub: allow to set node and align in k[v]realloc") introduced the ability to force a reallocation if the original object does not satisfy new alignment or NUMA node, even when the object is being shrunk. This introduced two bugs in the reallocation fallback path: 1. Data loss during NUMA migration: The jump to 'alloc_new' happens before 'ks' and 'orig_size' are initialized. As a result, the memcpy() in the 'alloc_new' block would copy 0 bytes into the new allocation. 2. Buffer overflow during shrinking: When shrinking an object while forcing a new alignment, 'new_size' is smaller than the old size. However, the memcpy() used the old size ('orig_size ?: ks'), leading to an out-of-bounds write. The same overflow bug exists in the kvrealloc() fallback path, where the old bucket size ksize(p) is copied into the new buffer without being bounded by the new size. A simple reproducer: // e.g. add to lkdtm as KREALLOC_SHRINK_OVERFLOW while (1) { void *p = kmalloc(128, GFP_KERNEL); p = krealloc_node_align(p, 64, 256, GFP_KERNEL, NUMA_NO_NODE); kfree(p); } demonstrates the issue: ================================================================== BUG: KFENCE: out-of-bounds write in memcpy_orig+0x68/0x130 Out-of-bounds write at 0xffff8883ad757038 (120B right of kfence-#47): memcpy_orig+0x68/0x130 krealloc_node_align_noprof+0x1c8/0x340 lkdtm_KREALLOC_SHRINK_OVERFLOW+0x8c/0xc0 [lkdtm] lkdtm_do_action+0x3a/0x60 [lkdtm] ... kfence-#47: 0xffff8883ad756fc0-0xffff8883ad756fff, size=64, cache=kmalloc-64 allocated by task 316 on cpu 7 at 97.680481s (0.021813s ago): krealloc_node_align_noprof+0x19c/0x340 lkdtm_KREALLOC_SHRINK_OVERFLOW+0x8c/0xc0 [lkdtm] lkdtm_do_action+0x3a/0x60 [lkdtm] ... ================================================================== Fix it by moving the old size calculation to the top of __do_krealloc() and bounding all copy lengths by the new allocation size. Fixes: 2cd8231796b5 ("mm/slub: allow to set node and align in k[v]realloc") Cc: stable@vger.kernel.org Reported-by: https://sashiko.dev/#/patchset/20260415143735.2974230-1-elver%40google.com Signed-off-by: Marco Elver Link: https://patch.msgid.link/20260416132837.3787694-1-elver@google.com Reviewed-by: Harry Yoo (Oracle) Signed-off-by: Vlastimil Babka (SUSE) Signed-off-by: Greg Kroah-Hartman

slab: distinguish lock and trylock for sheaf_flush_main()

2026-03-19T15:08:39+00:00

commit 48647d3f9a644d1e81af6558102d43cdb260597b upstream. sheaf_flush_main() can be called from __pcs_replace_full_main() where it's fine if the trylock fails, and pcs_flush_all() where it's not expected to and for some flush callers (when destroying the cache or memory hotremove) it would be actually a problem if it failed and left the main sheaf not flushed. The flush callers can however safely use local_lock() instead of trylock. The trylock failure should not happen in practice on !PREEMPT_RT, but can happen on PREEMPT_RT. The impact is limited in practice because when a trylock fails in the kmem_cache_destroy() path, it means someone is using the cache while destroying it, which is a bug on its own. The memory hotremove path is unlikely to be employed in a production RT config, but it's possible. To fix this, split the function into sheaf_flush_main() (using local_lock()) and sheaf_try_flush_main() (using local_trylock()) where both call __sheaf_flush_main_batch() to flush a single batch of objects. This will also allow lockdep to verify our context assumptions. The problem was raised in an off-list question by Marcelo. Fixes: 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves") Cc: stable@vger.kernel.org Reported-by: Marcelo Tosatti Signed-off-by: Vlastimil Babka Reviewed-by: Harry Yoo Reviewed-by: Hao Li Link: https://patch.msgid.link/20260211-b4-sheaf-flush-v1-1-4e7f492f0055@suse.cz Signed-off-by: Vlastimil Babka (SUSE) Signed-off-by: Greg Kroah-Hartman

mm/slab: fix an incorrect check in obj_exts_alloc_size()

2026-03-19T15:08:35+00:00

commit 8dafa9f5900c4855a65dbfee51e3bd00636deee1 upstream. obj_exts_alloc_size() prevents recursive allocation of slabobj_ext array from the same cache, to avoid creating slabs that are never freed. There is one mistake that returns the original size when memory allocation profiling is disabled. The assumption was that memcg-triggered slabobj_ext allocation is always served from KMALLOC_CGROUP type. But this is wrong [1]: when the caller specifies both __GFP_RECLAIMABLE and __GFP_ACCOUNT with SLUB_TINY enabled, the allocation is served from normal kmalloc. This is because kmalloc_type() prioritizes __GFP_RECLAIMABLE over __GFP_ACCOUNT, and SLUB_TINY aliases KMALLOC_RECLAIM with KMALLOC_NORMAL. As a result, the recursion guard is bypassed and the problematic slabs can be created. Fix this by removing the mem_alloc_profiling_enabled() check entirely. The remaining is_kmalloc_normal() check is still sufficient to detect whether the cache is of KMALLOC_NORMAL type and avoid bumping the size if it's not. Without SLUB_TINY, no functional change intended. With SLUB_TINY, allocations with __GFP_ACCOUNT|__GFP_RECLAIMABLE now allocate a larger array if the sizes equal. Reported-by: Zw Tang Fixes: 280ea9c3154b ("mm/slab: avoid allocating slabobj_ext array from its own slab") Closes: https://lore.kernel.org/linux-mm/CAPHJ_VKuMKSke8b11AZQw1PTSFN4n2C0gFxC6xGOG0ZLHgPmnA@mail.gmail.com [1] Cc: stable@vger.kernel.org Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260309072219.22653-1-harry.yoo@oracle.com Tested-by: Zw Tang Signed-off-by: Vlastimil Babka (SUSE) Signed-off-by: Greg Kroah-Hartman

mm/slab: use prandom if !allow_spin

2026-03-12T11:09:32+00:00

[ Upstream commit a1e244a9f177894969c6cd5ebbc6d72c19fc4a7a ] When CONFIG_SLAB_FREELIST_RANDOM is enabled and get_random_u32() is called in an NMI context, lockdep complains because it acquires a local_lock: ================================ WARNING: inconsistent lock state 6.19.0-rc5-slab-for-next+ #325 Tainted: G N -------------------------------- inconsistent {INITIAL USE} -> {IN-NMI} usage. kunit_try_catch/8312 [HC2[2]:SC0[0]:HE0:SE1] takes: ffff88a02ec49cc0 (batched_entropy_u32.lock){-.-.}-{3:3}, at: get_random_u32+0x7f/0x2e0 {INITIAL USE} state was registered at: lock_acquire+0xd9/0x2f0 get_random_u32+0x93/0x2e0 __get_random_u32_below+0x17/0x70 cache_random_seq_create+0x121/0x1c0 init_cache_random_seq+0x5d/0x110 do_kmem_cache_create+0x1e0/0xa30 __kmem_cache_create_args+0x4ec/0x830 create_kmalloc_caches+0xe6/0x130 kmem_cache_init+0x1b1/0x660 mm_core_init+0x1d8/0x4b0 start_kernel+0x620/0xcd0 x86_64_start_reservations+0x18/0x30 x86_64_start_kernel+0xf3/0x140 common_startup_64+0x13e/0x148 irq event stamp: 76 hardirqs last enabled at (75): [] exc_nmi+0x11a/0x240 hardirqs last disabled at (76): [] sysvec_irq_work+0x11/0x110 softirqs last enabled at (0): [] copy_process+0xc7a/0x2350 softirqs last disabled at (0): [<0000000000000000>] 0x0 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(batched_entropy_u32.lock); lock(batched_entropy_u32.lock); *** DEADLOCK *** Fix this by using pseudo-random number generator if !allow_spin. This means kmalloc_nolock() users won't get truly random numbers, but there is not much we can do about it. Note that an NMI handler might interrupt prandom_u32_state() and change the random state, but that's safe. Link: https://lore.kernel.org/all/0c33bdee-6de8-4d9f-92ca-4f72c1b6fb9f@suse.cz Fixes: af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().") Cc: stable@vger.kernel.org Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260210081900.329447-3-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka Signed-off-by: Sasha Levin

slub: remove CONFIG_SLUB_TINY specific code paths

2026-03-12T11:09:32+00:00

[ Upstream commit 31e0886fd57d426d18a239dd55e176032c9c1cb0 ] CONFIG_SLUB_TINY minimizes the SLUB's memory overhead in multiple ways, mainly by avoiding percpu caching of slabs and objects. It also reduces code size by replacing some code paths with simplified ones through ifdefs, but the benefits of that are smaller and would complicate the upcoming changes. Thus remove these code paths and associated ifdefs and simplify the code base. Link: https://patch.msgid.link/20251105-sheaves-cleanups-v1-4-b8218e1ac7ef@suse.cz Reviewed-by: Harry Yoo Signed-off-by: Vlastimil Babka Stable-dep-of: a1e244a9f177 ("mm/slab: use prandom if !allow_spin") Signed-off-by: Sasha Levin

mm/slab: do not access current->mems_allowed_seq if !allow_spin

2026-03-04T12:21:26+00:00

[ Upstream commit 144080a5823b2dbd635acb6decf7ab23182664f3 ] Lockdep complains when get_from_any_partial() is called in an NMI context, because current->mems_allowed_seq is seqcount_spinlock_t and not NMI-safe: ================================ WARNING: inconsistent lock state 6.19.0-rc5-kfree-rcu+ #315 Tainted: G N -------------------------------- inconsistent {INITIAL USE} -> {IN-NMI} usage. kunit_try_catch/9989 [HC1[1]:SC0[0]:HE0:SE1] takes: ffff889085799820 (&____s->seqcount#3){.-.-}-{0:0}, at: ___slab_alloc+0x58f/0xc00 {INITIAL USE} state was registered at: lock_acquire+0x185/0x320 kernel_init_freeable+0x391/0x1150 kernel_init+0x1f/0x220 ret_from_fork+0x736/0x8f0 ret_from_fork_asm+0x1a/0x30 irq event stamp: 56 hardirqs last enabled at (55): [] _raw_spin_unlock_irq+0x27/0x70 hardirqs last disabled at (56): [] __schedule+0x2a8a/0x6630 softirqs last enabled at (0): [] copy_process+0x1dc1/0x6a10 softirqs last disabled at (0): [<0000000000000000>] 0x0 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&____s->seqcount#3); lock(&____s->seqcount#3); *** DEADLOCK *** According to Documentation/locking/seqlock.rst, seqcount_t is not NMI-safe and seqcount_latch_t should be used when read path can interrupt the write-side critical section. In this case, do not access current->mems_allowed_seq and avoid retry. Fixes: af92793e52c3 ("slab: Introduce kmalloc_nolock() and kfree_nolock().") Cc: stable@vger.kernel.org Signed-off-by: Harry Yoo Link: https://patch.msgid.link/20260210081900.329447-2-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka Signed-off-by: Sasha Levin

mm/slab: use unsigned long for orig_size to ensure proper metadata align

2026-03-04T12:21:22+00:00

[ Upstream commit b85f369b81aed457acbea4ad3314218254a72fd2 ] When both KASAN and SLAB_STORE_USER are enabled, accesses to struct kasan_alloc_meta fields can be misaligned on 64-bit architectures. This occurs because orig_size is currently defined as unsigned int, which only guarantees 4-byte alignment. When struct kasan_alloc_meta is placed after orig_size, it may end up at a 4-byte boundary rather than the required 8-byte boundary on 64-bit systems. Note that 64-bit architectures without HAVE_EFFICIENT_UNALIGNED_ACCESS are assumed to require 64-bit accesses to be 64-bit aligned. See HAVE_64BIT_ALIGNED_ACCESS and commit adab66b71abf ("Revert: "ring-buffer: Remove HAVE_64BIT_ALIGNED_ACCESS"") for more details. Change orig_size from unsigned int to unsigned long to ensure proper alignment for any subsequent metadata. This should not waste additional memory because kmalloc objects are already aligned to at least ARCH_KMALLOC_MINALIGN. Closes: https://lore.kernel.org/all/aPrLF0OUK651M4dk@hyeyoo Suggested-by: Andrey Ryabinin Cc: stable@vger.kernel.org Fixes: 6edf2576a6cc ("mm/slub: enable debugging memory wasting of kmalloc") Signed-off-by: Harry Yoo Closes: https://lore.kernel.org/all/aPrLF0OUK651M4dk@hyeyoo/ Link: https://patch.msgid.link/20260113061845.159790-2-harry.yoo@oracle.com Signed-off-by: Vlastimil Babka Signed-off-by: Sasha Levin