kernel/linux.git/include/linux/rcupdate.h, branch v7.3-rc1

Merge tag 'slab-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab

2026-08-24T17:58:57+00:00

Pull slab updates from Vlastimil Babka: - Add kfree_rcu_nolock() that can be used from contexts where spinning on a lock might be unsafe, such as a BPF program attached to an arbitrary function, or in NMI context. This complements the existing kfree_nolock() support (Harry Yoo) - Runtime instead of compile-time slabobj_ext sizing. Avoid wasting memory when memory allocation profiling is compiled but not enabled, with initial partial support to also avoid wasting memory for objcg pointers when those are not needed, while profiling is enabled (Vlastimil Babka) - Various non-urgent fixes, cleanups and optimizations (Hao Li, Hongling Zeng, Li RongQing, Li Xiasong, Seongjun Hong, Shengming Hu) * tag 'slab-for-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab: (31 commits) mm/slab, kfence, memcg: completely remove obj_ext for kfence objects mm/slab: stop allocating objcg pointers when unnecessary mm/slab: add cache_ and slab_needs_objcg() helpers mm/slab: stop exporting kvfree_rcu_barrier[_on_cache]() slub_kunit: extend the test for kfree_rcu_nolock() mm/slab: introduce kfree_rcu_nolock() mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching mm/slab: reduce slabobj_ext memory with allocation profiling disabled mm/slab: introduce slab_obj_ext_has_codetag() mm/slab: allow kfree_rcu_sheaf() on PREEMPT_RT mm/slab: extend deferred free mechanism to handle rcu sheaves mm/slab: use call_rcu() in unknown context if irqs are enabled mm/slab: handle the !allow_spin case in kfree_rcu_sheaf() mm/slab: change struct slabobj_ext to a union mm/slab: replace slab.stride with obj_exts_in_object mm/slab: abstract slabobj_ext.ref access mm/slab: abstract slabobj_ext.objcg access mm/slab: make slab_obj_ext() determine object index mm: move struct slabobj_ext to mm/slab.h mm/slab: remove objs_per_slab() ...

Merge branches 'expcb.2026.07.24a', 'misc.2026.07.30a', 'rcu-tasks.2026.07.30a', 'srcu.2026.08.11a' and 'torture.2026.08.14a' into HEAD

2026-08-14T22:43:51+00:00

Changes: Make expedited grace periods expedite normal RCU callbacks Miscellaneous fixes: * Improve diagnostic output with character task states. * Mark accesses to inform KCSAN of concurrency design. * Move from kmalloc() to kmalloc_obj(). * Documentation updates. * Improve handling of RCU deferred quiescent states. * Clean up unused function arguments and structure fields. * Reduce show_rcu_gp_kthreads() stack space. Tasks RCU updates: * Clean up after SRCU re-implementation of Tasks Trace RCU. * Mark accesses to inform KCSAN of concurrency design. * Add ->lazy_timer status to diagnostic output. * Remove an unnecessary memory barrier. * Fix a data race, courtesy of KCSAN. * Documentation updates. * Convert cond_resched_tasks_rcu_qs() from macro to static inline function. SRCU updates: * Add Rust helpers for SRCU. * Avoid losing queued work at cleanup_srcu_struct() time. Torture-test updates: * Preparation work for immediate RCU priority deboosting. * Test RCU readers from real interrupt handlers (as opposed to softirq). * Simplify code through use of cpumask_next_wrap(). * Improve diagnostic output with character task states. * Add rcutorture.nwriters parameter to allow lightweight stall testing, and rcutorture.stall_only to make doing so easier. * Test an RCU Tasks Trace grace period implying an RCU grace period. * Make RCU Tasks Trace torturing track reader batches. * Fix a data race, courtesy of KCSAN. * Plug a shuffle_tmp_mask memory leak on kthread spawn failure.

mm/slab: introduce kfree_rcu_nolock()

2026-08-04T10:18:29+00:00

Currently, k[v]free_rcu() cannot be called in unknown context since it could lead to a deadlock when called in the middle of k[v]free_rcu(). Make users' lives easier by introducing kfree_rcu_nolock() variant, now that kfree_rcu_sheaf() is available on PREEMPT_RT and __kfree_rcu_sheaf() handles unknown context. When sheaves path fails, kfree_rcu_nolock() falls back to defer_kfree_rcu() that uses an irq work to free the object via kvfree_call_rcu(). In most cases, the sheaves path is expected to succeed and therefore it's unnecessary to introduce additional complexity to the existing kvfree_rcu batching by teaching it how to handle unknown context. Since defer_kfree_rcu() can be called on caches without sheaves, move deferred_work_barrier() and rcu_barrier() outside the branch in kvfree_rcu_barrier_on_cache(). Now that deferred kvfree_rcu objects are submitted to kvfree_call_rcu() after deferred_work_barrier() and may end up in RCU sheaves, deferred_work_barrier() must be invoked before flush_rcu_sheaves_on_cache(). Since the RCU sheaf path has not been used on !KVFREE_RCU_BATCHED kernels, always fall back when kvfree_rcu() is not batched, for consistency. kvfree_rcu_barrier{,_on_cache()}() on !KVFREE_RCU_BATCHED are moved to mm/slab_common.c to invoke deferred_work_barrier() before rcu_barrier(). Signed-off-by: Harry Yoo (Oracle) Link: https://patch.msgid.link/20260729-kfree_rcu_nolock-v5-7-a28cdcda9673@kernel.org Signed-off-by: Vlastimil Babka (SUSE)

mm/slab: introduce struct kvfree_rcu_head for kvfree_rcu batching

2026-08-04T10:18:11+00:00

rcu_head is overkill for kvfree_rcu() because the callback function is always either kfree(), vfree(), or free_large_kmalloc(), and thus there is no need for a function pointer. kvfree_rcu batching reuses the field to store the start address of an object, however, this is not strictly needed because we can calculate the start address in the slowpath. For the purpose of kvfree_rcu batching, it is sufficient to implement a linked list using a single pointer. Introduce a new struct called kvfree_rcu_head (the name was suggested by Vlastimil Babka), which is similar to rcu_head but is only a single pointer to build a linked list, without a function pointer, when CONFIG_KVFREE_RCU_BATCHED=y. When kvfree_rcu is not batched, kvfree_rcu_head is the same size as rcu_head. Note that shrinking struct kvfree_rcu_head on CONFIG_KVFREE_RCU_BATCHED=n kernels would inevitably require additional complexity and also some sort of batching (which defeats the purpose of the config option) because it cannot fall back to call_rcu(). For now there are no user-visible changes to the API. k[v]free_rcu() simply casts rcu_head to kvfree_rcu_head. While this does not affect the API, it allows kfree_rcu_nolock() to reuse kvfree_rcu batching as a fallback when trylock or sheaf allocation fails. Stop storing the object pointer in rcu_head.func and instead calculate the object's start address in kvfree_rcu_list(). Factor out the existing logic to calculate the start address from kvfree_rcu_cb() to kvmalloc_obj_start_addr(). To avoid losing the KASAN tag, calculate the offset and subtract it from the address of the kvfree_rcu_head. Signed-off-by: Harry Yoo (Oracle) Link: https://patch.msgid.link/20260729-kfree_rcu_nolock-v5-6-a28cdcda9673@kernel.org Signed-off-by: Vlastimil Babka (SUSE)

rcu-tasks: Convert cond_resched_tasks_rcu_qs() to static inline

2026-07-31T18:59:12+00:00

In order to make "cc -E" output less annoying, this commit converts cond_resched_tasks_rcu_qs() to static inline. You know, the READ_ONCE() and WRITE_ONCE() macros used to be *so* simple. ;-) Reported-by: Andrew Morton Signed-off-by: Paul E. McKenney

rcu: Mark __rcu_access_pointer() as context_unsafe()

2026-07-23T17:47:10+00:00

A simple comparison of a pointer returned by rcu_access_pointer() results in a context-analysis warning for lockless inspection of the RCU-protected (also known as __rcu-protected) pointer. This can be suppressed by placing context_unsafe() calls around calls rcu_access_pointer(), but this is messy and distracting. This commit therefore wraps the underlying __rcu_access_pointer() macro with a call to context_unsafe(), thereby informing the context-analysis code that rcu_access_pointer() may safely be invoked outside of an RCU read-side critical section. Reported-by: Christoph Hellwig Suggested-by: Marco Elver Signed-off-by: Paul E. McKenney Tested-by: Nilay Shroff Reviewed-by: Marco Elver

rcu: Rename struct rcu_gp_oldstate to rcu_gp_seq

2026-07-23T17:46:09+00:00

The polled grace-period state structure rcu_gp_oldstate holds a snapshot of the normal (and, on SMP, expedited) grace-period sequence numbers. Upcoming changes store this structure in the callback segment list, where the "oldstate" name reads poorly: there it represents the grace period a segment is waiting on and is also compared against the current grace-period state. Rename struct rcu_gp_oldstate to the more neutral struct rcu_gp_seq, and shorten its members rgos_norm and rgos_exp to norm and exp. Local variables and parameters of this type are renamed from rgosp/rgos to gsp/gs accordingly. While at it, provide a single definition of the structure in rcupdate.h rather than separate Tiny-RCU and Tree-RCU definitions, and give it the ->exp field unconditionally. Tiny RCU does not track expedited grace periods and leaves ->exp unused, but a single definition that always has ->exp lets the shared callback code in rcu_segcblist.c reference it without CONFIG_SMP guards, including on !SMP builds. No functional change. Signed-off-by: Puranjay Mohan Reviewed-by: Frederic Weisbecker Signed-off-by: Paul E. McKenney

rcu: Document rcu_access_pointer() feeding into cmpxchg()

2026-05-24T07:40:02+00:00

This commit documents the rcu_access_pointer() use case for fetching the old value of an RCU-protected pointer within a lockless updater for use by an atomic cmpxchg() operation. Reviewed-by: Frederic Weisbecker Reported-by: Maxim Mikityanskiy Signed-off-by: Paul E. McKenney Signed-off-by: Uladzislau Rezki (Sony)

bpf: Retire rcu_trace_implies_rcu_gp()

2026-04-07T19:24:49+00:00

RCU Tasks Trace grace period implies RCU grace period, and this guarantee is expected to remain in the future. Only BPF is the user of this predicate, hence retire the API and clean up all in-tree users. RCU Tasks Trace is now implemented on SRCU-fast and its grace period mechanism always has at least one call to synchronize_rcu() as it is required for SRCU-fast's correctness (it replaces the smp_mb() that SRCU-fast readers skip). So, RCU-tt GP will always imply RCU GP. Reviewed-by: Puranjay Mohan Reviewed-by: Paul E. McKenney Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260407162234.785270-1-memxor@gmail.com Signed-off-by: Alexei Starovoitov

Merge tag 'slab-for-7.0-part2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab

2026-02-16T21:41:38+00:00

Pull more slab updates from Vlastimil Babka: - Two stable fixes for kmalloc_nolock() usage from NMI context (Harry Yoo) - Allow kmalloc_nolock() allocations to be freed with kfree() and thus also kfree_rcu() and simplify slabobj_ext handling - we no longer need to track how it was allocated to use the matching freeing function (Harry Yoo) * tag 'slab-for-7.0-part2' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab: mm/slab: drop the OBJEXTS_NOSPIN_ALLOC flag from enum objext_flags mm/slab: allow freeing kmalloc_nolock()'d objects using kfree[_rcu]() mm/slab: use prandom if !allow_spin mm/slab: do not access current->mems_allowed_seq if !allow_spin