kernel/linux.git/include/linux/percpu-refcount.h, branch v6.6.132

percpu_ref: Replace kernel.h with the necessary inclusions

2021-12-09T20:41:09+00:00

When kernel.h is used in the headers it adds a lot into dependency hell, especially when there are circular dependencies are involved. Replace kernel.h inclusion with the list of what is really being used. Signed-off-by: Andy Shevchenko Signed-off-by: Dennis Zhou

percpu_ref: percpu_ref_tryget_live() version holding RCU

2021-10-21T14:37:26+00:00

Add percpu_ref_tryget_live_rcu(), which is a version of percpu_ref_tryget_live() but the user is responsible for enclosing it in a RCU read lock section. Signed-off-by: Pavel Begunkov Acked-by: Dennis Zhou Link: https://lore.kernel.org/r/3066500d7a6eb3e03f10adf98b87fdb3b1c49db8.1634822969.git.asml.silence@gmail.com Signed-off-by: Jens Axboe

lib: fix spelling mistakes in header files

2021-07-08T18:48:20+00:00

Fix some spelling mistakes in comments found by "codespell": Hoever ==> However poiter ==> pointer representaion ==> representation uppon ==> upon independend ==> independent aquired ==> acquired mis-match ==> mismatch scrach ==> scratch struture ==> structure Analagous ==> Analogous interation ==> iteration And some were discovered manually by Joe Perches and Christoph Lameter: stroed ==> stored arch independent ==> an architecture independent A example structure for ==> Example structure for Link: https://lkml.kernel.org/r/20210609150027.14805-2-thunder.leizhen@huawei.com Signed-off-by: Zhen Lei Cc: Christoph Lameter Cc: Masami Hiramatsu Cc: Dennis Zhou Cc: Tejun Heo Cc: Joe Perches Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds

percpu_ref: reduce memory footprint of percpu_ref in fast path

2020-10-06T13:29:36+00:00

'struct percpu_ref' is often embedded into one user structure, and the instance is usually referenced in fast path, however actually only 'percpu_count_ptr' is needed in fast path. So move other fields into one new structure of 'percpu_ref_data', and allocate it dynamically via kzalloc(), then memory footprint of 'percpu_ref' in fast path is reduced a lot and becomes suitable to put into hot cacheline of user structure. Signed-off-by: Ming Lei Tested-by: Veronika Kabatova Reviewed-by: Christoph Hellwig Acked-by: Tejun Heo Cc: Sagi Grimberg Cc: Tejun Heo Cc: Christoph Hellwig Cc: Jens Axboe Cc: Bart Van Assche Signed-off-by: Jens Axboe

include/linux: Remove smp_read_barrier_depends() from comments

2020-07-21T09:50:37+00:00

smp_read_barrier_depends() doesn't exist any more, so reword the two comments that mention it to refer to "dependency ordering" instead. Acked-by: Peter Zijlstra (Intel) Acked-by: Paul E. McKenney Signed-off-by: Will Deacon

pcpu_ref: add percpu_ref_tryget_many()

2020-01-21T00:04:02+00:00

Add percpu_ref_tryget_many(), which works the same way as percpu_ref_tryget(), but grabs specified number of refs. Signed-off-by: Pavel Begunkov Acked-by: Tejun Heo Acked-by: Dennis Zhou Cc: Christoph Lameter Signed-off-by: Jens Axboe

percpu-refcount: Use normal instead of RCU-sched"

2019-11-17T04:02:47+00:00

This is a revert of commit a4244454df129 ("percpu-refcount: use RCU-sched insted of normal RCU") which claims the only reason for using RCU-sched is "rcu_read_[un]lock() … are slightly more expensive than preempt_disable/enable()" and "As the RCU critical sections are extremely short, using sched-RCU shouldn't have any latency implications." The problem with using RCU-sched here is that it disables preemption and the release callback (called from percpu_ref_put_many()) must not acquire any sleeping locks like spinlock_t. This breaks PREEMPT_RT because some of the users acquire spinlock_t locks in their callbacks. Using rcu_read_lock() on PREEMPTION=n kernels is not any different compared to rcu_read_lock_sched(). On PREEMPTION=y kernels there are already performance issues due to additional preemption points. Looking at the code, the rcu_read_lock() is just an increment and unlock is almost just a decrement unless there is something special to do. Both are functions while disabling preemption is inlined. Doing a small benchmark, the minimal amount of time required was mostly the same. The average time required was higher due to the higher MAX value (which could be preemption). With DEBUG_PREEMPT=y it is rcu_read_lock_sched() that takes a little longer due to the additional debug code. Convert back to normal RCU. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Dennis Zhou

percpu_ref: release percpu memory early without PERCPU_REF_ALLOW_REINIT

2019-05-09T17:51:06+00:00

Release percpu memory after finishing the switch to the atomic mode if only PERCPU_REF_ALLOW_REINIT isn't set. Signed-off-by: Roman Gushchin Acked-by: Tejun Heo Signed-off-by: Dennis Zhou

percpu_ref: introduce PERCPU_REF_ALLOW_REINIT flag

2019-05-09T17:49:47+00:00

In most cases percpu reference counters are not switched to the percpu mode after they reach the atomic mode. Some obvious exceptions are reference counters which are initialized into the atomic mode (using PERCPU_REF_INIT_ATOMIC and PERCPU_REF_INIT_DEAD flags), and there are few other exceptions. But in most cases there is no way back, and once the reference counter is switched to the atomic mode, there is no reason to wait for percpu_ref_exit() to release the percpu memory. Of course, the size of a single counter is not so big, but because it can pin the whole percpu block in memory, the memory footprint can be noticeable (e.g. on my 32 CPUs machine a percpu block is 8Mb large). To make releasing of the percpu memory as early as possible, let's introduce the PERCPU_REF_ALLOW_REINIT flag with the following semantics: it has to be set in order to switch a percpu reference counter to the percpu mode after the initialization. PERCPU_REF_INIT_ATOMIC and PERCPU_REF_INIT_DEAD flags will implicitly assume PERCPU_REF_ALLOW_REINIT. This patch doesn't introduce any functional change to avoid any regressions. It will be done later in the patchset after adjusting all call sites, which are reviving percpu counters. Signed-off-by: Roman Gushchin Acked-by: Tejun Heo Signed-off-by: Dennis Zhou

percpu-refcount: Introduce percpu_ref_resurrect()

2018-09-26T21:11:29+00:00

This function will be used in a later patch to switch the struct request_queue q_usage_counter from killed back to live. In contrast to percpu_ref_reinit(), this new function does not require that the refcount is zero. Signed-off-by: Bart Van Assche Acked-by: Tejun Heo Reviewed-by: Ming Lei Cc: Christoph Hellwig Cc: Jianchao Wang Cc: Hannes Reinecke Cc: Johannes Thumshirn Signed-off-by: Jens Axboe