kernel/linux.git/arch/powerpc/lib/qspinlock.c, branch v6.7

powerpc/qspinlock: Rename yield_propagate_owner tunable

2023-10-20T11:43:34+00:00

Rename yield_propagate_owner to yield_sleepy_owner, which better describes what it does (what, not how). Signed-off-by: Nicholas Piggin Tested-by: Shrikanth Hegde Reviewed-by: "Nysal Jan K.A" Signed-off-by: Michael Ellerman Link: https://msgid.link/20231016124305.139923-7-npiggin@gmail.com

powerpc/qspinlock: Propagate sleepy if previous waiter is preempted

2023-10-20T11:43:34+00:00

The sleepy (aka lock-owner-is-preempted) condition is propagated down the queue by each waiter. If a waiter becomes preempted, it can no longer propagate sleepy. To allow subsequent waiters to yield to the lock owner, also check the lock owner in this case. Signed-off-by: Nicholas Piggin Tested-by: Shrikanth Hegde Reviewed-by: "Nysal Jan K.A" Signed-off-by: Michael Ellerman Link: https://msgid.link/20231016124305.139923-6-npiggin@gmail.com

powerpc/qspinlock: don't propagate the not-sleepy state

2023-10-20T11:43:34+00:00

To simplify things, don't propagate the not-sleepy condition back down the queue. Instead, have the waiters clear their own node->sleepy when finding the lock owner is not preempted. Signed-off-by: Nicholas Piggin Tested-by: Shrikanth Hegde Reviewed-by: "Nysal Jan K.A" Signed-off-by: Michael Ellerman Link: https://msgid.link/20231016124305.139923-5-npiggin@gmail.com

powerpc/qspinlock: propagate owner preemptedness rather than CPU number

2023-10-20T11:43:34+00:00

Rather than propagating the CPU number of the preempted lock owner, just propagate whether the owner was preempted. Waiters must read the lock value when yielding to it to prevent races anyway, so might as well always load the owner CPU from the lock. To further simplify the code, also don't propagate the -1 (or sleepy=false in the new scheme) down the queue. Instead, have the waiters clear it themselves when finding the lock owner is not preempted. Signed-off-by: Nicholas Piggin Tested-by: Shrikanth Hegde Reviewed-by: "Nysal Jan K.A" Signed-off-by: Michael Ellerman Link: https://msgid.link/20231016124305.139923-4-npiggin@gmail.com

powerpc/qspinlock: stop queued waiters trying to set lock sleepy

2023-10-20T11:43:34+00:00

If a queued waiter notices the lock owner or the previous waiter has been preempted, it attempts to mark the lock sleepy, but it does this as a try-set operation using the original lock value it got when queueing, which will become stale as the queue progresses, and the try-set will fail. Drop this and just set the sleepy seen clock. Signed-off-by: Nicholas Piggin Tested-by: Shrikanth Hegde Reviewed-by: "Nysal Jan K.A" Signed-off-by: Michael Ellerman Link: https://msgid.link/20231016124305.139923-3-npiggin@gmail.com

powerpc/qspinlock: Fix stale propagated yield_cpu

2023-10-18T10:07:21+00:00

yield_cpu is a sample of a preempted lock holder that gets propagated back through the queue. Queued waiters use this to yield to the preempted lock holder without continually sampling the lock word (which would defeat the purpose of MCS queueing by bouncing the cache line). The problem is that yield_cpu can become stale. It can take some time to be passed down the chain, and if any queued waiter gets preempted then it will cease to propagate the yield_cpu to later waiters. This can result in yielding to a CPU that no longer holds the lock, which is bad, but particularly if it is currently in H_CEDE (idle), then it appears to be preempted and some hypervisors (PowerVM) can cause very long H_CONFER latencies waiting for H_CEDE wakeup. This results in latency spikes and hard lockups on oversubscribed partitions with lock contention. This is a minimal fix. Before yielding to yield_cpu, sample the lock word to confirm yield_cpu is still the owner, and bail out of it is not. Thanks to a bunch of people who reported this and tracked down the exact problem using tracepoints and dispatch trace logs. Fixes: 28db61e207ea ("powerpc/qspinlock: allow propagation of yield CPU down the queue") Cc: stable@vger.kernel.org # v6.2+ Reported-by: Srikar Dronamraju Reported-by: Laurent Dufour Reported-by: Shrikanth Hegde Debugged-by: "Nysal Jan K.A" Signed-off-by: Nicholas Piggin Tested-by: Shrikanth Hegde Signed-off-by: Michael Ellerman Link: https://msgid.link/20231016124305.139923-2-npiggin@gmail.com

powerpc: qspinlock: Enforce qnode writes prior to publishing to queue

2023-06-21T05:13:57+00:00

Annotate the release barrier and memory clobber (in effect, producing a compiler barrier) in the publish_tail_cpu call. These barriers have the effect of ensuring that qnode attributes are all written to prior to publish the node to the waitqueue. Even while the initial write to the 'locked' attribute is guaranteed to terminate prior to the node being visible, KCSAN still complains that the write is reorderable by the compiler. Issue a kcsan_release() to inform KCSAN of the release barrier contained in publish_tail_cpu(). Signed-off-by: Rohan McLure Reviewed-by: Nicholas Piggin Signed-off-by: Michael Ellerman Link: https://msgid.link/20230510033117.1395895-3-rmclure@linux.ibm.com

powerpc: qspinlock: Mark accesses to qnode lock checks

2023-06-21T05:13:57+00:00

The powerpc implementation of qspinlocks will both poll and spin on the bitlock guarding a qnode. Mark these accesses with READ_ONCE to convey to KCSAN that polling is intentional here. Signed-off-by: Rohan McLure Reviewed-by: Nicholas Piggin Signed-off-by: Michael Ellerman Link: https://msgid.link/20230510033117.1395895-2-rmclure@linux.ibm.com

powerpc/qspinlock: Fix 32-bit build

2022-12-12T01:34:52+00:00

Some 32-bit configurations don't pull in the spin_begin/end/relax definitions. Fix is to restore a lost include. Reported-by: kernel test robot Fixes: 84990b169557 ("powerpc/qspinlock: add mcs queueing for contended waiters") Signed-off-by: Nicholas Piggin Signed-off-by: Michael Ellerman Link: https://lore.kernel.org/oe-kbuild-all/202212050224.i7uh9fOh-lkp@intel.com Link: https://lore.kernel.org/r/20221208123225.1566113-1-npiggin@gmail.com

powerpc/qspinlock: add compile-time tuning adjustments

2022-12-02T06:48:50+00:00

This adds compile-time options that allow the EH lock hint bit to be enabled or disabled, and adds some new options that may or may not help matters. To help with experimentation and tuning. Signed-off-by: Nicholas Piggin Signed-off-by: Michael Ellerman Link: https://lore.kernel.org/r/20221126095932.1234527-18-npiggin@gmail.com