summaryrefslogtreecommitdiff
path: root/arch/x86/include/asm/barrier.h
diff options
context:
space:
mode:
authorPeter Zijlstra <peterz@infradead.org>2019-04-24 14:38:23 +0300
committerGreg Kroah-Hartman <gregkh@linuxfoundation.org>2019-07-26 10:14:08 +0300
commit9e0bcb59b6c0badce64f9862fca5b0b997929f1f (patch)
tree5f1d465a0c29ca1cd9ee8bdd090a23d4709a8f78 /arch/x86/include/asm/barrier.h
parentdd0260fd1e3af66a3b3c87301f72cb04d3dcf147 (diff)
downloadlinux-9e0bcb59b6c0badce64f9862fca5b0b997929f1f.tar.xz
x86/atomic: Fix smp_mb__{before,after}_atomic()
[ Upstream commit 69d927bba39517d0980462efc051875b7f4db185 ] Recent probing at the Linux Kernel Memory Model uncovered a 'surprise'. Strongly ordered architectures where the atomic RmW primitive implies full memory ordering and smp_mb__{before,after}_atomic() are a simple barrier() (such as x86) fail for: *x = 1; atomic_inc(u); smp_mb__after_atomic(); r0 = *y; Because, while the atomic_inc() implies memory order, it (surprisingly) does not provide a compiler barrier. This then allows the compiler to re-order like so: atomic_inc(u); *x = 1; smp_mb__after_atomic(); r0 = *y; Which the CPU is then allowed to re-order (under TSO rules) like: atomic_inc(u); r0 = *y; *x = 1; And this very much was not intended. Therefore strengthen the atomic RmW ops to include a compiler barrier. NOTE: atomic_{or,and,xor} and the bitops already had the compiler barrier. Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Ingo Molnar <mingo@kernel.org> Signed-off-by: Sasha Levin <sashal@kernel.org>
Diffstat (limited to 'arch/x86/include/asm/barrier.h')
-rw-r--r--arch/x86/include/asm/barrier.h4
1 files changed, 2 insertions, 2 deletions
diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h
index 14de0432d288..84f848c2541a 100644
--- a/arch/x86/include/asm/barrier.h
+++ b/arch/x86/include/asm/barrier.h
@@ -80,8 +80,8 @@ do { \
})
/* Atomic operations are already serializing on x86 */
-#define __smp_mb__before_atomic() barrier()
-#define __smp_mb__after_atomic() barrier()
+#define __smp_mb__before_atomic() do { } while (0)
+#define __smp_mb__after_atomic() do { } while (0)
#include <asm-generic/barrier.h>