[PATCH] atomic: add_unless cmpxchg optimise

Without branch hints, the very unlikely chance of the loop repeating due to
cmpxchg failure is unrolled with gcc-4 that I have tested.

Improve this for architectures with a native cas/cmpxchg.  llsc archs
should try to implement this natively.

Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Andi Kleen <ak@muc.de>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Roman Zippel <zippel@linux-m68k.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

1 files changed, 7 insertions, 1 deletions

diff --git a/include/asm-x86_64/atomic.h b/include/asm-x86_64/atomic.h
index 4b5cd553e772..cecbf7baa6aa 100644
--- a/include/asm-x86_64/atomic.h
+++ b/include/asm-x86_64/atomic.h
@@ -405,8 +405,14 @@ static __inline__ long atomic64_sub_return(long i, atomic64_t *v)
 ({                                                              \
         int c, old;                                             \
         c = atomic_read(v);                                     \
-        while (c != (u) && (old = atomic_cmpxchg((v), c, c + (a))) != c) \
+        for (;;) {                                              \
+                if (unlikely(c == (u)))                         \
+                        break;                                  \
+                old = atomic_cmpxchg((v), c, c + (a));          \
+                if (likely(old == c))                           \
+                        break;                                  \
                 c = old;                                        \
+        }                                                       \
         c != (u);                                               \
 })
 #define atomic_inc_not_zero(v) atomic_add_unless((v), 1, 0)