[SPARC64]: More fully work around Spitfire Errata 51.

It appears that a memory barrier soon after a mispredicted
branch, not just in the delay slot, can cause the hang
condition of this cpu errata.

So move them out-of-line, and explicitly put them into
a "branch always, predict taken" delay slot which should
fully kill this problem.

Signed-off-by: David S. Miller <davem@davemloft.net>

1 files changed, 4 insertions, 4 deletions

diff --git a/include/asm-sparc64/atomic.h b/include/asm-sparc64/atomic.h
index d80f3379669b..e175afcf2cde 100644
--- a/include/asm-sparc64/atomic.h
+++ b/include/asm-sparc64/atomic.h
@@ -72,10 +72,10 @@ extern int atomic64_sub_ret(int, atomic64_t *);
 
 /* Atomic operations are already serializing */
 #ifdef CONFIG_SMP
-#define smp_mb__before_atomic_dec()     membar("#StoreLoad | #LoadLoad")
-#define smp_mb__after_atomic_dec()      membar("#StoreLoad | #StoreStore")
-#define smp_mb__before_atomic_inc()     membar("#StoreLoad | #LoadLoad")
-#define smp_mb__after_atomic_inc()      membar("#StoreLoad | #StoreStore")
+#define smp_mb__before_atomic_dec()     membar_storeload_loadload();
+#define smp_mb__after_atomic_dec()      membar_storeload_storestore();
+#define smp_mb__before_atomic_inc()     membar_storeload_loadload();
+#define smp_mb__after_atomic_inc()      membar_storeload_storestore();
 #else
 #define smp_mb__before_atomic_dec()     barrier()
 #define smp_mb__after_atomic_dec()      barrier()