Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull locking updates from Ingo Molnar:
 "The main changes are:

   - 'qspinlock' support, enabled on x86: queued spinlocks - these are
     now the spinlock variant used by x86 as they outperform ticket
     spinlocks in every category.  (Waiman Long)

   - 'pvqspinlock' support on x86: paravirtualized variant of queued
     spinlocks.  (Waiman Long, Peter Zijlstra)

   - 'qrwlock' support, enabled on x86: queued rwlocks.  Similar to
     queued spinlocks, they are now the variant used by x86:

       CONFIG_ARCH_USE_QUEUED_SPINLOCKS=y
       CONFIG_QUEUED_SPINLOCKS=y
       CONFIG_ARCH_USE_QUEUED_RWLOCKS=y
       CONFIG_QUEUED_RWLOCKS=y

   - various lockdep fixlets

   - various locking primitives cleanups, further WRITE_ONCE()
     propagation"

* 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (24 commits)
  locking/lockdep: Remove hard coded array size dependency
  locking/qrwlock: Don't contend with readers when setting _QW_WAITING
  lockdep: Do not break user-visible string
  locking/arch: Rename set_mb() to smp_store_mb()
  locking/arch: Add WRITE_ONCE() to set_mb()
  rtmutex: Warn if trylock is called from hard/softirq context
  arch: Remove __ARCH_HAVE_CMPXCHG
  locking/rtmutex: Drop usage of __HAVE_ARCH_CMPXCHG
  locking/qrwlock: Rename QUEUE_RWLOCK to QUEUED_RWLOCKS
  locking/pvqspinlock: Rename QUEUED_SPINLOCK to QUEUED_SPINLOCKS
  locking/pvqspinlock: Replace xchg() by the more descriptive set_mb()
  locking/pvqspinlock, x86: Enable PV qspinlock for Xen
  locking/pvqspinlock, x86: Enable PV qspinlock for KVM
  locking/pvqspinlock, x86: Implement the paravirt qspinlock call patching
  locking/pvqspinlock: Implement simple paravirt support for the qspinlock
  locking/qspinlock: Revert to test-and-set on hypervisors
  locking/qspinlock: Use a simple write to grab the lock
  locking/qspinlock: Optimize for smaller NR_CPUS
  locking/qspinlock: Extract out code snippets for the next patch
  locking/qspinlock: Add pending bit
  ...

1 files changed, 44 insertions, 0 deletions

diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 3417d0172a5d..0f189714e457 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -409,11 +409,24 @@ done:
         return taken;
 }
 
+/*
+ * Return true if the rwsem has active spinner
+ */
+static inline bool rwsem_has_spinner(struct rw_semaphore *sem)
+{
+        return osq_is_locked(&sem->osq);
+}
+
 #else
 static bool rwsem_optimistic_spin(struct rw_semaphore *sem)
 {
         return false;
 }
+
+static inline bool rwsem_has_spinner(struct rw_semaphore *sem)
+{
+        return false;
+}
 #endif
 
 /*
@@ -496,7 +509,38 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
 {
         unsigned long flags;
 
+        /*
+         * If a spinner is present, it is not necessary to do the wakeup.
+         * Try to do wakeup only if the trylock succeeds to minimize
+         * spinlock contention which may introduce too much delay in the
+         * unlock operation.
+         *
+         *    spinning writer           up_write/up_read caller
+         *    ---------------           -----------------------
+         * [S]   osq_unlock()           [L]   osq
+         *       MB                           RMB
+         * [RmW] rwsem_try_write_lock() [RmW] spin_trylock(wait_lock)
+         *
+         * Here, it is important to make sure that there won't be a missed
+         * wakeup while the rwsem is free and the only spinning writer goes
+         * to sleep without taking the rwsem. Even when the spinning writer
+         * is just going to break out of the waiting loop, it will still do
+         * a trylock in rwsem_down_write_failed() before sleeping. IOW, if
+         * rwsem_has_spinner() is true, it will guarantee at least one
+         * trylock attempt on the rwsem later on.
+         */
+        if (rwsem_has_spinner(sem)) {
+                /*
+                 * The smp_rmb() here is to make sure that the spinner
+                 * state is consulted before reading the wait_lock.
+                 */
+                smp_rmb();
+                if (!raw_spin_trylock_irqsave(&sem->wait_lock, flags))
+                        return sem;
+                goto locked;
+        }
         raw_spin_lock_irqsave(&sem->wait_lock, flags);
+locked:
 
         /* do nothing if list empty */
         if (!list_empty(&sem->wait_list))