mutex: implement adaptive spinning

Change mutex contention behaviour such that it will sometimes busy wait on
acquisition - moving its behaviour closer to that of spinlocks.

This concept got ported to mainline from the -rt tree, where it was originally
implemented for rtmutexes by Steven Rostedt, based on work by Gregory Haskins.

Testing with Ingo's test-mutex application (http://lkml.org/lkml/2006/1/8/50)
gave a 345% boost for VFS scalability on my testbox:

 # ./test-mutex-shm V 16 10 | grep "^avg ops"
 avg ops/sec:               296604

 # ./test-mutex-shm V 16 10 | grep "^avg ops"
 avg ops/sec:               85870

The key criteria for the busy wait is that the lock owner has to be running on
a (different) cpu. The idea is that as long as the owner is running, there is a
fair chance it'll release the lock soon, and thus we'll be better off spinning
instead of blocking/scheduling.

Since regular mutexes (as opposed to rtmutexes) do not atomically track the
owner, we add the owner in a non-atomic fashion and deal with the races in
the slowpath.

Furthermore, to ease the testing of the performance impact of this new code,
there is means to disable this behaviour runtime (without having to reboot
the system), when scheduler debugging is enabled (CONFIG_SCHED_DEBUG=y),
by issuing the following command:

 # echo NO_OWNER_SPIN > /debug/sched_features

This command re-enables spinning again (this is also the default):

 # echo OWNER_SPIN > /debug/sched_features

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Ingo Molnar <mingo@elte.hu>

1 files changed, 61 insertions, 0 deletions

diff --git a/kernel/sched.c b/kernel/sched.c
index b001c133c359..589e7308c615 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -4614,6 +4614,67 @@ need_resched:
 }
 EXPORT_SYMBOL(schedule);
 
+#ifdef CONFIG_SMP
+/*
+ * Look out! "owner" is an entirely speculative pointer
+ * access and not reliable.
+ */
+int mutex_spin_on_owner(struct mutex *lock, struct thread_info *owner)
+{
+        unsigned int cpu;
+        struct rq *rq;
+
+        if (!sched_feat(OWNER_SPIN))
+                return 0;
+
+#ifdef CONFIG_DEBUG_PAGEALLOC
+        /*
+         * Need to access the cpu field knowing that
+         * DEBUG_PAGEALLOC could have unmapped it if
+         * the mutex owner just released it and exited.
+         */
+        if (probe_kernel_address(&owner->cpu, cpu))
+                goto out;
+#else
+        cpu = owner->cpu;
+#endif
+
+        /*
+         * Even if the access succeeded (likely case),
+         * the cpu field may no longer be valid.
+         */
+        if (cpu >= nr_cpumask_bits)
+                goto out;
+
+        /*
+         * We need to validate that we can do a
+         * get_cpu() and that we have the percpu area.
+         */
+        if (!cpu_online(cpu))
+                goto out;
+
+        rq = cpu_rq(cpu);
+
+        for (;;) {
+                /*
+                 * Owner changed, break to re-assess state.
+                 */
+                if (lock->owner != owner)
+                        break;
+
+                /*
+                 * Is that owner really running on that cpu?
+                 */
+                if (task_thread_info(rq->curr) != owner || need_resched())
+                        return 0;
+
+                cpu_relax();
+        }
+out:
+        return 1;
+}
+#endif
+
 #ifdef CONFIG_PREEMPT
 /*
  * this is the entry point to schedule() from in-kernel preemption