sched: adaptive scheduler granularity

Instead of specifying the preemption granularity, specify the wanted
latency. By fixing the granlarity to a constany the wakeup latency
it a function of the number of running tasks on the rq.

Invert this relation.

sysctl_sched_granularity becomes a minimum for the dynamic granularity
computed from the new sysctl_sched_latency.

Then use this latency to do more intelligent granularity decisions: if
there are fewer tasks running then we can schedule coarser. This helps
performance while still always keeping the latency target.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Ingo Molnar <mingo@elte.hu>

1 files changed, 9 insertions, 5 deletions

diff --git a/kernel/sched.c b/kernel/sched.c
index 6798328a2e0e..da26f46d50d7 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -4911,14 +4911,18 @@ cpumask_t nohz_cpu_mask = CPU_MASK_NONE;
 static inline void sched_init_granularity(void)
 {
         unsigned int factor = 1 + ilog2(num_online_cpus());
-        const unsigned long gran_limit = 100000000;
+        const unsigned long limit = 100000000;
 
         sysctl_sched_granularity *= factor;
-        if (sysctl_sched_granularity > gran_limit)
-                sysctl_sched_granularity = gran_limit;
+        if (sysctl_sched_granularity > limit)
+                sysctl_sched_granularity = limit;
 
-        sysctl_sched_runtime_limit = sysctl_sched_granularity * 5;
-        sysctl_sched_wakeup_granularity = sysctl_sched_granularity / 2;
+        sysctl_sched_latency *= factor;
+        if (sysctl_sched_latency > limit)
+                sysctl_sched_latency = limit;
+
+        sysctl_sched_runtime_limit = sysctl_sched_latency * 5;
+        sysctl_sched_wakeup_granularity = sysctl_sched_latency / 2;
 }
 
 #ifdef CONFIG_SMP