sched/fair: Rewrite cfs_rq->removed_*avg

Since on wakeup migration we don't hold the rq->lock for the old CPU
we cannot update its state. Instead we add the removed 'load' to an
atomic variable and have the next update on that CPU collect and
process it.

Currently we have 2 atomic variables; which already have the issue
that they can be read out-of-sync. Also, two atomic ops on a single
cacheline is already more expensive than an uncontended lock.

Since we want to add more, convert the thing over to an explicit
cacheline with a lock in.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>

1 files changed, 4 insertions, 4 deletions

diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 2f93e4a2d9f6..2f22342c48ff 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -564,10 +564,10 @@ void print_cfs_rq(struct seq_file *m, int cpu, struct cfs_rq *cfs_rq)
                         cfs_rq->runnable_load_avg);
         SEQ_printf(m, "  .%-30s: %lu\n", "util_avg",
                         cfs_rq->avg.util_avg);
-        SEQ_printf(m, "  .%-30s: %ld\n", "removed_load_avg",
-                        atomic_long_read(&cfs_rq->removed_load_avg));
-        SEQ_printf(m, "  .%-30s: %ld\n", "removed_util_avg",
-                        atomic_long_read(&cfs_rq->removed_util_avg));
+        SEQ_printf(m, "  .%-30s: %ld\n", "removed.load_avg",
+                        cfs_rq->removed.load_avg);
+        SEQ_printf(m, "  .%-30s: %ld\n", "removed.util_avg",
+                        cfs_rq->removed.util_avg);
 #ifdef CONFIG_FAIR_GROUP_SCHED
         SEQ_printf(m, "  .%-30s: %lu\n", "tg_load_avg_contrib",
                         cfs_rq->tg_load_avg_contrib);