rcu: Make cond_resched_rcu_qs() supply RCU-sched expedited QS

Although cond_resched_rcu_qs() supplies quiescent states to all flavors
of normal RCU grace periods, it does nothing for expedited RCU-sched
grace periods.  This commit therefore adds a check for a need for a
quiescent state from the current CPU by an expedited RCU-sched grace
period, and invokes rcu_sched_qs() to supply that quiescent state if so.

Note that the check is racy in that we might be migrated to some other
CPU just after checking the per-CPU variable.  This is OK because the
act of migration will do a context switch, which will supply the needed
quiescent state.  The only downside is that we might do an unnecessary
call to rcu_sched_qs(), but the probability is low and the overhead
is small.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

1 files changed, 15 insertions, 0 deletions

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 687d8a5f35c7..178575c01d09 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -370,6 +370,21 @@ void rcu_all_qs(void)
                 rcu_momentary_dyntick_idle();
                 local_irq_restore(flags);
         }
+        if (unlikely(raw_cpu_read(rcu_sched_data.cpu_no_qs.b.exp))) {
+                /*
+                 * Yes, we just checked a per-CPU variable with preemption
+                 * enabled, so we might be migrated to some other CPU at
+                 * this point.  That is OK because in that case, the
+                 * migration will supply the needed quiescent state.
+                 * We might end up needlessly disabling preemption and
+                 * invoking rcu_sched_qs() on the destination CPU, but
+                 * the probability and cost are both quite low, so this
+                 * should not be a problem in practice.
+                 */
+                preempt_disable();
+                rcu_sched_qs();
+                preempt_enable();
+        }
         this_cpu_inc(rcu_qs_ctr);
         barrier(); /* Avoid RCU read-side critical sections leaking up. */
 }