rcu: Make rcu_barrier() less disruptive

The rcu_barrier() primitive interrupts each and every CPU, registering
a callback on every CPU.  Once all of these callbacks have been invoked,
rcu_barrier() knows that every callback that was registered before
the call to rcu_barrier() has also been invoked.

However, there is no point in registering a callback on a CPU that
currently has no callbacks, most especially if that CPU is in a
deep idle state.  This commit therefore makes rcu_barrier() avoid
interrupting CPUs that have no callbacks.  Doing this requires reworking
the handling of orphaned callbacks, otherwise callbacks could slip through
rcu_barrier()'s net by being orphaned from a CPU that rcu_barrier() had
not yet interrupted to a CPU that rcu_barrier() had already interrupted.
This reworking was needed anyway to take a first step towards weaning
RCU from the CPU_DYING notifier's use of stop_cpu().

Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

1 files changed, 11 insertions, 0 deletions

diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index 36ca28ecedc6..1e49c5685960 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -371,6 +371,17 @@ struct rcu_state {
 
         raw_spinlock_t onofflock;               /* exclude on/offline and */
                                                 /*  starting new GP. */
+        struct rcu_head *orphan_nxtlist;        /* Orphaned callbacks that */
+                                                /*  need a grace period. */
+        struct rcu_head **orphan_nxttail;       /* Tail of above. */
+        struct rcu_head *orphan_donelist;       /* Orphaned callbacks that */
+                                                /*  are ready to invoke. */
+        struct rcu_head **orphan_donetail;      /* Tail of above. */
+        long qlen_lazy;                         /* Number of lazy callbacks. */
+        long qlen;                              /* Total number of callbacks. */
+        struct task_struct *rcu_barrier_in_progress;
+                                                /* Task doing rcu_barrier(), */
+                                                /*  or NULL if no barrier. */
         raw_spinlock_t fqslock;                 /* Only one task forcing */
                                                 /*  quiescent states. */
         unsigned long jiffies_force_qs;         /* Time at which to invoke */