rps: shortcut net_rps_action()

net_rps_action() is a bit expensive on NR_CPUS=64..4096 kernels, even if
RPS is not active.

Tom Herbert used two bitmasks to hold information needed to send IPI,
but a single LIFO list seems more appropriate.

Move all RPS logic into net_rps_action() to cleanup net_rx_action() code
(remove two ifdefs)

Move rps_remote_softirq_cpus into softnet_data to share its first cache
line, filling an existing hole.

In a future patch, we could call net_rps_action() from process_backlog()
to make sure we send IPI before handling this cpu backlog.

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

1 files changed, 6 insertions, 3 deletions

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 649a0252686e..83ab3da149ad 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1381,17 +1381,20 @@ static inline int unregister_gifconf(unsigned int family)
 }
 
 /*
- * Incoming packets are placed on per-cpu queues so that
- * no locking is needed.
+ * Incoming packets are placed on per-cpu queues
  */
 struct softnet_data {
         struct Qdisc            *output_queue;
         struct list_head        poll_list;
         struct sk_buff          *completion_queue;
 
-        /* Elements below can be accessed between CPUs for RPS */
 #ifdef CONFIG_RPS
+        struct softnet_data     *rps_ipi_list;
+
+        /* Elements below can be accessed between CPUs for RPS */
         struct call_single_data csd ____cacheline_aligned_in_smp;
+        struct softnet_data     *rps_ipi_next;
+        unsigned int            cpu;
         unsigned int            input_queue_head;
 #endif
         struct sk_buff_head     input_pkt_queue;