udp: Use hash2 for long hash1 chains in __udp*_lib_mcast_deliver.

Many multicast sources can have the same port which can result in a very
large list when hashing by port only. Hash by address and port instead
if this is the case. This makes multicast more similar to unicast.

On a 24-core machine receiving from 500 multicast sockets on the same
port, before this patch 80% of system CPU was used up by spin locking
and only ~25% of packets were successfully delivered.

With this patch, all packets are delivered and kernel overhead is ~8%
system CPU on spinlocks.

Signed-off-by: David Held <drheld@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

1 files changed, 14 insertions, 0 deletions

diff --git a/include/net/sock.h b/include/net/sock.h
index 29e48a6d1ded..28f734601b50 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -660,6 +660,20 @@ static inline void sk_add_bind_node(struct sock *sk,
 #define sk_for_each_bound(__sk, list) \
         hlist_for_each_entry(__sk, list, sk_bind_node)
 
+/**
+ * sk_nulls_for_each_entry_offset - iterate over a list at a given struct offset
+ * @tpos:       the type * to use as a loop cursor.
+ * @pos:        the &struct hlist_node to use as a loop cursor.
+ * @head:       the head for your list.
+ * @offset:     offset of hlist_node within the struct.
+ *
+ */
+#define sk_nulls_for_each_entry_offset(tpos, pos, head, offset)                \
+        for (pos = (head)->first;                                              \
+             (!is_a_nulls(pos)) &&                                             \
+                ({ tpos = (typeof(*tpos) *)((void *)pos - offset); 1;});       \
+             pos = pos->next)
+
 static inline struct user_namespace *sk_user_ns(struct sock *sk)
 {
         /* Careful only use this in a context where these parameters