net: speedup udp receive path

Since commit 95766fff ([UDP]: Add memory accounting.), 
each received packet needs one extra sock_lock()/sock_release() pair.

This added latency because of possible backlog handling. Then later,
ticket spinlocks added yet another latency source in case of DDOS.

This patch introduces lock_sock_bh() and unlock_sock_bh()
synchronization primitives, avoiding one atomic operation and backlog
processing.

skb_free_datagram_locked() uses them instead of full blown
lock_sock()/release_sock(). skb is orphaned inside locked section for
proper socket memory reclaim, and finally freed outside of it.

UDP receive path now take the socket spinlock only once.

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

1 files changed, 7 insertions, 3 deletions

diff --git a/net/core/datagram.c b/net/core/datagram.c
index 5574a5ddf908..95b851f3d713 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -229,9 +229,13 @@ EXPORT_SYMBOL(skb_free_datagram);
 
 void skb_free_datagram_locked(struct sock *sk, struct sk_buff *skb)
 {
-        lock_sock(sk);
-        skb_free_datagram(sk, skb);
-        release_sock(sk);
+        lock_sock_bh(sk);
+        skb_orphan(skb);
+        sk_mem_reclaim_partial(sk);
+        unlock_sock_bh(sk);
+
+        /* skb is now orphaned, might be freed outside of locked section */
+        consume_skb(skb);
 }
 EXPORT_SYMBOL(skb_free_datagram_locked);