net: introduce DST_NOCACHE flag

While doing stress tests with IP route cache disabled, and multi queue
devices, I noticed a very high contention on one rwlock used in
neighbour code.

When many cpus are trying to send frames (possibly using a high
performance multiqueue device) to the same neighbour, they fight for the
neigh->lock rwlock in order to call neigh_hh_init(), and fight on
hh->hh_refcnt (a pair of atomic_inc/atomic_dec_and_test())

But we dont need to call neigh_hh_init() for dst that are used only
once. It costs four atomic operations at least, on two contended cache
lines, plus the high contention on neigh->lock rwlock.

Introduce a new dst flag, DST_NOCACHE, that is set when dst was not
inserted in route cache.

With the stress test bench, sending 160000000 frames on one neighbour,
results are :

Before patch:

real	2m28.406s
user	0m11.781s
sys	36m17.964s


After patch:

real	1m26.532s
user	0m12.185s
sys	20m3.903s

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

1 files changed, 5 insertions, 4 deletions

diff --git a/include/net/dst.h b/include/net/dst.h
index aa53fbc34b2b..a217c838ec0d 100644
--- a/include/net/dst.h
+++ b/include/net/dst.h
@@ -43,10 +43,11 @@ struct dst_entry {
         short                   error;
         short                   obsolete;
         int                     flags;
-#define DST_HOST                1
-#define DST_NOXFRM              2
-#define DST_NOPOLICY            4
-#define DST_NOHASH              8
+#define DST_HOST                0x0001
+#define DST_NOXFRM              0x0002
+#define DST_NOPOLICY            0x0004
+#define DST_NOHASH              0x0008
+#define DST_NOCACHE             0x0010
         unsigned long           expires;
 
         unsigned short          header_len;     /* more space at head required */