tcp: change tcp_skb_pcount() location

Our goal is to access no more than one cache line access per skb in
a write or receive queue when doing the various walks.

After recent TCP_SKB_CB() reorganizations, it is almost done.

Last part is tcp_skb_pcount() which currently uses
skb_shinfo(skb)->gso_segs, which is a terrible choice, because it needs
3 cache lines in current kernel (skb->head, skb->end, and
shinfo->gso_segs are all in 3 different cache lines, far from skb->cb)

This very simple patch reuses space currently taken by tcp_tw_isn
only in input path, as tcp_skb_pcount is only needed for skb stored in
write queue.

This considerably speeds up tcp_ack(), granted we avoid shinfo->tx_flags
to get SKBTX_ACK_TSTAMP, which seems possible.

This also speeds up all sack processing in general.

This speeds up tcp_sendmsg() because it no longer has to access/dirty
shinfo.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>

1 files changed, 21 insertions, 2 deletions

diff --git a/include/net/tcp.h b/include/net/tcp.h
index 4dc6641ee990..02a9a2c366bf 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -698,7 +698,16 @@ static inline u32 tcp_skb_timestamp(const struct sk_buff *skb)
 struct tcp_skb_cb {
         __u32           seq;            /* Starting sequence number     */
         __u32           end_seq;        /* SEQ + FIN + SYN + datalen    */
-        __u32           tcp_tw_isn;     /* isn chosen by tcp_timewait_state_process() */
+        union {
+                /* Note : tcp_tw_isn is used in input path only
+                 *        (isn chosen by tcp_timewait_state_process())
+                 *
+                 *        tcp_gso_segs is used in write queue only,
+                 *        cf tcp_skb_pcount()
+                 */
+                __u32           tcp_tw_isn;
+                __u32           tcp_gso_segs;
+        };
         __u8            tcp_flags;      /* TCP header flags. (tcp[13])  */
 
         __u8            sacked;         /* State flags for SACK/FACK.   */
@@ -746,7 +755,17 @@ TCP_ECN_create_request(struct request_sock *req, const struct sk_buff *skb,
  */
 static inline int tcp_skb_pcount(const struct sk_buff *skb)
 {
-        return skb_shinfo(skb)->gso_segs;
+        return TCP_SKB_CB(skb)->tcp_gso_segs;
+}
+
+static inline void tcp_skb_pcount_set(struct sk_buff *skb, int segs)
+{
+        TCP_SKB_CB(skb)->tcp_gso_segs = segs;
+}
+
+static inline void tcp_skb_pcount_add(struct sk_buff *skb, int segs)
+{
+        TCP_SKB_CB(skb)->tcp_gso_segs += segs;
 }
 
 /* This is valid iff tcp_skb_pcount() > 1. */