aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorDavid Howells <dhowells@redhat.com>2016-09-08 06:10:12 -0400
committerDavid Howells <dhowells@redhat.com>2016-09-08 06:10:12 -0400
commit248f219cb8bcbfbd7f132752d44afa2df7c241d1 (patch)
tree6961c9529a7fe0e36a9d59805872283308087720
parent00e907127e6f86d0f9b122d9b4347a8aa09a8b61 (diff)
rxrpc: Rewrite the data and ack handling code
Rewrite the data and ack handling code such that: (1) Parsing of received ACK and ABORT packets and the distribution and the filing of DATA packets happens entirely within the data_ready context called from the UDP socket. This allows us to process and discard ACK and ABORT packets much more quickly (they're no longer stashed on a queue for a background thread to process). (2) We avoid calling skb_clone(), pskb_pull() and pskb_trim(). We instead keep track of the offset and length of the content of each packet in the sk_buff metadata. This means we don't do any allocation in the receive path. (3) Jumbo DATA packet parsing is now done in data_ready context. Rather than cloning the packet once for each subpacket and pulling/trimming it, we file the packet multiple times with an annotation for each indicating which subpacket is there. From that we can directly calculate the offset and length. (4) A call's receive queue can be accessed without taking locks (memory barriers do have to be used, though). (5) Incoming calls are set up from preallocated resources and immediately made live. They can than have packets queued upon them and ACKs generated. If insufficient resources exist, DATA packet #1 is given a BUSY reply and other DATA packets are discarded). (6) sk_buffs no longer take a ref on their parent call. To make this work, the following changes are made: (1) Each call's receive buffer is now a circular buffer of sk_buff pointers (rxtx_buffer) rather than a number of sk_buff_heads spread between the call and the socket. This permits each sk_buff to be in the buffer multiple times. The receive buffer is reused for the transmit buffer. (2) A circular buffer of annotations (rxtx_annotations) is kept parallel to the data buffer. Transmission phase annotations indicate whether a buffered packet has been ACK'd or not and whether it needs retransmission. Receive phase annotations indicate whether a slot holds a whole packet or a jumbo subpacket and, if the latter, which subpacket. They also note whether the packet has been decrypted in place. (3) DATA packet window tracking is much simplified. Each phase has just two numbers representing the window (rx_hard_ack/rx_top and tx_hard_ack/tx_top). The hard_ack number is the sequence number before base of the window, representing the last packet the other side says it has consumed. hard_ack starts from 0 and the first packet is sequence number 1. The top number is the sequence number of the highest-numbered packet residing in the buffer. Packets between hard_ack+1 and top are soft-ACK'd to indicate they've been received, but not yet consumed. Four macros, before(), before_eq(), after() and after_eq() are added to compare sequence numbers within the window. This allows for the top of the window to wrap when the hard-ack sequence number gets close to the limit. Two flags, RXRPC_CALL_RX_LAST and RXRPC_CALL_TX_LAST, are added also to indicate when rx_top and tx_top point at the packets with the LAST_PACKET bit set, indicating the end of the phase. (4) Calls are queued on the socket 'receive queue' rather than packets. This means that we don't need have to invent dummy packets to queue to indicate abnormal/terminal states and we don't have to keep metadata packets (such as ABORTs) around (5) The offset and length of a (sub)packet's content are now passed to the verify_packet security op. This is currently expected to decrypt the packet in place and validate it. However, there's now nowhere to store the revised offset and length of the actual data within the decrypted blob (there may be a header and padding to skip) because an sk_buff may represent multiple packets, so a locate_data security op is added to retrieve these details from the sk_buff content when needed. (6) recvmsg() now has to handle jumbo subpackets, where each subpacket is individually secured and needs to be individually decrypted. The code to do this is broken out into rxrpc_recvmsg_data() and shared with the kernel API. It now iterates over the call's receive buffer rather than walking the socket receive queue. Additional changes: (1) The timers are condensed to a single timer that is set for the soonest of three timeouts (delayed ACK generation, DATA retransmission and call lifespan). (2) Transmission of ACK and ABORT packets is effected immediately from process-context socket ops/kernel API calls that cause them instead of them being punted off to a background work item. The data_ready handler still has to defer to the background, though. (3) A shutdown op is added to the AF_RXRPC socket so that the AFS filesystem can shut down the socket and flush its own work items before closing the socket to deal with any in-progress service calls. Future additional changes that will need to be considered: (1) Make sure that a call doesn't hog the front of the queue by receiving data from the network as fast as userspace is consuming it to the exclusion of other calls. (2) Transmit delayed ACKs from within recvmsg() when we've consumed sufficiently more packets to avoid the background work item needing to run. Signed-off-by: David Howells <dhowells@redhat.com>
Diffstat
-rw-r--r--fs/afs/rxrpc.c51
-rw-r--r--include/net/af_rxrpc.h3
-rw-r--r--include/rxrpc/packet.h7
-rw-r--r--net/rxrpc/af_rxrpc.c57
-rw-r--r--net/rxrpc/ar-internal.h177
-rw-r--r--net/rxrpc/call_accept.c472
-rw-r--r--net/rxrpc/call_event.c1357
-rw-r--r--net/rxrpc/call_object.c535
-rw-r--r--net/rxrpc/conn_event.c137
-rw-r--r--net/rxrpc/conn_object.c6
-rw-r--r--net/rxrpc/conn_service.c101
-rw-r--r--net/rxrpc/input.c1044
-rw-r--r--net/rxrpc/insecure.c13
-rw-r--r--net/rxrpc/local_event.c2
-rw-r--r--net/rxrpc/local_object.c7
-rw-r--r--net/rxrpc/misc.c2
-rw-r--r--net/rxrpc/output.c125
-rw-r--r--net/rxrpc/peer_event.c17
-rw-r--r--net/rxrpc/peer_object.c82
-rw-r--r--net/rxrpc/recvmsg.c764
-rw-r--r--net/rxrpc/rxkad.c108
-rw-r--r--net/rxrpc/security.c10
-rw-r--r--net/rxrpc/sendmsg.c126
-rw-r--r--net/rxrpc/skbuff.c127
24 files changed, 1993 insertions, 3337 deletions
diff --git a/fs/afs/rxrpc.c b/fs/afs/rxrpc.c
index 720ef05a24fe..59bdaa7527b6 100644
--- a/fs/afs/rxrpc.c
+++ b/fs/afs/rxrpc.c
@@ -55,10 +55,8 @@ static const struct afs_call_type afs_RXCMxxxx = {
55 .abort_to_error = afs_abort_to_error, 55 .abort_to_error = afs_abort_to_error,
56}; 56};
57 57
58static void afs_collect_incoming_call(struct work_struct *);
59static void afs_charge_preallocation(struct work_struct *); 58static void afs_charge_preallocation(struct work_struct *);
60 59
61static DECLARE_WORK(afs_collect_incoming_call_work, afs_collect_incoming_call);
62static DECLARE_WORK(afs_charge_preallocation_work, afs_charge_preallocation); 60static DECLARE_WORK(afs_charge_preallocation_work, afs_charge_preallocation);
63 61
64static int afs_wait_atomic_t(atomic_t *p) 62static int afs_wait_atomic_t(atomic_t *p)
@@ -144,6 +142,8 @@ void afs_close_socket(void)
144 _debug("no outstanding calls"); 142 _debug("no outstanding calls");
145 143
146 flush_workqueue(afs_async_calls); 144 flush_workqueue(afs_async_calls);
145 kernel_sock_shutdown(afs_socket, SHUT_RDWR);
146 flush_workqueue(afs_async_calls);
147 sock_release(afs_socket); 147 sock_release(afs_socket);
148 148
149 _debug("dework"); 149 _debug("dework");
@@ -602,51 +602,6 @@ static void afs_process_async_call(struct work_struct *work)
602 _leave(""); 602 _leave("");
603} 603}
604 604
605/*
606 * accept the backlog of incoming calls
607 */
608static void afs_collect_incoming_call(struct work_struct *work)
609{
610 struct rxrpc_call *rxcall;
611 struct afs_call *call = NULL;
612
613 _enter("");
614
615 do {
616 if (!call) {
617 call = kzalloc(sizeof(struct afs_call), GFP_KERNEL);
618 if (!call) {
619 rxrpc_kernel_reject_call(afs_socket);
620 return;
621 }
622
623 INIT_WORK(&call->async_work, afs_process_async_call);
624 call->wait_mode = &afs_async_incoming_call;
625 call->type = &afs_RXCMxxxx;
626 init_waitqueue_head(&call->waitq);
627 call->state = AFS_CALL_AWAIT_OP_ID;
628
629 _debug("CALL %p{%s} [%d]",
630 call, call->type->name,
631 atomic_read(&afs_outstanding_calls));
632 atomic_inc(&afs_outstanding_calls);
633 }
634
635 rxcall = rxrpc_kernel_accept_call(afs_socket,
636 (unsigned long)call,
637 afs_wake_up_async_call);
638 if (!IS_ERR(rxcall)) {
639 call->rxcall = rxcall;
640 call->need_attention = true;
641 queue_work(afs_async_calls, &call->async_work);
642 call = NULL;
643 }
644 } while (!call);
645
646 if (call)
647 afs_free_call(call);
648}
649
650static void afs_rx_attach(struct rxrpc_call *rxcall, unsigned long user_call_ID) 605static void afs_rx_attach(struct rxrpc_call *rxcall, unsigned long user_call_ID)
651{ 606{
652 struct afs_call *call = (struct afs_call *)user_call_ID; 607 struct afs_call *call = (struct afs_call *)user_call_ID;
@@ -704,7 +659,7 @@ static void afs_rx_discard_new_call(struct rxrpc_call *rxcall,
704static void afs_rx_new_call(struct sock *sk, struct rxrpc_call *rxcall, 659static void afs_rx_new_call(struct sock *sk, struct rxrpc_call *rxcall,
705 unsigned long user_call_ID) 660 unsigned long user_call_ID)
706{ 661{
707 queue_work(afs_wq, &afs_collect_incoming_call_work); 662 atomic_inc(&afs_outstanding_calls);
708 queue_work(afs_wq, &afs_charge_preallocation_work); 663 queue_work(afs_wq, &afs_charge_preallocation_work);
709} 664}
710 665
diff --git a/include/net/af_rxrpc.h b/include/net/af_rxrpc.h
index 9cf551be916b..1061a472a3e3 100644
--- a/include/net/af_rxrpc.h
+++ b/include/net/af_rxrpc.h
@@ -42,9 +42,6 @@ int rxrpc_kernel_recv_data(struct socket *, struct rxrpc_call *,
42void rxrpc_kernel_abort_call(struct socket *, struct rxrpc_call *, 42void rxrpc_kernel_abort_call(struct socket *, struct rxrpc_call *,
43 u32, int, const char *); 43 u32, int, const char *);
44void rxrpc_kernel_end_call(struct socket *, struct rxrpc_call *); 44void rxrpc_kernel_end_call(struct socket *, struct rxrpc_call *);
45struct rxrpc_call *rxrpc_kernel_accept_call(struct socket *, unsigned long,
46 rxrpc_notify_rx_t);
47int rxrpc_kernel_reject_call(struct socket *);
48void rxrpc_kernel_get_peer(struct socket *, struct rxrpc_call *, 45void rxrpc_kernel_get_peer(struct socket *, struct rxrpc_call *,
49 struct sockaddr_rxrpc *); 46 struct sockaddr_rxrpc *);
50int rxrpc_kernel_charge_accept(struct socket *, rxrpc_notify_rx_t, 47int rxrpc_kernel_charge_accept(struct socket *, rxrpc_notify_rx_t,
diff --git a/include/rxrpc/packet.h b/include/rxrpc/packet.h
index b0ae5c1a6ce6..fd6eb3a60a8c 100644
--- a/include/rxrpc/packet.h
+++ b/include/rxrpc/packet.h
@@ -133,6 +133,13 @@ struct rxrpc_ackpacket {
133 133
134} __packed; 134} __packed;
135 135
136/* Some ACKs refer to specific packets and some are general and can be updated. */
137#define RXRPC_ACK_UPDATEABLE ((1 << RXRPC_ACK_REQUESTED) | \
138 (1 << RXRPC_ACK_PING_RESPONSE) | \
139 (1 << RXRPC_ACK_DELAY) | \
140 (1 << RXRPC_ACK_IDLE))
141
142
136/* 143/*
137 * ACK packets can have a further piece of information tagged on the end 144 * ACK packets can have a further piece of information tagged on the end
138 */ 145 */
diff --git a/net/rxrpc/af_rxrpc.c b/net/rxrpc/af_rxrpc.c
index 1e8cf3ded81f..caa226dd436e 100644
--- a/net/rxrpc/af_rxrpc.c
+++ b/net/rxrpc/af_rxrpc.c
@@ -155,7 +155,7 @@ static int rxrpc_bind(struct socket *sock, struct sockaddr *saddr, int len)
155 } 155 }
156 156
157 if (rx->srx.srx_service) { 157 if (rx->srx.srx_service) {
158 write_lock_bh(&local->services_lock); 158 write_lock(&local->services_lock);
159 hlist_for_each_entry(prx, &local->services, listen_link) { 159 hlist_for_each_entry(prx, &local->services, listen_link) {
160 if (prx->srx.srx_service == rx->srx.srx_service) 160 if (prx->srx.srx_service == rx->srx.srx_service)
161 goto service_in_use; 161 goto service_in_use;
@@ -163,7 +163,7 @@ static int rxrpc_bind(struct socket *sock, struct sockaddr *saddr, int len)
163 163
164 rx->local = local; 164 rx->local = local;
165 hlist_add_head_rcu(&rx->listen_link, &local->services); 165 hlist_add_head_rcu(&rx->listen_link, &local->services);
166 write_unlock_bh(&local->services_lock); 166 write_unlock(&local->services_lock);
167 167
168 rx->sk.sk_state = RXRPC_SERVER_BOUND; 168 rx->sk.sk_state = RXRPC_SERVER_BOUND;
169 } else { 169 } else {
@@ -176,7 +176,7 @@ static int rxrpc_bind(struct socket *sock, struct sockaddr *saddr, int len)
176 return 0; 176 return 0;
177 177
178service_in_use: 178service_in_use:
179 write_unlock_bh(&local->services_lock); 179 write_unlock(&local->services_lock);
180 rxrpc_put_local(local); 180 rxrpc_put_local(local);
181 ret = -EADDRINUSE; 181 ret = -EADDRINUSE;
182error_unlock: 182error_unlock:
@@ -515,15 +515,16 @@ error:
515static unsigned int rxrpc_poll(struct file *file, struct socket *sock, 515static unsigned int rxrpc_poll(struct file *file, struct socket *sock,
516 poll_table *wait) 516 poll_table *wait)
517{ 517{
518 unsigned int mask;
519 struct sock *sk = sock->sk; 518 struct sock *sk = sock->sk;
519 struct rxrpc_sock *rx = rxrpc_sk(sk);
520 unsigned int mask;
520 521
521 sock_poll_wait(file, sk_sleep(sk), wait); 522 sock_poll_wait(file, sk_sleep(sk), wait);
522 mask = 0; 523 mask = 0;
523 524
524 /* the socket is readable if there are any messages waiting on the Rx 525 /* the socket is readable if there are any messages waiting on the Rx
525 * queue */ 526 * queue */
526 if (!skb_queue_empty(&sk->sk_receive_queue)) 527 if (!list_empty(&rx->recvmsg_q))
527 mask |= POLLIN | POLLRDNORM; 528 mask |= POLLIN | POLLRDNORM;
528 529
529 /* the socket is writable if there is space to add new data to the 530 /* the socket is writable if there is space to add new data to the
@@ -575,8 +576,11 @@ static int rxrpc_create(struct net *net, struct socket *sock, int protocol,
575 rx->calls = RB_ROOT; 576 rx->calls = RB_ROOT;
576 577
577 INIT_HLIST_NODE(&rx->listen_link); 578 INIT_HLIST_NODE(&rx->listen_link);
578 INIT_LIST_HEAD(&rx->secureq); 579 spin_lock_init(&rx->incoming_lock);
579 INIT_LIST_HEAD(&rx->acceptq); 580 INIT_LIST_HEAD(&rx->sock_calls);
581 INIT_LIST_HEAD(&rx->to_be_accepted);
582 INIT_LIST_HEAD(&rx->recvmsg_q);
583 rwlock_init(&rx->recvmsg_lock);
580 rwlock_init(&rx->call_lock); 584 rwlock_init(&rx->call_lock);
581 memset(&rx->srx, 0, sizeof(rx->srx)); 585 memset(&rx->srx, 0, sizeof(rx->srx));
582 586
@@ -585,6 +589,39 @@ static int rxrpc_create(struct net *net, struct socket *sock, int protocol,
585} 589}
586 590
587/* 591/*
592 * Kill all the calls on a socket and shut it down.
593 */
594static int rxrpc_shutdown(struct socket *sock, int flags)
595{
596 struct sock *sk = sock->sk;
597 struct rxrpc_sock *rx = rxrpc_sk(sk);
598 int ret = 0;
599
600 _enter("%p,%d", sk, flags);
601
602 if (flags != SHUT_RDWR)
603 return -EOPNOTSUPP;
604 if (sk->sk_state == RXRPC_CLOSE)
605 return -ESHUTDOWN;
606
607 lock_sock(sk);
608
609 spin_lock_bh(&sk->sk_receive_queue.lock);
610 if (sk->sk_state < RXRPC_CLOSE) {
611 sk->sk_state = RXRPC_CLOSE;
612 sk->sk_shutdown = SHUTDOWN_MASK;
613 } else {
614 ret = -ESHUTDOWN;
615 }
616 spin_unlock_bh(&sk->sk_receive_queue.lock);
617
618 rxrpc_discard_prealloc(rx);
619
620 release_sock(sk);
621 return ret;
622}
623
624/*
588 * RxRPC socket destructor 625 * RxRPC socket destructor
589 */ 626 */
590static void rxrpc_sock_destructor(struct sock *sk) 627static void rxrpc_sock_destructor(struct sock *sk)
@@ -623,9 +660,9 @@ static int rxrpc_release_sock(struct sock *sk)
623 ASSERTCMP(rx->listen_link.next, !=, LIST_POISON1); 660 ASSERTCMP(rx->listen_link.next, !=, LIST_POISON1);
624 661
625 if (!hlist_unhashed(&rx->listen_link)) { 662 if (!hlist_unhashed(&rx->listen_link)) {
626 write_lock_bh(&rx->local->services_lock); 663 write_lock(&rx->local->services_lock);
627 hlist_del_rcu(&rx->listen_link); 664 hlist_del_rcu(&rx->listen_link);
628 write_unlock_bh(&rx->local->services_lock); 665 write_unlock(&rx->local->services_lock);
629 } 666 }
630 667
631 /* try to flush out this socket */ 668 /* try to flush out this socket */
@@ -678,7 +715,7 @@ static const struct proto_ops rxrpc_rpc_ops = {
678 .poll = rxrpc_poll, 715 .poll = rxrpc_poll,
679 .ioctl = sock_no_ioctl, 716 .ioctl = sock_no_ioctl,
680 .listen = rxrpc_listen, 717 .listen = rxrpc_listen,
681 .shutdown = sock_no_shutdown, 718 .shutdown = rxrpc_shutdown,
682 .setsockopt = rxrpc_setsockopt, 719 .setsockopt = rxrpc_setsockopt,
683 .getsockopt = sock_no_getsockopt, 720 .getsockopt = sock_no_getsockopt,
684 .sendmsg = rxrpc_sendmsg, 721 .sendmsg = rxrpc_sendmsg,
diff --git a/net/rxrpc/ar-internal.h b/net/rxrpc/ar-internal.h
index 45e1c269f90e..b1cb79ec4e96 100644
--- a/net/rxrpc/ar-internal.h
+++ b/net/rxrpc/ar-internal.h
@@ -94,9 +94,12 @@ struct rxrpc_sock {
94 rxrpc_discard_new_call_t discard_new_call; /* Func to discard a new call */ 94 rxrpc_discard_new_call_t discard_new_call; /* Func to discard a new call */
95 struct rxrpc_local *local; /* local endpoint */ 95 struct rxrpc_local *local; /* local endpoint */
96 struct hlist_node listen_link; /* link in the local endpoint's listen list */ 96 struct hlist_node listen_link; /* link in the local endpoint's listen list */
97 struct list_head secureq; /* calls awaiting connection security clearance */
98 struct list_head acceptq; /* calls awaiting acceptance */
99 struct rxrpc_backlog *backlog; /* Preallocation for services */ 97 struct rxrpc_backlog *backlog; /* Preallocation for services */
98 spinlock_t incoming_lock; /* Incoming call vs service shutdown lock */
99 struct list_head sock_calls; /* List of calls owned by this socket */
100 struct list_head to_be_accepted; /* calls awaiting acceptance */
101 struct list_head recvmsg_q; /* Calls awaiting recvmsg's attention */
102 rwlock_t recvmsg_lock; /* Lock for recvmsg_q */
100 struct key *key; /* security for this socket */ 103 struct key *key; /* security for this socket */
101 struct key *securities; /* list of server security descriptors */ 104 struct key *securities; /* list of server security descriptors */
102 struct rb_root calls; /* User ID -> call mapping */ 105 struct rb_root calls; /* User ID -> call mapping */
@@ -138,13 +141,16 @@ struct rxrpc_host_header {
138 * - max 48 bytes (struct sk_buff::cb) 141 * - max 48 bytes (struct sk_buff::cb)
139 */ 142 */
140struct rxrpc_skb_priv { 143struct rxrpc_skb_priv {
141 struct rxrpc_call *call; /* call with which associated */ 144 union {
142 unsigned long resend_at; /* time in jiffies at which to resend */ 145 unsigned long resend_at; /* time in jiffies at which to resend */
146 struct {
147 u8 nr_jumbo; /* Number of jumbo subpackets */
148 };
149 };
143 union { 150 union {
144 unsigned int offset; /* offset into buffer of next read */ 151 unsigned int offset; /* offset into buffer of next read */
145 int remain; /* amount of space remaining for next write */ 152 int remain; /* amount of space remaining for next write */
146 u32 error; /* network error code */ 153 u32 error; /* network error code */
147 bool need_resend; /* T if needs resending */
148 }; 154 };
149 155
150 struct rxrpc_host_header hdr; /* RxRPC packet header from this packet */ 156 struct rxrpc_host_header hdr; /* RxRPC packet header from this packet */
@@ -179,7 +185,11 @@ struct rxrpc_security {
179 185
180 /* verify the security on a received packet */ 186 /* verify the security on a received packet */
181 int (*verify_packet)(struct rxrpc_call *, struct sk_buff *, 187 int (*verify_packet)(struct rxrpc_call *, struct sk_buff *,
182 rxrpc_seq_t, u16); 188 unsigned int, unsigned int, rxrpc_seq_t, u16);
189
190 /* Locate the data in a received packet that has been verified. */
191 void (*locate_data)(struct rxrpc_call *, struct sk_buff *,
192 unsigned int *, unsigned int *);
183 193
184 /* issue a challenge */ 194 /* issue a challenge */
185 int (*issue_challenge)(struct rxrpc_connection *); 195 int (*issue_challenge)(struct rxrpc_connection *);
@@ -211,7 +221,6 @@ struct rxrpc_local {
211 struct work_struct processor; 221 struct work_struct processor;
212 struct hlist_head services; /* services listening on this endpoint */ 222 struct hlist_head services; /* services listening on this endpoint */
213 struct rw_semaphore defrag_sem; /* control re-enablement of IP DF bit */ 223 struct rw_semaphore defrag_sem; /* control re-enablement of IP DF bit */
214 struct sk_buff_head accept_queue; /* incoming calls awaiting acceptance */
215 struct sk_buff_head reject_queue; /* packets awaiting rejection */ 224 struct sk_buff_head reject_queue; /* packets awaiting rejection */
216 struct sk_buff_head event_queue; /* endpoint event packets awaiting processing */ 225 struct sk_buff_head event_queue; /* endpoint event packets awaiting processing */
217 struct rb_root client_conns; /* Client connections by socket params */ 226 struct rb_root client_conns; /* Client connections by socket params */
@@ -388,38 +397,21 @@ struct rxrpc_connection {
388 */ 397 */
389enum rxrpc_call_flag { 398enum rxrpc_call_flag {
390 RXRPC_CALL_RELEASED, /* call has been released - no more message to userspace */ 399 RXRPC_CALL_RELEASED, /* call has been released - no more message to userspace */
391 RXRPC_CALL_TERMINAL_MSG, /* call has given the socket its final message */
392 RXRPC_CALL_RCVD_LAST, /* all packets received */
393 RXRPC_CALL_RUN_RTIMER, /* Tx resend timer started */
394 RXRPC_CALL_TX_SOFT_ACK, /* sent some soft ACKs */
395 RXRPC_CALL_INIT_ACCEPT, /* acceptance was initiated */
396 RXRPC_CALL_HAS_USERID, /* has a user ID attached */ 400 RXRPC_CALL_HAS_USERID, /* has a user ID attached */
397 RXRPC_CALL_EXPECT_OOS, /* expect out of sequence packets */
398 RXRPC_CALL_IS_SERVICE, /* Call is service call */ 401 RXRPC_CALL_IS_SERVICE, /* Call is service call */
399 RXRPC_CALL_EXPOSED, /* The call was exposed to the world */ 402 RXRPC_CALL_EXPOSED, /* The call was exposed to the world */
400 RXRPC_CALL_RX_NO_MORE, /* Don't indicate MSG_MORE from recvmsg() */ 403 RXRPC_CALL_RX_LAST, /* Received the last packet (at rxtx_top) */
404 RXRPC_CALL_TX_LAST, /* Last packet in Tx buffer (at rxtx_top) */
401}; 405};
402 406
403/* 407/*
404 * Events that can be raised on a call. 408 * Events that can be raised on a call.
405 */ 409 */
406enum rxrpc_call_event { 410enum rxrpc_call_event {
407 RXRPC_CALL_EV_RCVD_ACKALL, /* ACKALL or reply received */
408 RXRPC_CALL_EV_RCVD_BUSY, /* busy packet received */
409 RXRPC_CALL_EV_RCVD_ABORT, /* abort packet received */
410 RXRPC_CALL_EV_RCVD_ERROR, /* network error received */
411 RXRPC_CALL_EV_ACK_FINAL, /* need to generate final ACK (and release call) */
412 RXRPC_CALL_EV_ACK, /* need to generate ACK */ 411 RXRPC_CALL_EV_ACK, /* need to generate ACK */
413 RXRPC_CALL_EV_REJECT_BUSY, /* need to generate busy message */
414 RXRPC_CALL_EV_ABORT, /* need to generate abort */ 412 RXRPC_CALL_EV_ABORT, /* need to generate abort */
415 RXRPC_CALL_EV_CONN_ABORT, /* local connection abort generated */ 413 RXRPC_CALL_EV_TIMER, /* Timer expired */
416 RXRPC_CALL_EV_RESEND_TIMER, /* Tx resend timer expired */
417 RXRPC_CALL_EV_RESEND, /* Tx resend required */ 414 RXRPC_CALL_EV_RESEND, /* Tx resend required */
418 RXRPC_CALL_EV_DRAIN_RX_OOS, /* drain the Rx out of sequence queue */
419 RXRPC_CALL_EV_LIFE_TIMER, /* call's lifetimer ran out */
420 RXRPC_CALL_EV_ACCEPTED, /* incoming call accepted by userspace app */
421 RXRPC_CALL_EV_SECURED, /* incoming call's connection is now secure */
422 RXRPC_CALL_EV_POST_ACCEPT, /* need to post an "accept?" message to the app */
423}; 415};
424 416
425/* 417/*
@@ -431,7 +423,6 @@ enum rxrpc_call_state {
431 RXRPC_CALL_CLIENT_SEND_REQUEST, /* - client sending request phase */ 423 RXRPC_CALL_CLIENT_SEND_REQUEST, /* - client sending request phase */
432 RXRPC_CALL_CLIENT_AWAIT_REPLY, /* - client awaiting reply */ 424 RXRPC_CALL_CLIENT_AWAIT_REPLY, /* - client awaiting reply */
433 RXRPC_CALL_CLIENT_RECV_REPLY, /* - client receiving reply phase */ 425 RXRPC_CALL_CLIENT_RECV_REPLY, /* - client receiving reply phase */
434 RXRPC_CALL_CLIENT_FINAL_ACK, /* - client sending final ACK phase */
435 RXRPC_CALL_SERVER_PREALLOC, /* - service preallocation */ 426 RXRPC_CALL_SERVER_PREALLOC, /* - service preallocation */
436 RXRPC_CALL_SERVER_SECURING, /* - server securing request connection */ 427 RXRPC_CALL_SERVER_SECURING, /* - server securing request connection */
437 RXRPC_CALL_SERVER_ACCEPTING, /* - server accepting request */ 428 RXRPC_CALL_SERVER_ACCEPTING, /* - server accepting request */
@@ -448,7 +439,6 @@ enum rxrpc_call_state {
448 */ 439 */
449enum rxrpc_call_completion { 440enum rxrpc_call_completion {
450 RXRPC_CALL_SUCCEEDED, /* - Normal termination */ 441 RXRPC_CALL_SUCCEEDED, /* - Normal termination */
451 RXRPC_CALL_SERVER_BUSY, /* - call rejected by busy server */
452 RXRPC_CALL_REMOTELY_ABORTED, /* - call aborted by peer */ 442 RXRPC_CALL_REMOTELY_ABORTED, /* - call aborted by peer */
453 RXRPC_CALL_LOCALLY_ABORTED, /* - call aborted locally on error or close */ 443 RXRPC_CALL_LOCALLY_ABORTED, /* - call aborted locally on error or close */
454 RXRPC_CALL_LOCAL_ERROR, /* - call failed due to local error */ 444 RXRPC_CALL_LOCAL_ERROR, /* - call failed due to local error */
@@ -465,24 +455,23 @@ struct rxrpc_call {
465 struct rxrpc_connection *conn; /* connection carrying call */ 455 struct rxrpc_connection *conn; /* connection carrying call */
466 struct rxrpc_peer *peer; /* Peer record for remote address */ 456 struct rxrpc_peer *peer; /* Peer record for remote address */
467 struct rxrpc_sock __rcu *socket; /* socket responsible */ 457 struct rxrpc_sock __rcu *socket; /* socket responsible */
468 struct timer_list lifetimer; /* lifetime remaining on call */ 458 unsigned long ack_at; /* When deferred ACK needs to happen */
469 struct timer_list ack_timer; /* ACK generation timer */ 459 unsigned long resend_at; /* When next resend needs to happen */
470 struct timer_list resend_timer; /* Tx resend timer */ 460 unsigned long expire_at; /* When the call times out */
471 struct work_struct processor; /* packet processor and ACK generator */ 461 struct timer_list timer; /* Combined event timer */
462 struct work_struct processor; /* Event processor */
472 rxrpc_notify_rx_t notify_rx; /* kernel service Rx notification function */ 463 rxrpc_notify_rx_t notify_rx; /* kernel service Rx notification function */
473 struct list_head link; /* link in master call list */ 464 struct list_head link; /* link in master call list */
474 struct list_head chan_wait_link; /* Link in conn->waiting_calls */ 465 struct list_head chan_wait_link; /* Link in conn->waiting_calls */
475 struct hlist_node error_link; /* link in error distribution list */ 466 struct hlist_node error_link; /* link in error distribution list */
476 struct list_head accept_link; /* calls awaiting acceptance */ 467 struct list_head accept_link; /* Link in rx->acceptq */
477 struct rb_node sock_node; /* node in socket call tree */ 468 struct list_head recvmsg_link; /* Link in rx->recvmsg_q */
478 struct sk_buff_head rx_queue; /* received packets */ 469 struct list_head sock_link; /* Link in rx->sock_calls */
479 struct sk_buff_head rx_oos_queue; /* packets received out of sequence */ 470 struct rb_node sock_node; /* Node in rx->calls */
480 struct sk_buff_head knlrecv_queue; /* Queue for kernel_recv [TODO: replace this] */
481 struct sk_buff *tx_pending; /* Tx socket buffer being filled */ 471 struct sk_buff *tx_pending; /* Tx socket buffer being filled */
482 wait_queue_head_t waitq; /* Wait queue for channel or Tx */ 472 wait_queue_head_t waitq; /* Wait queue for channel or Tx */
483 __be32 crypto_buf[2]; /* Temporary packet crypto buffer */ 473 __be32 crypto_buf[2]; /* Temporary packet crypto buffer */
484 unsigned long user_call_ID; /* user-defined call ID */ 474 unsigned long user_call_ID; /* user-defined call ID */
485 unsigned long creation_jif; /* time of call creation */
486 unsigned long flags; 475 unsigned long flags;
487 unsigned long events; 476 unsigned long events;
488 spinlock_t lock; 477 spinlock_t lock;
@@ -492,40 +481,55 @@ struct rxrpc_call {
492 enum rxrpc_call_state state; /* current state of call */ 481 enum rxrpc_call_state state; /* current state of call */
493 enum rxrpc_call_completion completion; /* Call completion condition */ 482 enum rxrpc_call_completion completion; /* Call completion condition */
494 atomic_t usage; 483 atomic_t usage;
495 atomic_t sequence; /* Tx data packet sequence counter */
496 u16 service_id; /* service ID */ 484 u16 service_id; /* service ID */
497 u8 security_ix; /* Security type */ 485 u8 security_ix; /* Security type */
498 u32 call_id; /* call ID on connection */ 486 u32 call_id; /* call ID on connection */
499 u32 cid; /* connection ID plus channel index */ 487 u32 cid; /* connection ID plus channel index */
500 int debug_id; /* debug ID for printks */ 488 int debug_id; /* debug ID for printks */
501 489
502 /* transmission-phase ACK management */ 490 /* Rx/Tx circular buffer, depending on phase.
503 u8 acks_head; /* offset into window of first entry */ 491 *
504 u8 acks_tail; /* offset into window of last entry */ 492 * In the Rx phase, packets are annotated with 0 or the number of the
505 u8 acks_winsz; /* size of un-ACK'd window */ 493 * segment of a jumbo packet each buffer refers to. There can be up to
506 u8 acks_unacked; /* lowest unacked packet in last ACK received */ 494 * 47 segments in a maximum-size UDP packet.
507 int acks_latest; /* serial number of latest ACK received */ 495 *
508 rxrpc_seq_t acks_hard; /* highest definitively ACK'd msg seq */ 496 * In the Tx phase, packets are annotated with which buffers have been
509 unsigned long *acks_window; /* sent packet window 497 * acked.
510 * - elements are pointers with LSB set if ACK'd 498 */
499#define RXRPC_RXTX_BUFF_SIZE 64
500#define RXRPC_RXTX_BUFF_MASK (RXRPC_RXTX_BUFF_SIZE - 1)
501 struct sk_buff **rxtx_buffer;
502 u8 *rxtx_annotations;
503#define RXRPC_TX_ANNO_ACK 0
504#define RXRPC_TX_ANNO_UNACK 1
505#define RXRPC_TX_ANNO_NAK 2
506#define RXRPC_TX_ANNO_RETRANS 3
507#define RXRPC_RX_ANNO_JUMBO 0x3f /* Jumbo subpacket number + 1 if not zero */
508#define RXRPC_RX_ANNO_JLAST 0x40 /* Set if last element of a jumbo packet */
509#define RXRPC_RX_ANNO_VERIFIED 0x80 /* Set if verified and decrypted */
510 rxrpc_seq_t tx_hard_ack; /* Dead slot in buffer; the first transmitted but
511 * not hard-ACK'd packet follows this.
512 */
513 rxrpc_seq_t tx_top; /* Highest Tx slot allocated. */
514 rxrpc_seq_t rx_hard_ack; /* Dead slot in buffer; the first received but not
515 * consumed packet follows this.
511 */ 516 */
517 rxrpc_seq_t rx_top; /* Highest Rx slot allocated. */
518 rxrpc_seq_t rx_expect_next; /* Expected next packet sequence number */
519 u8 rx_winsize; /* Size of Rx window */
520 u8 tx_winsize; /* Maximum size of Tx window */
521 u8 nr_jumbo_dup; /* Number of jumbo duplicates */
512 522
513 /* receive-phase ACK management */ 523 /* receive-phase ACK management */
514 rxrpc_seq_t rx_data_expect; /* next data seq ID expected to be received */
515 rxrpc_seq_t rx_data_post; /* next data seq ID expected to be posted */
516 rxrpc_seq_t rx_data_recv; /* last data seq ID encountered by recvmsg */
517 rxrpc_seq_t rx_data_eaten; /* last data seq ID consumed by recvmsg */
518 rxrpc_seq_t rx_first_oos; /* first packet in rx_oos_queue (or 0) */
519 rxrpc_seq_t ackr_win_top; /* top of ACK window (rx_data_eaten is bottom) */
520 rxrpc_seq_t ackr_prev_seq; /* previous sequence number received */
521 u8 ackr_reason; /* reason to ACK */ 524 u8 ackr_reason; /* reason to ACK */
522 u16 ackr_skew; /* skew on packet being ACK'd */ 525 u16 ackr_skew; /* skew on packet being ACK'd */
523 rxrpc_serial_t ackr_serial; /* serial of packet being ACK'd */ 526 rxrpc_serial_t ackr_serial; /* serial of packet being ACK'd */
524 atomic_t ackr_not_idle; /* number of packets in Rx queue */ 527 rxrpc_seq_t ackr_prev_seq; /* previous sequence number received */
528 unsigned short rx_pkt_offset; /* Current recvmsg packet offset */
529 unsigned short rx_pkt_len; /* Current recvmsg packet len */
525 530
526 /* received packet records, 1 bit per record */ 531 /* transmission-phase ACK management */
527#define RXRPC_ACKR_WINDOW_ASZ DIV_ROUND_UP(RXRPC_MAXACKS, BITS_PER_LONG) 532 rxrpc_serial_t acks_latest; /* serial number of latest ACK received */
528 unsigned long ackr_window[RXRPC_ACKR_WINDOW_ASZ + 1];
529}; 533};
530 534
531enum rxrpc_call_trace { 535enum rxrpc_call_trace {
@@ -535,10 +539,8 @@ enum rxrpc_call_trace {
535 rxrpc_call_queued_ref, 539 rxrpc_call_queued_ref,
536 rxrpc_call_seen, 540 rxrpc_call_seen,
537 rxrpc_call_got, 541 rxrpc_call_got,
538 rxrpc_call_got_skb,
539 rxrpc_call_got_userid, 542 rxrpc_call_got_userid,
540 rxrpc_call_put, 543 rxrpc_call_put,
541 rxrpc_call_put_skb,
542 rxrpc_call_put_userid, 544 rxrpc_call_put_userid,
543 rxrpc_call_put_noqueue, 545 rxrpc_call_put_noqueue,
544 rxrpc_call__nr_trace 546 rxrpc_call__nr_trace
@@ -561,6 +563,9 @@ extern struct workqueue_struct *rxrpc_workqueue;
561 */ 563 */
562int rxrpc_service_prealloc(struct rxrpc_sock *, gfp_t); 564int rxrpc_service_prealloc(struct rxrpc_sock *, gfp_t);
563void rxrpc_discard_prealloc(struct rxrpc_sock *); 565void rxrpc_discard_prealloc(struct rxrpc_sock *);
566struct rxrpc_call *rxrpc_new_incoming_call(struct rxrpc_local *,
567 struct rxrpc_connection *,
568 struct sk_buff *);
564void rxrpc_accept_incoming_calls(struct rxrpc_local *); 569void rxrpc_accept_incoming_calls(struct rxrpc_local *);
565struct rxrpc_call *rxrpc_accept_call(struct rxrpc_sock *, unsigned long, 570struct rxrpc_call *rxrpc_accept_call(struct rxrpc_sock *, unsigned long,
566 rxrpc_notify_rx_t); 571 rxrpc_notify_rx_t);
@@ -569,8 +574,7 @@ int rxrpc_reject_call(struct rxrpc_sock *);
569/* 574/*
570 * call_event.c 575 * call_event.c
571 */ 576 */
572void __rxrpc_propose_ACK(struct rxrpc_call *, u8, u16, u32, bool); 577void rxrpc_propose_ACK(struct rxrpc_call *, u8, u16, u32, bool, bool);
573void rxrpc_propose_ACK(struct rxrpc_call *, u8, u16, u32, bool);
574void rxrpc_process_call(struct work_struct *); 578void rxrpc_process_call(struct work_struct *);
575 579
576/* 580/*
@@ -589,9 +593,8 @@ struct rxrpc_call *rxrpc_new_client_call(struct rxrpc_sock *,
589 struct rxrpc_conn_parameters *, 593 struct rxrpc_conn_parameters *,
590 struct sockaddr_rxrpc *, 594 struct sockaddr_rxrpc *,
591 unsigned long, gfp_t); 595 unsigned long, gfp_t);
592struct rxrpc_call *rxrpc_incoming_call(struct rxrpc_sock *, 596void rxrpc_incoming_call(struct rxrpc_sock *, struct rxrpc_call *,
593 struct rxrpc_connection *, 597 struct sk_buff *);
594 struct sk_buff *);
595void rxrpc_release_call(struct rxrpc_sock *, struct rxrpc_call *); 598void rxrpc_release_call(struct rxrpc_sock *, struct rxrpc_call *);
596void rxrpc_release_calls_on_socket(struct rxrpc_sock *); 599void rxrpc_release_calls_on_socket(struct rxrpc_sock *);
597bool __rxrpc_queue_call(struct rxrpc_call *); 600bool __rxrpc_queue_call(struct rxrpc_call *);
@@ -599,8 +602,6 @@ bool rxrpc_queue_call(struct rxrpc_call *);
599void rxrpc_see_call(struct rxrpc_call *); 602void rxrpc_see_call(struct rxrpc_call *);
600void rxrpc_get_call(struct rxrpc_call *, enum rxrpc_call_trace); 603void rxrpc_get_call(struct rxrpc_call *, enum rxrpc_call_trace);
601void rxrpc_put_call(struct rxrpc_call *, enum rxrpc_call_trace); 604void rxrpc_put_call(struct rxrpc_call *, enum rxrpc_call_trace);
602void rxrpc_get_call_for_skb(struct rxrpc_call *, struct sk_buff *);
603void rxrpc_put_call_for_skb(struct rxrpc_call *, struct sk_buff *);
604void rxrpc_cleanup_call(struct rxrpc_call *); 605void rxrpc_cleanup_call(struct rxrpc_call *);
605void __exit rxrpc_destroy_all_calls(void); 606void __exit rxrpc_destroy_all_calls(void);
606 607
@@ -672,13 +673,8 @@ static inline bool __rxrpc_abort_call(const char *why, struct rxrpc_call *call,
672{ 673{
673 trace_rxrpc_abort(why, call->cid, call->call_id, seq, 674 trace_rxrpc_abort(why, call->cid, call->call_id, seq,
674 abort_code, error); 675 abort_code, error);
675 if (__rxrpc_set_call_completion(call, 676 return __rxrpc_set_call_completion(call, RXRPC_CALL_LOCALLY_ABORTED,
676 RXRPC_CALL_LOCALLY_ABORTED, 677 abort_code, error);
677 abort_code, error)) {
678 set_bit(RXRPC_CALL_EV_ABORT, &call->events);
679 return true;
680 }
681 return false;
682} 678}
683 679
684static inline bool rxrpc_abort_call(const char *why, struct rxrpc_call *call, 680static inline bool rxrpc_abort_call(const char *why, struct rxrpc_call *call,
@@ -713,8 +709,6 @@ void __exit rxrpc_destroy_all_client_connections(void);
713 * conn_event.c 709 * conn_event.c
714 */ 710 */
715void rxrpc_process_connection(struct work_struct *); 711void rxrpc_process_connection(struct work_struct *);
716void rxrpc_reject_packet(struct rxrpc_local *, struct sk_buff *);
717void rxrpc_reject_packets(struct rxrpc_local *);
718 712
719/* 713/*
720 * conn_object.c 714 * conn_object.c
@@ -783,18 +777,14 @@ static inline bool rxrpc_queue_conn(struct rxrpc_connection *conn)
783 */ 777 */
784struct rxrpc_connection *rxrpc_find_service_conn_rcu(struct rxrpc_peer *, 778struct rxrpc_connection *rxrpc_find_service_conn_rcu(struct rxrpc_peer *,
785 struct sk_buff *); 779 struct sk_buff *);
786struct rxrpc_connection *rxrpc_incoming_connection(struct rxrpc_local *,
787 struct sockaddr_rxrpc *,
788 struct sk_buff *);
789struct rxrpc_connection *rxrpc_prealloc_service_connection(gfp_t); 780struct rxrpc_connection *rxrpc_prealloc_service_connection(gfp_t);
781void rxrpc_new_incoming_connection(struct rxrpc_connection *, struct sk_buff *);
790void rxrpc_unpublish_service_conn(struct rxrpc_connection *); 782void rxrpc_unpublish_service_conn(struct rxrpc_connection *);
791 783
792/* 784/*
793 * input.c 785 * input.c
794 */ 786 */
795void rxrpc_data_ready(struct sock *); 787void rxrpc_data_ready(struct sock *);
796int rxrpc_queue_rcv_skb(struct rxrpc_call *, struct sk_buff *, bool, bool);
797void rxrpc_fast_process_packet(struct rxrpc_call *, struct sk_buff *);
798 788
799/* 789/*
800 * insecure.c 790 * insecure.c
@@ -868,6 +858,7 @@ extern const char *rxrpc_acks(u8 reason);
868 */ 858 */
869int rxrpc_send_call_packet(struct rxrpc_call *, u8); 859int rxrpc_send_call_packet(struct rxrpc_call *, u8);
870int rxrpc_send_data_packet(struct rxrpc_connection *, struct sk_buff *); 860int rxrpc_send_data_packet(struct rxrpc_connection *, struct sk_buff *);
861void rxrpc_reject_packets(struct rxrpc_local *);
871 862
872/* 863/*
873 * peer_event.c 864 * peer_event.c
@@ -883,6 +874,8 @@ struct rxrpc_peer *rxrpc_lookup_peer_rcu(struct rxrpc_local *,
883struct rxrpc_peer *rxrpc_lookup_peer(struct rxrpc_local *, 874struct rxrpc_peer *rxrpc_lookup_peer(struct rxrpc_local *,
884 struct sockaddr_rxrpc *, gfp_t); 875 struct sockaddr_rxrpc *, gfp_t);
885struct rxrpc_peer *rxrpc_alloc_peer(struct rxrpc_local *, gfp_t); 876struct rxrpc_peer *rxrpc_alloc_peer(struct rxrpc_local *, gfp_t);
877struct rxrpc_peer *rxrpc_lookup_incoming_peer(struct rxrpc_local *,
878 struct rxrpc_peer *);
886 879
887static inline struct rxrpc_peer *rxrpc_get_peer(struct rxrpc_peer *peer) 880static inline struct rxrpc_peer *rxrpc_get_peer(struct rxrpc_peer *peer)
888{ 881{
@@ -912,6 +905,7 @@ extern const struct file_operations rxrpc_connection_seq_fops;
912/* 905/*
913 * recvmsg.c 906 * recvmsg.c
914 */ 907 */
908void rxrpc_notify_socket(struct rxrpc_call *);
915int rxrpc_recvmsg(struct socket *, struct msghdr *, size_t, int); 909int rxrpc_recvmsg(struct socket *, struct msghdr *, size_t, int);
916 910
917/* 911/*
@@ -961,6 +955,23 @@ static inline void rxrpc_sysctl_exit(void) {}
961 */ 955 */
962int rxrpc_extract_addr_from_skb(struct sockaddr_rxrpc *, struct sk_buff *); 956int rxrpc_extract_addr_from_skb(struct sockaddr_rxrpc *, struct sk_buff *);
963 957
958static inline bool before(u32 seq1, u32 seq2)
959{
960 return (s32)(seq1 - seq2) < 0;
961}
962static inline bool before_eq(u32 seq1, u32 seq2)
963{
964 return (s32)(seq1 - seq2) <= 0;
965}
966static inline bool after(u32 seq1, u32 seq2)
967{
968 return (s32)(seq1 - seq2) > 0;
969}
970static inline bool after_eq(u32 seq1, u32 seq2)
971{
972 return (s32)(seq1 - seq2) >= 0;
973}
974
964/* 975/*
965 * debug tracing 976 * debug tracing
966 */ 977 */
diff --git a/net/rxrpc/call_accept.c b/net/rxrpc/call_accept.c
index cc7194e05a15..b8acec0d596e 100644
--- a/net/rxrpc/call_accept.c
+++ b/net/rxrpc/call_accept.c
@@ -129,6 +129,8 @@ static int rxrpc_service_prealloc_one(struct rxrpc_sock *rx,
129 set_bit(RXRPC_CALL_HAS_USERID, &call->flags); 129 set_bit(RXRPC_CALL_HAS_USERID, &call->flags);
130 } 130 }
131 131
132 list_add(&call->sock_link, &rx->sock_calls);
133
132 write_unlock(&rx->call_lock); 134 write_unlock(&rx->call_lock);
133 135
134 write_lock(&rxrpc_call_lock); 136 write_lock(&rxrpc_call_lock);
@@ -186,6 +188,12 @@ void rxrpc_discard_prealloc(struct rxrpc_sock *rx)
186 return; 188 return;
187 rx->backlog = NULL; 189 rx->backlog = NULL;
188 190
191 /* Make sure that there aren't any incoming calls in progress before we
192 * clear the preallocation buffers.
193 */
194 spin_lock_bh(&rx->incoming_lock);
195 spin_unlock_bh(&rx->incoming_lock);
196
189 head = b->peer_backlog_head; 197 head = b->peer_backlog_head;
190 tail = b->peer_backlog_tail; 198 tail = b->peer_backlog_tail;
191 while (CIRC_CNT(head, tail, size) > 0) { 199 while (CIRC_CNT(head, tail, size) > 0) {
@@ -224,251 +232,179 @@ void rxrpc_discard_prealloc(struct rxrpc_sock *rx)
224} 232}
225 233
226/* 234/*
227 * generate a connection-level abort 235 * Allocate a new incoming call from the prealloc pool, along with a connection
236 * and a peer as necessary.
228 */ 237 */
229static int rxrpc_busy(struct rxrpc_local *local, struct sockaddr_rxrpc *srx, 238static struct rxrpc_call *rxrpc_alloc_incoming_call(struct rxrpc_sock *rx,
230 struct rxrpc_wire_header *whdr) 239 struct rxrpc_local *local,
240 struct rxrpc_connection *conn,
241 struct sk_buff *skb)
231{ 242{
232 struct msghdr msg; 243 struct rxrpc_backlog *b = rx->backlog;
233 struct kvec iov[1]; 244 struct rxrpc_peer *peer, *xpeer;
234 size_t len; 245 struct rxrpc_call *call;
235 int ret; 246 unsigned short call_head, conn_head, peer_head;
236 247 unsigned short call_tail, conn_tail, peer_tail;
237 _enter("%d,,", local->debug_id); 248 unsigned short call_count, conn_count;
238 249
239 whdr->type = RXRPC_PACKET_TYPE_BUSY; 250 /* #calls >= #conns >= #peers must hold true. */
240 whdr->serial = htonl(1); 251 call_head = smp_load_acquire(&b->call_backlog_head);
241 252 call_tail = b->call_backlog_tail;
242 msg.msg_name = &srx->transport.sin; 253 call_count = CIRC_CNT(call_head, call_tail, RXRPC_BACKLOG_MAX);
243 msg.msg_namelen = sizeof(srx->transport.sin); 254 conn_head = smp_load_acquire(&b->conn_backlog_head);
244 msg.msg_control = NULL; 255 conn_tail = b->conn_backlog_tail;
245 msg.msg_controllen = 0; 256 conn_count = CIRC_CNT(conn_head, conn_tail, RXRPC_BACKLOG_MAX);
246 msg.msg_flags = 0; 257 ASSERTCMP(conn_count, >=, call_count);
247 258 peer_head = smp_load_acquire(&b->peer_backlog_head);
248 iov[0].iov_base = whdr; 259 peer_tail = b->peer_backlog_tail;
249 iov[0].iov_len = sizeof(*whdr); 260 ASSERTCMP(CIRC_CNT(peer_head, peer_tail, RXRPC_BACKLOG_MAX), >=,
250 261 conn_count);
251 len = iov[0].iov_len; 262
252 263 if (call_count == 0)
253 _proto("Tx BUSY %%1"); 264 return NULL;
265
266 if (!conn) {
267 /* No connection. We're going to need a peer to start off
268 * with. If one doesn't yet exist, use a spare from the
269 * preallocation set. We dump the address into the spare in
270 * anticipation - and to save on stack space.
271 */
272 xpeer = b->peer_backlog[peer_tail];
273 if (rxrpc_extract_addr_from_skb(&xpeer->srx, skb) < 0)
274 return NULL;
275
276 peer = rxrpc_lookup_incoming_peer(local, xpeer);
277 if (peer == xpeer) {
278 b->peer_backlog[peer_tail] = NULL;
279 smp_store_release(&b->peer_backlog_tail,
280 (peer_tail + 1) &
281 (RXRPC_BACKLOG_MAX - 1));
282 }
254 283
255 ret = kernel_sendmsg(local->socket, &msg, iov, 1, len); 284 /* Now allocate and set up the connection */
256 if (ret < 0) { 285 conn = b->conn_backlog[conn_tail];
257 _leave(" = -EAGAIN [sendmsg failed: %d]", ret); 286 b->conn_backlog[conn_tail] = NULL;
258 return -EAGAIN; 287 smp_store_release(&b->conn_backlog_tail,
288 (conn_tail + 1) & (RXRPC_BACKLOG_MAX - 1));
289 rxrpc_get_local(local);
290 conn->params.local = local;
291 conn->params.peer = peer;
292 rxrpc_new_incoming_connection(conn, skb);
293 } else {
294 rxrpc_get_connection(conn);
259 } 295 }
260 296
261 _leave(" = 0"); 297 /* And now we can allocate and set up a new call */
262 return 0; 298 call = b->call_backlog[call_tail];
299 b->call_backlog[call_tail] = NULL;
300 smp_store_release(&b->call_backlog_tail,
301 (call_tail + 1) & (RXRPC_BACKLOG_MAX - 1));
302
303 call->conn = conn;
304 call->peer = rxrpc_get_peer(conn->params.peer);
305 return call;
263} 306}
264 307
265/* 308/*
266 * accept an incoming call that needs peer, transport and/or connection setting 309 * Set up a new incoming call. Called in BH context with the RCU read lock
267 * up 310 * held.
311 *
312 * If this is for a kernel service, when we allocate the call, it will have
313 * three refs on it: (1) the kernel service, (2) the user_call_ID tree, (3) the
314 * retainer ref obtained from the backlog buffer. Prealloc calls for userspace
315 * services only have the ref from the backlog buffer. We want to pass this
316 * ref to non-BH context to dispose of.
317 *
318 * If we want to report an error, we mark the skb with the packet type and
319 * abort code and return NULL.
268 */ 320 */
269static int rxrpc_accept_incoming_call(struct rxrpc_local *local, 321struct rxrpc_call *rxrpc_new_incoming_call(struct rxrpc_local *local,
270 struct rxrpc_sock *rx, 322 struct rxrpc_connection *conn,
271 struct sk_buff *skb, 323 struct sk_buff *skb)
272 struct sockaddr_rxrpc *srx)
273{ 324{
274 struct rxrpc_connection *conn; 325 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
275 struct rxrpc_skb_priv *sp, *nsp; 326 struct rxrpc_sock *rx;
276 struct rxrpc_call *call; 327 struct rxrpc_call *call;
277 struct sk_buff *notification;
278 int ret;
279 328
280 _enter(""); 329 _enter("");
281 330
282 sp = rxrpc_skb(skb); 331 /* Get the socket providing the service */
283 332 hlist_for_each_entry_rcu_bh(rx, &local->services, listen_link) {
284 /* get a notification message to send to the server app */ 333 if (rx->srx.srx_service == sp->hdr.serviceId)
285 notification = alloc_skb(0, GFP_NOFS); 334 goto found_service;
286 if (!notification) {
287 _debug("no memory");
288 ret = -ENOMEM;
289 goto error_nofree;
290 }
291 rxrpc_new_skb(notification);
292 notification->mark = RXRPC_SKB_MARK_NEW_CALL;
293
294 conn = rxrpc_incoming_connection(local, srx, skb);
295 if (IS_ERR(conn)) {
296 _debug("no conn");
297 ret = PTR_ERR(conn);
298 goto error;
299 }
300
301 call = rxrpc_incoming_call(rx, conn, skb);
302 rxrpc_put_connection(conn);
303 if (IS_ERR(call)) {
304 _debug("no call");
305 ret = PTR_ERR(call);
306 goto error;
307 } 335 }
308 336
309 /* attach the call to the socket */ 337 trace_rxrpc_abort("INV", sp->hdr.cid, sp->hdr.callNumber, sp->hdr.seq,
310 read_lock_bh(&local->services_lock); 338 RX_INVALID_OPERATION, EOPNOTSUPP);
311 if (rx->sk.sk_state == RXRPC_CLOSE) 339 skb->mark = RXRPC_SKB_MARK_LOCAL_ABORT;
312 goto invalid_service; 340 skb->priority = RX_INVALID_OPERATION;
313 341 _leave(" = NULL [service]");
314 write_lock(&rx->call_lock); 342 return NULL;
315 if (!test_and_set_bit(RXRPC_CALL_INIT_ACCEPT, &call->flags)) {
316 rxrpc_get_call(call, rxrpc_call_got);
317
318 spin_lock(&call->conn->state_lock);
319 if (sp->hdr.securityIndex > 0 &&
320 call->conn->state == RXRPC_CONN_SERVICE_UNSECURED) {
321 _debug("await conn sec");
322 list_add_tail(&call->accept_link, &rx->secureq);
323 call->conn->state = RXRPC_CONN_SERVICE_CHALLENGING;
324 set_bit(RXRPC_CONN_EV_CHALLENGE, &call->conn->events);
325 rxrpc_queue_conn(call->conn);
326 } else {
327 _debug("conn ready");
328 call->state = RXRPC_CALL_SERVER_ACCEPTING;
329 list_add_tail(&call->accept_link, &rx->acceptq);
330 rxrpc_get_call_for_skb(call, notification);
331 nsp = rxrpc_skb(notification);
332 nsp->call = call;
333
334 ASSERTCMP(atomic_read(&call->usage), >=, 3);
335
336 _debug("notify");
337 spin_lock(&call->lock);
338 ret = rxrpc_queue_rcv_skb(call, notification, true,
339 false);
340 spin_unlock(&call->lock);
341 notification = NULL;
342 BUG_ON(ret < 0);
343 }
344 spin_unlock(&call->conn->state_lock);
345 343
346 _debug("queued"); 344found_service:
345 spin_lock(&rx->incoming_lock);
346 if (rx->sk.sk_state == RXRPC_CLOSE) {
347 trace_rxrpc_abort("CLS", sp->hdr.cid, sp->hdr.callNumber,
348 sp->hdr.seq, RX_INVALID_OPERATION, ESHUTDOWN);
349 skb->mark = RXRPC_SKB_MARK_LOCAL_ABORT;
350 skb->priority = RX_INVALID_OPERATION;
351 _leave(" = NULL [close]");
352 call = NULL;
353 goto out;
347 } 354 }
348 write_unlock(&rx->call_lock);
349 355
350 _debug("process"); 356 call = rxrpc_alloc_incoming_call(rx, local, conn, skb);
351 rxrpc_fast_process_packet(call, skb); 357 if (!call) {
352 358 skb->mark = RXRPC_SKB_MARK_BUSY;
353 _debug("done"); 359 _leave(" = NULL [busy]");
354 read_unlock_bh(&local->services_lock); 360 call = NULL;
355 rxrpc_free_skb(notification); 361 goto out;
356 rxrpc_put_call(call, rxrpc_call_put); 362 }
357 _leave(" = 0");
358 return 0;
359
360invalid_service:
361 _debug("invalid");
362 read_unlock_bh(&local->services_lock);
363
364 rxrpc_release_call(rx, call);
365 rxrpc_put_call(call, rxrpc_call_put);
366 ret = -ECONNREFUSED;
367error:
368 rxrpc_free_skb(notification);
369error_nofree:
370 _leave(" = %d", ret);
371 return ret;
372}
373 363
374/* 364 /* Make the call live. */
375 * accept incoming calls that need peer, transport and/or connection setting up 365 rxrpc_incoming_call(rx, call, skb);
376 * - the packets we get are all incoming client DATA packets that have seq == 1 366 conn = call->conn;
377 */
378void rxrpc_accept_incoming_calls(struct rxrpc_local *local)
379{
380 struct rxrpc_skb_priv *sp;
381 struct sockaddr_rxrpc srx;
382 struct rxrpc_sock *rx;
383 struct rxrpc_wire_header whdr;
384 struct sk_buff *skb;
385 int ret;
386 367
387 _enter("%d", local->debug_id); 368 if (rx->notify_new_call)
369 rx->notify_new_call(&rx->sk, call, call->user_call_ID);
388 370
389 skb = skb_dequeue(&local->accept_queue); 371 spin_lock(&conn->state_lock);
390 if (!skb) { 372 switch (conn->state) {
391 _leave("\n"); 373 case RXRPC_CONN_SERVICE_UNSECURED:
392 return; 374 conn->state = RXRPC_CONN_SERVICE_CHALLENGING;
393 } 375 set_bit(RXRPC_CONN_EV_CHALLENGE, &call->conn->events);
376 rxrpc_queue_conn(call->conn);
377 break;
394 378
395 _net("incoming call skb %p", skb); 379 case RXRPC_CONN_SERVICE:
396 380 write_lock(&call->state_lock);
397 rxrpc_see_skb(skb); 381 if (rx->discard_new_call)
398 sp = rxrpc_skb(skb); 382 call->state = RXRPC_CALL_SERVER_RECV_REQUEST;
399 383 else
400 /* Set up a response packet header in case we need it */ 384 call->state = RXRPC_CALL_SERVER_ACCEPTING;
401 whdr.epoch = htonl(sp->hdr.epoch); 385 write_unlock(&call->state_lock);
402 whdr.cid = htonl(sp->hdr.cid); 386 break;
403 whdr.callNumber = htonl(sp->hdr.callNumber);
404 whdr.seq = htonl(sp->hdr.seq);
405 whdr.serial = 0;
406 whdr.flags = 0;
407 whdr.type = 0;
408 whdr.userStatus = 0;
409 whdr.securityIndex = sp->hdr.securityIndex;
410 whdr._rsvd = 0;
411 whdr.serviceId = htons(sp->hdr.serviceId);
412
413 if (rxrpc_extract_addr_from_skb(&srx, skb) < 0)
414 goto drop;
415
416 /* get the socket providing the service */
417 read_lock_bh(&local->services_lock);
418 hlist_for_each_entry(rx, &local->services, listen_link) {
419 if (rx->srx.srx_service == sp->hdr.serviceId &&
420 rx->sk.sk_state != RXRPC_CLOSE)
421 goto found_service;
422 }
423 read_unlock_bh(&local->services_lock);
424 goto invalid_service;
425 387
426found_service: 388 case RXRPC_CONN_REMOTELY_ABORTED:
427 _debug("found service %hd", rx->srx.srx_service); 389 rxrpc_set_call_completion(call, RXRPC_CALL_REMOTELY_ABORTED,
428 if (sk_acceptq_is_full(&rx->sk)) 390 conn->remote_abort, ECONNABORTED);
429 goto backlog_full; 391 break;
430 sk_acceptq_added(&rx->sk); 392 case RXRPC_CONN_LOCALLY_ABORTED:
431 read_unlock_bh(&local->services_lock); 393 rxrpc_abort_call("CON", call, sp->hdr.seq,
432 394 conn->local_abort, ECONNABORTED);
433 ret = rxrpc_accept_incoming_call(local, rx, skb, &srx); 395 break;
434 if (ret < 0)
435 sk_acceptq_removed(&rx->sk);
436 switch (ret) {
437 case -ECONNRESET: /* old calls are ignored */
438 case -ECONNABORTED: /* aborted calls are reaborted or ignored */
439 case 0:
440 return;
441 case -ECONNREFUSED:
442 goto invalid_service;
443 case -EBUSY:
444 goto busy;
445 case -EKEYREJECTED:
446 goto security_mismatch;
447 default: 396 default:
448 BUG(); 397 BUG();
449 } 398 }
399 spin_unlock(&conn->state_lock);
450 400
451backlog_full: 401 if (call->state == RXRPC_CALL_SERVER_ACCEPTING)
452 read_unlock_bh(&local->services_lock); 402 rxrpc_notify_socket(call);
453busy:
454 rxrpc_busy(local, &srx, &whdr);
455 rxrpc_free_skb(skb);
456 return;
457
458drop:
459 rxrpc_free_skb(skb);
460 return;
461 403
462invalid_service: 404 _leave(" = %p{%d}", call, call->debug_id);
463 skb->priority = RX_INVALID_OPERATION; 405out:
464 rxrpc_reject_packet(local, skb); 406 spin_unlock(&rx->incoming_lock);
465 return; 407 return call;
466
467 /* can't change connection security type mid-flow */
468security_mismatch:
469 skb->priority = RX_PROTOCOL_ERROR;
470 rxrpc_reject_packet(local, skb);
471 return;
472} 408}
473 409
474/* 410/*
@@ -490,11 +426,10 @@ struct rxrpc_call *rxrpc_accept_call(struct rxrpc_sock *rx,
490 write_lock(&rx->call_lock); 426 write_lock(&rx->call_lock);
491 427
492 ret = -ENODATA; 428 ret = -ENODATA;
493 if (list_empty(&rx->acceptq)) 429 if (list_empty(&rx->to_be_accepted))
494 goto out; 430 goto out;
495 431
496 /* check the user ID isn't already in use */ 432 /* check the user ID isn't already in use */
497 ret = -EBADSLT;
498 pp = &rx->calls.rb_node; 433 pp = &rx->calls.rb_node;
499 parent = NULL; 434 parent = NULL;
500 while (*pp) { 435 while (*pp) {
@@ -506,11 +441,14 @@ struct rxrpc_call *rxrpc_accept_call(struct rxrpc_sock *rx,
506 else if (user_call_ID > call->user_call_ID) 441 else if (user_call_ID > call->user_call_ID)
507 pp = &(*pp)->rb_right; 442 pp = &(*pp)->rb_right;
508 else 443 else
509 goto out; 444 goto id_in_use;
510 } 445 }
511 446
512 /* dequeue the first call and check it's still valid */ 447 /* Dequeue the first call and check it's still valid. We gain
513 call = list_entry(rx->acceptq.next, struct rxrpc_call, accept_link); 448 * responsibility for the queue's reference.
449 */
450 call = list_entry(rx->to_be_accepted.next,
451 struct rxrpc_call, accept_link);
514 list_del_init(&call->accept_link); 452 list_del_init(&call->accept_link);
515 sk_acceptq_removed(&rx->sk); 453 sk_acceptq_removed(&rx->sk);
516 rxrpc_see_call(call); 454 rxrpc_see_call(call);
@@ -528,31 +466,35 @@ struct rxrpc_call *rxrpc_accept_call(struct rxrpc_sock *rx,
528 } 466 }
529 467
530 /* formalise the acceptance */ 468 /* formalise the acceptance */
531 rxrpc_get_call(call, rxrpc_call_got_userid); 469 rxrpc_get_call(call, rxrpc_call_got);
532 call->notify_rx = notify_rx; 470 call->notify_rx = notify_rx;
533 call->user_call_ID = user_call_ID; 471 call->user_call_ID = user_call_ID;
472 rxrpc_get_call(call, rxrpc_call_got_userid);
534 rb_link_node(&call->sock_node, parent, pp); 473 rb_link_node(&call->sock_node, parent, pp);
535 rb_insert_color(&call->sock_node, &rx->calls); 474 rb_insert_color(&call->sock_node, &rx->calls);
536 if (test_and_set_bit(RXRPC_CALL_HAS_USERID, &call->flags)) 475 if (test_and_set_bit(RXRPC_CALL_HAS_USERID, &call->flags))
537 BUG(); 476 BUG();
538 if (test_and_set_bit(RXRPC_CALL_EV_ACCEPTED, &call->events))
539 BUG();
540 477
541 write_unlock_bh(&call->state_lock); 478 write_unlock_bh(&call->state_lock);
542 write_unlock(&rx->call_lock); 479 write_unlock(&rx->call_lock);
543 rxrpc_queue_call(call); 480 rxrpc_notify_socket(call);
481 rxrpc_service_prealloc(rx, GFP_KERNEL);
544 _leave(" = %p{%d}", call, call->debug_id); 482 _leave(" = %p{%d}", call, call->debug_id);
545 return call; 483 return call;
546 484
547out_release: 485out_release:
486 _debug("release %p", call);
548 write_unlock_bh(&call->state_lock); 487 write_unlock_bh(&call->state_lock);
549 write_unlock(&rx->call_lock); 488 write_unlock(&rx->call_lock);
550 _debug("release %p", call);
551 rxrpc_release_call(rx, call); 489 rxrpc_release_call(rx, call);
552 _leave(" = %d", ret); 490 rxrpc_put_call(call, rxrpc_call_put);
553 return ERR_PTR(ret); 491 goto out;
554out: 492
493id_in_use:
494 ret = -EBADSLT;
555 write_unlock(&rx->call_lock); 495 write_unlock(&rx->call_lock);
496out:
497 rxrpc_service_prealloc(rx, GFP_KERNEL);
556 _leave(" = %d", ret); 498 _leave(" = %d", ret);
557 return ERR_PTR(ret); 499 return ERR_PTR(ret);
558} 500}
@@ -564,6 +506,7 @@ out:
564int rxrpc_reject_call(struct rxrpc_sock *rx) 506int rxrpc_reject_call(struct rxrpc_sock *rx)
565{ 507{
566 struct rxrpc_call *call; 508 struct rxrpc_call *call;
509 bool abort = false;
567 int ret; 510 int ret;
568 511
569 _enter(""); 512 _enter("");
@@ -572,15 +515,16 @@ int rxrpc_reject_call(struct rxrpc_sock *rx)
572 515
573 write_lock(&rx->call_lock); 516 write_lock(&rx->call_lock);
574 517
575 ret = -ENODATA; 518 if (list_empty(&rx->to_be_accepted)) {
576 if (list_empty(&rx->acceptq)) {
577 write_unlock(&rx->call_lock); 519 write_unlock(&rx->call_lock);
578 _leave(" = -ENODATA");
579 return -ENODATA; 520 return -ENODATA;
580 } 521 }
581 522
582 /* dequeue the first call and check it's still valid */ 523 /* Dequeue the first call and check it's still valid. We gain
583 call = list_entry(rx->acceptq.next, struct rxrpc_call, accept_link); 524 * responsibility for the queue's reference.
525 */
526 call = list_entry(rx->to_be_accepted.next,
527 struct rxrpc_call, accept_link);
584 list_del_init(&call->accept_link); 528 list_del_init(&call->accept_link);
585 sk_acceptq_removed(&rx->sk); 529 sk_acceptq_removed(&rx->sk);
586 rxrpc_see_call(call); 530 rxrpc_see_call(call);
@@ -588,66 +532,28 @@ int rxrpc_reject_call(struct rxrpc_sock *rx)
588 write_lock_bh(&call->state_lock); 532 write_lock_bh(&call->state_lock);
589 switch (call->state) { 533 switch (call->state) {
590 case RXRPC_CALL_SERVER_ACCEPTING: 534 case RXRPC_CALL_SERVER_ACCEPTING:
591 __rxrpc_set_call_completion(call, RXRPC_CALL_SERVER_BUSY, 535 __rxrpc_abort_call("REJ", call, 1, RX_USER_ABORT, ECONNABORTED);
592 0, ECONNABORTED); 536 abort = true;
593 if (test_and_set_bit(RXRPC_CALL_EV_REJECT_BUSY, &call->events)) 537 /* fall through */
594 rxrpc_queue_call(call);
595 ret = 0;
596 break;
597 case RXRPC_CALL_COMPLETE: 538 case RXRPC_CALL_COMPLETE:
598 ret = call->error; 539 ret = call->error;
599 break; 540 goto out_discard;
600 default: 541 default:
601 BUG(); 542 BUG();
602 } 543 }
603 544
545out_discard:
604 write_unlock_bh(&call->state_lock); 546 write_unlock_bh(&call->state_lock);
605 write_unlock(&rx->call_lock); 547 write_unlock(&rx->call_lock);
606 rxrpc_release_call(rx, call); 548 if (abort) {
607 _leave(" = %d", ret); 549 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ABORT);
608 return ret; 550 rxrpc_release_call(rx, call);
609} 551 rxrpc_put_call(call, rxrpc_call_put);
610 552 }
611/** 553 rxrpc_service_prealloc(rx, GFP_KERNEL);
612 * rxrpc_kernel_accept_call - Allow a kernel service to accept an incoming call
613 * @sock: The socket on which the impending call is waiting
614 * @user_call_ID: The tag to attach to the call
615 * @notify_rx: Where to send notifications instead of socket queue
616 *
617 * Allow a kernel service to accept an incoming call, assuming the incoming
618 * call is still valid. The caller should immediately trigger their own
619 * notification as there must be data waiting.
620 */
621struct rxrpc_call *rxrpc_kernel_accept_call(struct socket *sock,
622 unsigned long user_call_ID,
623 rxrpc_notify_rx_t notify_rx)
624{
625 struct rxrpc_call *call;
626
627 _enter(",%lx", user_call_ID);
628 call = rxrpc_accept_call(rxrpc_sk(sock->sk), user_call_ID, notify_rx);
629 _leave(" = %p", call);
630 return call;
631}
632EXPORT_SYMBOL(rxrpc_kernel_accept_call);
633
634/**
635 * rxrpc_kernel_reject_call - Allow a kernel service to reject an incoming call
636 * @sock: The socket on which the impending call is waiting
637 *
638 * Allow a kernel service to reject an incoming call with a BUSY message,
639 * assuming the incoming call is still valid.
640 */
641int rxrpc_kernel_reject_call(struct socket *sock)
642{
643 int ret;
644
645 _enter("");
646 ret = rxrpc_reject_call(rxrpc_sk(sock->sk));
647 _leave(" = %d", ret); 554 _leave(" = %d", ret);
648 return ret; 555 return ret;
649} 556}
650EXPORT_SYMBOL(rxrpc_kernel_reject_call);
651 557
652/* 558/*
653 * rxrpc_kernel_charge_accept - Charge up socket with preallocated calls 559 * rxrpc_kernel_charge_accept - Charge up socket with preallocated calls
diff --git a/net/rxrpc/call_event.c b/net/rxrpc/call_event.c
index af88ad7d2cf9..2b976e789562 100644
--- a/net/rxrpc/call_event.c
+++ b/net/rxrpc/call_event.c
@@ -22,1257 +22,286 @@
22#include "ar-internal.h" 22#include "ar-internal.h"
23 23
24/* 24/*
25 * propose an ACK be sent 25 * Set the timer
26 */ 26 */
27void __rxrpc_propose_ACK(struct rxrpc_call *call, u8 ack_reason, 27static void rxrpc_set_timer(struct rxrpc_call *call)
28 u16 skew, u32 serial, bool immediate)
29{ 28{
30 unsigned long expiry; 29 unsigned long t, now = jiffies;
31 s8 prior = rxrpc_ack_priority[ack_reason];
32
33 ASSERTCMP(prior, >, 0);
34
35 _enter("{%d},%s,%%%x,%u",
36 call->debug_id, rxrpc_acks(ack_reason), serial, immediate);
37 30
38 if (prior < rxrpc_ack_priority[call->ackr_reason]) { 31 _enter("{%ld,%ld,%ld:%ld}",
39 if (immediate) 32 call->ack_at - now, call->resend_at - now, call->expire_at - now,
40 goto cancel_timer; 33 call->timer.expires - now);
41 return; 34
42 } 35 read_lock_bh(&call->state_lock);
43
44 /* update DELAY, IDLE, REQUESTED and PING_RESPONSE ACK serial
45 * numbers */
46 if (prior == rxrpc_ack_priority[call->ackr_reason]) {
47 if (prior <= 4) {
48 call->ackr_skew = skew;
49 call->ackr_serial = serial;
50 }
51 if (immediate)
52 goto cancel_timer;
53 return;
54 }
55
56 call->ackr_reason = ack_reason;
57 call->ackr_serial = serial;
58
59 switch (ack_reason) {
60 case RXRPC_ACK_DELAY:
61 _debug("run delay timer");
62 expiry = rxrpc_soft_ack_delay;
63 goto run_timer;
64
65 case RXRPC_ACK_IDLE:
66 if (!immediate) {
67 _debug("run defer timer");
68 expiry = rxrpc_idle_ack_delay;
69 goto run_timer;
70 }
71 goto cancel_timer;
72 36
73 case RXRPC_ACK_REQUESTED: 37 if (call->state < RXRPC_CALL_COMPLETE) {
74 expiry = rxrpc_requested_ack_delay; 38 t = call->ack_at;
75 if (!expiry) 39 if (time_before(call->resend_at, t))
76 goto cancel_timer; 40 t = call->resend_at;
77 if (!immediate || serial == 1) { 41 if (time_before(call->expire_at, t))
78 _debug("run defer timer"); 42 t = call->expire_at;
79 goto run_timer; 43 if (!timer_pending(&call->timer) ||
44 time_before(t, call->timer.expires)) {
45 _debug("set timer %ld", t - now);
46 mod_timer(&call->timer, t);
80 } 47 }
81
82 default:
83 _debug("immediate ACK");
84 goto cancel_timer;
85 } 48 }
86
87run_timer:
88 expiry += jiffies;
89 if (!timer_pending(&call->ack_timer) ||
90 time_after(call->ack_timer.expires, expiry))
91 mod_timer(&call->ack_timer, expiry);
92 return;
93
94cancel_timer:
95 _debug("cancel timer %%%u", serial);
96 try_to_del_timer_sync(&call->ack_timer);
97 read_lock_bh(&call->state_lock);
98 if (call->state < RXRPC_CALL_COMPLETE &&
99 !test_and_set_bit(RXRPC_CALL_EV_ACK, &call->events))
100 rxrpc_queue_call(call);
101 read_unlock_bh(&call->state_lock); 49 read_unlock_bh(&call->state_lock);
102} 50}
103 51
104/* 52/*
105 * propose an ACK be sent, locking the call structure 53 * propose an ACK be sent
106 */ 54 */
107void rxrpc_propose_ACK(struct rxrpc_call *call, u8 ack_reason, 55static void __rxrpc_propose_ACK(struct rxrpc_call *call, u8 ack_reason,
108 u16 skew, u32 serial, bool immediate) 56 u16 skew, u32 serial, bool immediate,
57 bool background)
109{ 58{
59 unsigned long now, ack_at, expiry = rxrpc_soft_ack_delay;
110 s8 prior = rxrpc_ack_priority[ack_reason]; 60 s8 prior = rxrpc_ack_priority[ack_reason];
111 61
112 if (prior > rxrpc_ack_priority[call->ackr_reason]) { 62 _enter("{%d},%s,%%%x,%u",
113 spin_lock_bh(&call->lock); 63 call->debug_id, rxrpc_acks(ack_reason), serial, immediate);
114 __rxrpc_propose_ACK(call, ack_reason, skew, serial, immediate);
115 spin_unlock_bh(&call->lock);
116 }
117}
118
119/*
120 * set the resend timer
121 */
122static void rxrpc_set_resend(struct rxrpc_call *call, u8 resend,
123 unsigned long resend_at)
124{
125 read_lock_bh(&call->state_lock);
126 if (call->state == RXRPC_CALL_COMPLETE)
127 resend = 0;
128
129 if (resend & 1) {
130 _debug("SET RESEND");
131 set_bit(RXRPC_CALL_EV_RESEND, &call->events);
132 }
133
134 if (resend & 2) {
135 _debug("MODIFY RESEND TIMER");
136 set_bit(RXRPC_CALL_RUN_RTIMER, &call->flags);
137 mod_timer(&call->resend_timer, resend_at);
138 } else {
139 _debug("KILL RESEND TIMER");
140 del_timer_sync(&call->resend_timer);
141 clear_bit(RXRPC_CALL_EV_RESEND_TIMER, &call->events);
142 clear_bit(RXRPC_CALL_RUN_RTIMER, &call->flags);
143 }
144 read_unlock_bh(&call->state_lock);
145}
146
147/*
148 * resend packets
149 */
150static void rxrpc_resend(struct rxrpc_call *call)
151{
152 struct rxrpc_wire_header *whdr;
153 struct rxrpc_skb_priv *sp;
154 struct sk_buff *txb;
155 unsigned long *p_txb, resend_at;
156 bool stop;
157 int loop;
158 u8 resend;
159
160 _enter("{%d,%d,%d,%d},",
161 call->acks_hard, call->acks_unacked,
162 atomic_read(&call->sequence),
163 CIRC_CNT(call->acks_head, call->acks_tail, call->acks_winsz));
164
165 stop = false;
166 resend = 0;
167 resend_at = 0;
168
169 for (loop = call->acks_tail;
170 loop != call->acks_head || stop;
171 loop = (loop + 1) & (call->acks_winsz - 1)
172 ) {
173 p_txb = call->acks_window + loop;
174 smp_read_barrier_depends();
175 if (*p_txb & 1)
176 continue;
177
178 txb = (struct sk_buff *) *p_txb;
179 sp = rxrpc_skb(txb);
180
181 if (sp->need_resend) {
182 sp->need_resend = false;
183
184 /* each Tx packet has a new serial number */
185 sp->hdr.serial = atomic_inc_return(&call->conn->serial);
186
187 whdr = (struct rxrpc_wire_header *)txb->head;
188 whdr->serial = htonl(sp->hdr.serial);
189
190 _proto("Tx DATA %%%u { #%d }",
191 sp->hdr.serial, sp->hdr.seq);
192 if (rxrpc_send_data_packet(call->conn, txb) < 0) {
193 stop = true;
194 sp->resend_at = jiffies + 3;
195 } else {
196 if (rxrpc_is_client_call(call))
197 rxrpc_expose_client_call(call);
198 sp->resend_at =
199 jiffies + rxrpc_resend_timeout;
200 }
201 }
202
203 if (time_after_eq(jiffies + 1, sp->resend_at)) {
204 sp->need_resend = true;
205 resend |= 1;
206 } else if (resend & 2) {
207 if (time_before(sp->resend_at, resend_at))
208 resend_at = sp->resend_at;
209 } else {
210 resend_at = sp->resend_at;
211 resend |= 2;
212 }
213 }
214
215 rxrpc_set_resend(call, resend, resend_at);
216 _leave("");
217}
218
219/*
220 * handle resend timer expiry
221 */
222static void rxrpc_resend_timer(struct rxrpc_call *call)
223{
224 struct rxrpc_skb_priv *sp;
225 struct sk_buff *txb;
226 unsigned long *p_txb, resend_at;
227 int loop;
228 u8 resend;
229
230 _enter("%d,%d,%d",
231 call->acks_tail, call->acks_unacked, call->acks_head);
232
233 if (call->state == RXRPC_CALL_COMPLETE)
234 return;
235
236 resend = 0;
237 resend_at = 0;
238
239 for (loop = call->acks_unacked;
240 loop != call->acks_head;
241 loop = (loop + 1) & (call->acks_winsz - 1)
242 ) {
243 p_txb = call->acks_window + loop;
244 smp_read_barrier_depends();
245 txb = (struct sk_buff *) (*p_txb & ~1);
246 sp = rxrpc_skb(txb);
247
248 ASSERT(!(*p_txb & 1));
249 64
250 if (sp->need_resend) { 65 /* Update DELAY, IDLE, REQUESTED and PING_RESPONSE ACK serial
251 ; 66 * numbers, but we don't alter the timeout.
252 } else if (time_after_eq(jiffies + 1, sp->resend_at)) { 67 */
253 sp->need_resend = true; 68 _debug("prior %u %u vs %u %u",
254 resend |= 1; 69 ack_reason, prior,
255 } else if (resend & 2) { 70 call->ackr_reason, rxrpc_ack_priority[call->ackr_reason]);
256 if (time_before(sp->resend_at, resend_at)) 71 if (ack_reason == call->ackr_reason) {
257 resend_at = sp->resend_at; 72 if (RXRPC_ACK_UPDATEABLE & (1 << ack_reason)) {
258 } else { 73 call->ackr_serial = serial;
259 resend_at = sp->resend_at; 74 call->ackr_skew = skew;
260 resend |= 2;
261 } 75 }
76 if (!immediate)
77 return;
78 } else if (prior > rxrpc_ack_priority[call->ackr_reason]) {
79 call->ackr_reason = ack_reason;
80 call->ackr_serial = serial;
81 call->ackr_skew = skew;
262 } 82 }
263 83
264 rxrpc_set_resend(call, resend, resend_at); 84 switch (ack_reason) {
265 _leave(""); 85 case RXRPC_ACK_REQUESTED:
266} 86 if (rxrpc_requested_ack_delay < expiry)
267 87 expiry = rxrpc_requested_ack_delay;
268/* 88 if (serial == 1)
269 * process soft ACKs of our transmitted packets 89 immediate = false;
270 * - these indicate packets the peer has or has not received, but hasn't yet 90 break;
271 * given to the consumer, and so can still be discarded and re-requested
272 */
273static int rxrpc_process_soft_ACKs(struct rxrpc_call *call,
274 struct rxrpc_ackpacket *ack,
275 struct sk_buff *skb)
276{
277 struct rxrpc_skb_priv *sp;
278 struct sk_buff *txb;
279 unsigned long *p_txb, resend_at;
280 int loop;
281 u8 sacks[RXRPC_MAXACKS], resend;
282
283 _enter("{%d,%d},{%d},",
284 call->acks_hard,
285 CIRC_CNT(call->acks_head, call->acks_tail, call->acks_winsz),
286 ack->nAcks);
287 91
288 if (skb_copy_bits(skb, 0, sacks, ack->nAcks) < 0) 92 case RXRPC_ACK_DELAY:
289 goto protocol_error; 93 if (rxrpc_soft_ack_delay < expiry)
94 expiry = rxrpc_soft_ack_delay;
95 break;
290 96
291 resend = 0; 97 case RXRPC_ACK_IDLE:
292 resend_at = 0; 98 if (rxrpc_soft_ack_delay < expiry)
293 for (loop = 0; loop < ack->nAcks; loop++) { 99 expiry = rxrpc_idle_ack_delay;
294 p_txb = call->acks_window; 100 break;
295 p_txb += (call->acks_tail + loop) & (call->acks_winsz - 1);
296 smp_read_barrier_depends();
297 txb = (struct sk_buff *) (*p_txb & ~1);
298 sp = rxrpc_skb(txb);
299 101
300 switch (sacks[loop]) { 102 default:
301 case RXRPC_ACK_TYPE_ACK: 103 immediate = true;
302 sp->need_resend = false; 104 break;
303 *p_txb |= 1;
304 break;
305 case RXRPC_ACK_TYPE_NACK:
306 sp->need_resend = true;
307 *p_txb &= ~1;
308 resend = 1;
309 break;
310 default:
311 _debug("Unsupported ACK type %d", sacks[loop]);
312 goto protocol_error;
313 }
314 } 105 }
315 106
316 smp_mb(); 107 now = jiffies;
317 call->acks_unacked = (call->acks_tail + loop) & (call->acks_winsz - 1); 108 if (test_bit(RXRPC_CALL_EV_ACK, &call->events)) {
318 109 _debug("already scheduled");
319 /* anything not explicitly ACK'd is implicitly NACK'd, but may just not 110 } else if (immediate || expiry == 0) {
320 * have been received or processed yet by the far end */ 111 _debug("immediate ACK %lx", call->events);
321 for (loop = call->acks_unacked; 112 if (!test_and_set_bit(RXRPC_CALL_EV_ACK, &call->events) &&
322 loop != call->acks_head; 113 background)
323 loop = (loop + 1) & (call->acks_winsz - 1) 114 rxrpc_queue_call(call);
324 ) { 115 } else {
325 p_txb = call->acks_window + loop; 116 ack_at = now + expiry;
326 smp_read_barrier_depends(); 117 _debug("deferred ACK %ld < %ld", expiry, call->ack_at - now);
327 txb = (struct sk_buff *) (*p_txb & ~1); 118 if (time_before(ack_at, call->ack_at)) {
328 sp = rxrpc_skb(txb); 119 call->ack_at = ack_at;
329 120 rxrpc_set_timer(call);
330 if (*p_txb & 1) {
331 /* packet must have been discarded */
332 sp->need_resend = true;
333 *p_txb &= ~1;
334 resend |= 1;
335 } else if (sp->need_resend) {
336 ;
337 } else if (time_after_eq(jiffies + 1, sp->resend_at)) {
338 sp->need_resend = true;
339 resend |= 1;
340 } else if (resend & 2) {
341 if (time_before(sp->resend_at, resend_at))
342 resend_at = sp->resend_at;
343 } else {
344 resend_at = sp->resend_at;
345 resend |= 2;
346 } 121 }
347 } 122 }
348
349 rxrpc_set_resend(call, resend, resend_at);
350 _leave(" = 0");
351 return 0;
352
353protocol_error:
354 _leave(" = -EPROTO");
355 return -EPROTO;
356} 123}
357 124
358/* 125/*
359 * discard hard-ACK'd packets from the Tx window 126 * propose an ACK be sent, locking the call structure
360 */
361static void rxrpc_rotate_tx_window(struct rxrpc_call *call, u32 hard)
362{
363 unsigned long _skb;
364 int tail = call->acks_tail, old_tail;
365 int win = CIRC_CNT(call->acks_head, tail, call->acks_winsz);
366
367 _enter("{%u,%u},%u", call->acks_hard, win, hard);
368
369 ASSERTCMP(hard - call->acks_hard, <=, win);
370
371 while (call->acks_hard < hard) {
372 smp_read_barrier_depends();
373 _skb = call->acks_window[tail] & ~1;
374 rxrpc_free_skb((struct sk_buff *) _skb);
375 old_tail = tail;
376 tail = (tail + 1) & (call->acks_winsz - 1);
377 call->acks_tail = tail;
378 if (call->acks_unacked == old_tail)
379 call->acks_unacked = tail;
380 call->acks_hard++;
381 }
382
383 wake_up(&call->waitq);
384}
385
386/*
387 * clear the Tx window in the event of a failure
388 */ 127 */
389static void rxrpc_clear_tx_window(struct rxrpc_call *call) 128void rxrpc_propose_ACK(struct rxrpc_call *call, u8 ack_reason,
129 u16 skew, u32 serial, bool immediate, bool background)
390{ 130{
391 rxrpc_rotate_tx_window(call, atomic_read(&call->sequence)); 131 spin_lock_bh(&call->lock);
132 __rxrpc_propose_ACK(call, ack_reason, skew, serial,
133 immediate, background);
134 spin_unlock_bh(&call->lock);
392} 135}
393 136
394/* 137/*
395 * drain the out of sequence received packet queue into the packet Rx queue 138 * Perform retransmission of NAK'd and unack'd packets.
396 */ 139 */
397static int rxrpc_drain_rx_oos_queue(struct rxrpc_call *call) 140static void rxrpc_resend(struct rxrpc_call *call)
398{ 141{
142 struct rxrpc_wire_header *whdr;
399 struct rxrpc_skb_priv *sp; 143 struct rxrpc_skb_priv *sp;
400 struct sk_buff *skb; 144 struct sk_buff *skb;
401 bool terminal; 145 rxrpc_seq_t cursor, seq, top;
402 int ret; 146 unsigned long resend_at, now;
147 int ix;
148 u8 annotation;
403 149
404 _enter("{%d,%d}", call->rx_data_post, call->rx_first_oos); 150 _enter("{%d,%d}", call->tx_hard_ack, call->tx_top);
405 151
406 spin_lock_bh(&call->lock); 152 spin_lock_bh(&call->lock);
407 153
408 ret = -ECONNRESET; 154 cursor = call->tx_hard_ack;
409 if (test_bit(RXRPC_CALL_RELEASED, &call->flags)) 155 top = call->tx_top;
410 goto socket_unavailable; 156 ASSERT(before_eq(cursor, top));
157 if (cursor == top)
158 goto out_unlock;
159
160 /* Scan the packet list without dropping the lock and decide which of
161 * the packets in the Tx buffer we're going to resend and what the new
162 * resend timeout will be.
163 */
164 now = jiffies;
165 resend_at = now + rxrpc_resend_timeout;
166 seq = cursor + 1;
167 do {
168 ix = seq & RXRPC_RXTX_BUFF_MASK;
169 annotation = call->rxtx_annotations[ix];
170 if (annotation == RXRPC_TX_ANNO_ACK)
171 continue;
411 172
412 skb = skb_dequeue(&call->rx_oos_queue); 173 skb = call->rxtx_buffer[ix];
413 if (skb) {
414 rxrpc_see_skb(skb); 174 rxrpc_see_skb(skb);
415 sp = rxrpc_skb(skb); 175 sp = rxrpc_skb(skb);
416 176
417 _debug("drain OOS packet %d [%d]", 177 if (annotation == RXRPC_TX_ANNO_UNACK) {
418 sp->hdr.seq, call->rx_first_oos); 178 if (time_after(sp->resend_at, now)) {
419 179 if (time_before(sp->resend_at, resend_at))
420 if (sp->hdr.seq != call->rx_first_oos) { 180 resend_at = sp->resend_at;
421 skb_queue_head(&call->rx_oos_queue, skb); 181 continue;
422 call->rx_first_oos = rxrpc_skb(skb)->hdr.seq;
423 _debug("requeue %p {%u}", skb, call->rx_first_oos);
424 } else {
425 skb->mark = RXRPC_SKB_MARK_DATA;
426 terminal = ((sp->hdr.flags & RXRPC_LAST_PACKET) &&
427 !(sp->hdr.flags & RXRPC_CLIENT_INITIATED));
428 ret = rxrpc_queue_rcv_skb(call, skb, true, terminal);
429 BUG_ON(ret < 0);
430 _debug("drain #%u", call->rx_data_post);
431 call->rx_data_post++;
432
433 /* find out what the next packet is */
434 skb = skb_peek(&call->rx_oos_queue);
435 rxrpc_see_skb(skb);
436 if (skb)
437 call->rx_first_oos = rxrpc_skb(skb)->hdr.seq;
438 else
439 call->rx_first_oos = 0;
440 _debug("peek %p {%u}", skb, call->rx_first_oos);
441 }
442 }
443
444 ret = 0;
445socket_unavailable:
446 spin_unlock_bh(&call->lock);
447 _leave(" = %d", ret);
448 return ret;
449}
450
451/*
452 * insert an out of sequence packet into the buffer
453 */
454static void rxrpc_insert_oos_packet(struct rxrpc_call *call,
455 struct sk_buff *skb)
456{
457 struct rxrpc_skb_priv *sp, *psp;
458 struct sk_buff *p;
459 u32 seq;
460
461 sp = rxrpc_skb(skb);
462 seq = sp->hdr.seq;
463 _enter(",,{%u}", seq);
464
465 skb->destructor = rxrpc_packet_destructor;
466 ASSERTCMP(sp->call, ==, NULL);
467 sp->call = call;
468 rxrpc_get_call_for_skb(call, skb);
469
470 /* insert into the buffer in sequence order */
471 spin_lock_bh(&call->lock);
472
473 skb_queue_walk(&call->rx_oos_queue, p) {
474 psp = rxrpc_skb(p);
475 if (psp->hdr.seq > seq) {
476 _debug("insert oos #%u before #%u", seq, psp->hdr.seq);
477 skb_insert(p, skb, &call->rx_oos_queue);
478 goto inserted;
479 }
480 }
481
482 _debug("append oos #%u", seq);
483 skb_queue_tail(&call->rx_oos_queue, skb);
484inserted:
485
486 /* we might now have a new front to the queue */
487 if (call->rx_first_oos == 0 || seq < call->rx_first_oos)
488 call->rx_first_oos = seq;
489
490 read_lock(&call->state_lock);
491 if (call->state < RXRPC_CALL_COMPLETE &&
492 call->rx_data_post == call->rx_first_oos) {
493 _debug("drain rx oos now");
494 set_bit(RXRPC_CALL_EV_DRAIN_RX_OOS, &call->events);
495 }
496 read_unlock(&call->state_lock);
497
498 spin_unlock_bh(&call->lock);
499 _leave(" [stored #%u]", call->rx_first_oos);
500}
501
502/*
503 * clear the Tx window on final ACK reception
504 */
505static void rxrpc_zap_tx_window(struct rxrpc_call *call)
506{
507 struct rxrpc_skb_priv *sp;
508 struct sk_buff *skb;
509 unsigned long _skb, *acks_window;
510 u8 winsz = call->acks_winsz;
511 int tail;
512
513 acks_window = call->acks_window;
514 call->acks_window = NULL;
515
516 while (CIRC_CNT(call->acks_head, call->acks_tail, winsz) > 0) {
517 tail = call->acks_tail;
518 smp_read_barrier_depends();
519 _skb = acks_window[tail] & ~1;
520 smp_mb();
521 call->acks_tail = (call->acks_tail + 1) & (winsz - 1);
522
523 skb = (struct sk_buff *) _skb;
524 sp = rxrpc_skb(skb);
525 _debug("+++ clear Tx %u", sp->hdr.seq);
526 rxrpc_free_skb(skb);
527 }
528
529 kfree(acks_window);
530}
531
532/*
533 * process the extra information that may be appended to an ACK packet
534 */
535static void rxrpc_extract_ackinfo(struct rxrpc_call *call, struct sk_buff *skb,
536 unsigned int latest, int nAcks)
537{
538 struct rxrpc_ackinfo ackinfo;
539 struct rxrpc_peer *peer;
540 unsigned int mtu;
541
542 if (skb_copy_bits(skb, nAcks + 3, &ackinfo, sizeof(ackinfo)) < 0) {
543 _leave(" [no ackinfo]");
544 return;
545 }
546
547 _proto("Rx ACK %%%u Info { rx=%u max=%u rwin=%u jm=%u }",
548 latest,
549 ntohl(ackinfo.rxMTU), ntohl(ackinfo.maxMTU),
550 ntohl(ackinfo.rwind), ntohl(ackinfo.jumbo_max));
551
552 mtu = min(ntohl(ackinfo.rxMTU), ntohl(ackinfo.maxMTU));
553
554 peer = call->peer;
555 if (mtu < peer->maxdata) {
556 spin_lock_bh(&peer->lock);
557 peer->maxdata = mtu;
558 peer->mtu = mtu + peer->hdrsize;
559 spin_unlock_bh(&peer->lock);
560 _net("Net MTU %u (maxdata %u)", peer->mtu, peer->maxdata);
561 }
562}
563
564/*
565 * process packets in the reception queue
566 */
567static int rxrpc_process_rx_queue(struct rxrpc_call *call,
568 u32 *_abort_code)
569{
570 struct rxrpc_ackpacket ack;
571 struct rxrpc_skb_priv *sp;
572 struct sk_buff *skb;
573 bool post_ACK;
574 int latest;
575 u32 hard, tx;
576
577 _enter("");
578
579process_further:
580 skb = skb_dequeue(&call->rx_queue);
581 if (!skb)
582 return -EAGAIN;
583
584 rxrpc_see_skb(skb);
585 _net("deferred skb %p", skb);
586
587 sp = rxrpc_skb(skb);
588
589 _debug("process %s [st %d]", rxrpc_pkts[sp->hdr.type], call->state);
590
591 post_ACK = false;
592
593 switch (sp->hdr.type) {
594 /* data packets that wind up here have been received out of
595 * order, need security processing or are jumbo packets */
596 case RXRPC_PACKET_TYPE_DATA:
597 _proto("OOSQ DATA %%%u { #%u }", sp->hdr.serial, sp->hdr.seq);
598
599 /* secured packets must be verified and possibly decrypted */
600 if (call->conn->security->verify_packet(call, skb,
601 sp->hdr.seq,
602 sp->hdr.cksum) < 0)
603 goto protocol_error;
604
605 rxrpc_insert_oos_packet(call, skb);
606 goto process_further;
607
608 /* partial ACK to process */
609 case RXRPC_PACKET_TYPE_ACK:
610 if (skb_copy_bits(skb, 0, &ack, sizeof(ack)) < 0) {
611 _debug("extraction failure");
612 goto protocol_error;
613 }
614 if (!skb_pull(skb, sizeof(ack)))
615 BUG();
616
617 latest = sp->hdr.serial;
618 hard = ntohl(ack.firstPacket);
619 tx = atomic_read(&call->sequence);
620
621 _proto("Rx ACK %%%u { m=%hu f=#%u p=#%u s=%%%u r=%s n=%u }",
622 latest,
623 ntohs(ack.maxSkew),
624 hard,
625 ntohl(ack.previousPacket),
626 ntohl(ack.serial),
627 rxrpc_acks(ack.reason),
628 ack.nAcks);
629
630 rxrpc_extract_ackinfo(call, skb, latest, ack.nAcks);
631
632 if (ack.reason == RXRPC_ACK_PING) {
633 _proto("Rx ACK %%%u PING Request", latest);
634 rxrpc_propose_ACK(call, RXRPC_ACK_PING_RESPONSE,
635 skb->priority, sp->hdr.serial, true);
636 }
637
638 /* discard any out-of-order or duplicate ACKs */
639 if (latest - call->acks_latest <= 0) {
640 _debug("discard ACK %d <= %d",
641 latest, call->acks_latest);
642 goto discard;
643 }
644 call->acks_latest = latest;
645
646 if (call->state != RXRPC_CALL_CLIENT_SEND_REQUEST &&
647 call->state != RXRPC_CALL_CLIENT_AWAIT_REPLY &&
648 call->state != RXRPC_CALL_SERVER_SEND_REPLY &&
649 call->state != RXRPC_CALL_SERVER_AWAIT_ACK)
650 goto discard;
651
652 _debug("Tx=%d H=%u S=%d", tx, call->acks_hard, call->state);
653
654 if (hard > 0) {
655 if (hard - 1 > tx) {
656 _debug("hard-ACK'd packet %d not transmitted"
657 " (%d top)",
658 hard - 1, tx);
659 goto protocol_error;
660 }
661
662 if ((call->state == RXRPC_CALL_CLIENT_AWAIT_REPLY ||
663 call->state == RXRPC_CALL_SERVER_AWAIT_ACK) &&
664 hard > tx) {
665 call->acks_hard = tx;
666 goto all_acked;
667 } 182 }
668
669 smp_rmb();
670 rxrpc_rotate_tx_window(call, hard - 1);
671 }
672
673 if (ack.nAcks > 0) {
674 if (hard - 1 + ack.nAcks > tx) {
675 _debug("soft-ACK'd packet %d+%d not"
676 " transmitted (%d top)",
677 hard - 1, ack.nAcks, tx);
678 goto protocol_error;
679 }
680
681 if (rxrpc_process_soft_ACKs(call, &ack, skb) < 0)
682 goto protocol_error;
683 } 183 }
684 goto discard;
685 184
686 /* complete ACK to process */ 185 /* Okay, we need to retransmit a packet. */
687 case RXRPC_PACKET_TYPE_ACKALL: 186 call->rxtx_annotations[ix] = RXRPC_TX_ANNO_RETRANS;
688 goto all_acked; 187 seq++;
689 188 } while (before_eq(seq, top));
690 /* abort and busy are handled elsewhere */ 189
691 case RXRPC_PACKET_TYPE_BUSY: 190 call->resend_at = resend_at;
692 case RXRPC_PACKET_TYPE_ABORT: 191
693 BUG(); 192 /* Now go through the Tx window and perform the retransmissions. We
694 193 * have to drop the lock for each send. If an ACK comes in whilst the
695 /* connection level events - also handled elsewhere */ 194 * lock is dropped, it may clear some of the retransmission markers for
696 case RXRPC_PACKET_TYPE_CHALLENGE: 195 * packets that it soft-ACKs.
697 case RXRPC_PACKET_TYPE_RESPONSE: 196 */
698 case RXRPC_PACKET_TYPE_DEBUG: 197 seq = cursor + 1;
699 BUG(); 198 do {
700 } 199 ix = seq & RXRPC_RXTX_BUFF_MASK;
701 200 annotation = call->rxtx_annotations[ix];
702 /* if we've had a hard ACK that covers all the packets we've sent, then 201 if (annotation != RXRPC_TX_ANNO_RETRANS)
703 * that ends that phase of the operation */ 202 continue;
704all_acked:
705 write_lock_bh(&call->state_lock);
706 _debug("ack all %d", call->state);
707
708 switch (call->state) {
709 case RXRPC_CALL_CLIENT_AWAIT_REPLY:
710 call->state = RXRPC_CALL_CLIENT_RECV_REPLY;
711 break;
712 case RXRPC_CALL_SERVER_AWAIT_ACK:
713 _debug("srv complete");
714 __rxrpc_call_completed(call);
715 post_ACK = true;
716 break;
717 case RXRPC_CALL_CLIENT_SEND_REQUEST:
718 case RXRPC_CALL_SERVER_RECV_REQUEST:
719 goto protocol_error_unlock; /* can't occur yet */
720 default:
721 write_unlock_bh(&call->state_lock);
722 goto discard; /* assume packet left over from earlier phase */
723 }
724
725 write_unlock_bh(&call->state_lock);
726
727 /* if all the packets we sent are hard-ACK'd, then we can discard
728 * whatever we've got left */
729 _debug("clear Tx %d",
730 CIRC_CNT(call->acks_head, call->acks_tail, call->acks_winsz));
731
732 del_timer_sync(&call->resend_timer);
733 clear_bit(RXRPC_CALL_RUN_RTIMER, &call->flags);
734 clear_bit(RXRPC_CALL_EV_RESEND_TIMER, &call->events);
735
736 if (call->acks_window)
737 rxrpc_zap_tx_window(call);
738 203
739 if (post_ACK) { 204 skb = call->rxtx_buffer[ix];
740 /* post the final ACK message for userspace to pick up */ 205 rxrpc_get_skb(skb);
741 _debug("post ACK");
742 skb->mark = RXRPC_SKB_MARK_FINAL_ACK;
743 sp->call = call;
744 rxrpc_get_call_for_skb(call, skb);
745 spin_lock_bh(&call->lock);
746 if (rxrpc_queue_rcv_skb(call, skb, true, true) < 0)
747 BUG();
748 spin_unlock_bh(&call->lock); 206 spin_unlock_bh(&call->lock);
749 goto process_further; 207 sp = rxrpc_skb(skb);
750 }
751
752discard:
753 rxrpc_free_skb(skb);
754 goto process_further;
755
756protocol_error_unlock:
757 write_unlock_bh(&call->state_lock);
758protocol_error:
759 rxrpc_free_skb(skb);
760 _leave(" = -EPROTO");
761 return -EPROTO;
762}
763
764/*
765 * post a message to the socket Rx queue for recvmsg() to pick up
766 */
767static int rxrpc_post_message(struct rxrpc_call *call, u32 mark, u32 error,
768 bool fatal)
769{
770 struct rxrpc_skb_priv *sp;
771 struct sk_buff *skb;
772 int ret;
773
774 _enter("{%d,%lx},%u,%u,%d",
775 call->debug_id, call->flags, mark, error, fatal);
776
777 /* remove timers and things for fatal messages */
778 if (fatal) {
779 del_timer_sync(&call->resend_timer);
780 del_timer_sync(&call->ack_timer);
781 clear_bit(RXRPC_CALL_RUN_RTIMER, &call->flags);
782 }
783 208
784 if (mark != RXRPC_SKB_MARK_NEW_CALL && 209 /* Each Tx packet needs a new serial number */
785 !test_bit(RXRPC_CALL_HAS_USERID, &call->flags)) { 210 sp->hdr.serial = atomic_inc_return(&call->conn->serial);
786 _leave("[no userid]");
787 return 0;
788 }
789 211
790 if (!test_bit(RXRPC_CALL_TERMINAL_MSG, &call->flags)) { 212 whdr = (struct rxrpc_wire_header *)skb->head;
791 skb = alloc_skb(0, GFP_NOFS); 213 whdr->serial = htonl(sp->hdr.serial);
792 if (!skb)
793 return -ENOMEM;
794 214
795 rxrpc_new_skb(skb); 215 if (rxrpc_send_data_packet(call->conn, skb) < 0) {
216 call->resend_at = now + 2;
217 rxrpc_free_skb(skb);
218 return;
219 }
796 220
797 skb->mark = mark; 221 if (rxrpc_is_client_call(call))
798 222 rxrpc_expose_client_call(call);
799 sp = rxrpc_skb(skb); 223 sp->resend_at = now + rxrpc_resend_timeout;
800 memset(sp, 0, sizeof(*sp));
801 sp->error = error;
802 sp->call = call;
803 rxrpc_get_call_for_skb(call, skb);
804 224
225 rxrpc_free_skb(skb);
805 spin_lock_bh(&call->lock); 226 spin_lock_bh(&call->lock);
806 ret = rxrpc_queue_rcv_skb(call, skb, true, fatal);
807 spin_unlock_bh(&call->lock);
808 BUG_ON(ret < 0);
809 }
810 227
811 return 0; 228 /* We need to clear the retransmit state, but there are two
229 * things we need to be aware of: A new ACK/NAK might have been
230 * received and the packet might have been hard-ACK'd (in which
231 * case it will no longer be in the buffer).
232 */
233 if (after(seq, call->tx_hard_ack) &&
234 (call->rxtx_annotations[ix] == RXRPC_TX_ANNO_RETRANS ||
235 call->rxtx_annotations[ix] == RXRPC_TX_ANNO_NAK))
236 call->rxtx_annotations[ix] = RXRPC_TX_ANNO_UNACK;
237
238 if (after(call->tx_hard_ack, seq))
239 seq = call->tx_hard_ack;
240 seq++;
241 } while (before_eq(seq, top));
242
243out_unlock:
244 spin_unlock_bh(&call->lock);
245 _leave("");
812} 246}
813 247
814/* 248/*
815 * Handle background processing of incoming call packets and ACK / abort 249 * Handle retransmission and deferred ACK/abort generation.
816 * generation. A ref on the call is donated to us by whoever queued the work
817 * item.
818 */ 250 */
819void rxrpc_process_call(struct work_struct *work) 251void rxrpc_process_call(struct work_struct *work)
820{ 252{
821 struct rxrpc_call *call = 253 struct rxrpc_call *call =
822 container_of(work, struct rxrpc_call, processor); 254 container_of(work, struct rxrpc_call, processor);
823 struct rxrpc_wire_header whdr; 255 unsigned long now;
824 struct rxrpc_ackpacket ack;
825 struct rxrpc_ackinfo ackinfo;
826 struct msghdr msg;
827 struct kvec iov[5];
828 enum rxrpc_call_event genbit;
829 unsigned long bits;
830 __be32 data, pad;
831 size_t len;
832 bool requeue = false;
833 int loop, nbit, ioc, ret, mtu;
834 u32 serial, abort_code = RX_PROTOCOL_ERROR;
835 u8 *acks = NULL;
836 256
837 rxrpc_see_call(call); 257 rxrpc_see_call(call);
838 258
839 //printk("\n--------------------\n"); 259 //printk("\n--------------------\n");
840 _enter("{%d,%s,%lx} [%lu]", 260 _enter("{%d,%s,%lx}",
841 call->debug_id, rxrpc_call_states[call->state], call->events, 261 call->debug_id, rxrpc_call_states[call->state], call->events);
842 (jiffies - call->creation_jif) / (HZ / 10));
843
844 if (call->state >= RXRPC_CALL_COMPLETE) {
845 rxrpc_put_call(call, rxrpc_call_put);
846 return;
847 }
848
849 if (!call->conn)
850 goto skip_msg_init;
851
852 /* there's a good chance we're going to have to send a message, so set
853 * one up in advance */
854 msg.msg_name = &call->peer->srx.transport;
855 msg.msg_namelen = call->peer->srx.transport_len;
856 msg.msg_control = NULL;
857 msg.msg_controllen = 0;
858 msg.msg_flags = 0;
859 262
860 whdr.epoch = htonl(call->conn->proto.epoch); 263recheck_state:
861 whdr.cid = htonl(call->cid); 264 if (test_and_clear_bit(RXRPC_CALL_EV_ABORT, &call->events)) {
862 whdr.callNumber = htonl(call->call_id); 265 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ABORT);
863 whdr.seq = 0; 266 goto recheck_state;
864 whdr.type = RXRPC_PACKET_TYPE_ACK;
865 whdr.flags = call->conn->out_clientflag;
866 whdr.userStatus = 0;
867 whdr.securityIndex = call->conn->security_ix;
868 whdr._rsvd = 0;
869 whdr.serviceId = htons(call->service_id);
870
871 memset(iov, 0, sizeof(iov));
872 iov[0].iov_base = &whdr;
873 iov[0].iov_len = sizeof(whdr);
874skip_msg_init:
875
876 /* deal with events of a final nature */
877 if (test_bit(RXRPC_CALL_EV_RCVD_ERROR, &call->events)) {
878 enum rxrpc_skb_mark mark;
879
880 clear_bit(RXRPC_CALL_EV_CONN_ABORT, &call->events);
881 clear_bit(RXRPC_CALL_EV_REJECT_BUSY, &call->events);
882 clear_bit(RXRPC_CALL_EV_ABORT, &call->events);
883
884 if (call->completion == RXRPC_CALL_NETWORK_ERROR) {
885 mark = RXRPC_SKB_MARK_NET_ERROR;
886 _debug("post net error %d", call->error);
887 } else {
888 mark = RXRPC_SKB_MARK_LOCAL_ERROR;
889 _debug("post net local error %d", call->error);
890 }
891
892 if (rxrpc_post_message(call, mark, call->error, true) < 0)
893 goto no_mem;
894 clear_bit(RXRPC_CALL_EV_RCVD_ERROR, &call->events);
895 goto kill_ACKs;
896 }
897
898 if (test_bit(RXRPC_CALL_EV_CONN_ABORT, &call->events)) {
899 ASSERTCMP(call->state, ==, RXRPC_CALL_COMPLETE);
900
901 clear_bit(RXRPC_CALL_EV_REJECT_BUSY, &call->events);
902 clear_bit(RXRPC_CALL_EV_ABORT, &call->events);
903
904 _debug("post conn abort");
905
906 if (rxrpc_post_message(call, RXRPC_SKB_MARK_LOCAL_ERROR,
907 call->error, true) < 0)
908 goto no_mem;
909 clear_bit(RXRPC_CALL_EV_CONN_ABORT, &call->events);
910 goto kill_ACKs;
911 }
912
913 if (test_bit(RXRPC_CALL_EV_REJECT_BUSY, &call->events)) {
914 whdr.type = RXRPC_PACKET_TYPE_BUSY;
915 genbit = RXRPC_CALL_EV_REJECT_BUSY;
916 goto send_message;
917 }
918
919 if (test_bit(RXRPC_CALL_EV_ABORT, &call->events)) {
920 ASSERTCMP(call->state, ==, RXRPC_CALL_COMPLETE);
921
922 if (rxrpc_post_message(call, RXRPC_SKB_MARK_LOCAL_ERROR,
923 call->error, true) < 0)
924 goto no_mem;
925 whdr.type = RXRPC_PACKET_TYPE_ABORT;
926 data = htonl(call->abort_code);
927 iov[1].iov_base = &data;
928 iov[1].iov_len = sizeof(data);
929 genbit = RXRPC_CALL_EV_ABORT;
930 goto send_message;
931 }
932
933 if (test_bit(RXRPC_CALL_EV_ACK_FINAL, &call->events)) {
934 genbit = RXRPC_CALL_EV_ACK_FINAL;
935
936 ack.bufferSpace = htons(8);
937 ack.maxSkew = 0;
938 ack.serial = 0;
939 ack.reason = RXRPC_ACK_IDLE;
940 ack.nAcks = 0;
941 call->ackr_reason = 0;
942
943 spin_lock_bh(&call->lock);
944 ack.serial = htonl(call->ackr_serial);
945 ack.previousPacket = htonl(call->ackr_prev_seq);
946 ack.firstPacket = htonl(call->rx_data_eaten + 1);
947 spin_unlock_bh(&call->lock);
948
949 pad = 0;
950
951 iov[1].iov_base = &ack;
952 iov[1].iov_len = sizeof(ack);
953 iov[2].iov_base = &pad;
954 iov[2].iov_len = 3;
955 iov[3].iov_base = &ackinfo;
956 iov[3].iov_len = sizeof(ackinfo);
957 goto send_ACK;
958 } 267 }
959 268
960 if (call->events & ((1 << RXRPC_CALL_EV_RCVD_BUSY) | 269 if (call->state == RXRPC_CALL_COMPLETE) {
961 (1 << RXRPC_CALL_EV_RCVD_ABORT)) 270 del_timer_sync(&call->timer);
962 ) { 271 goto out_put;
963 u32 mark;
964
965 if (test_bit(RXRPC_CALL_EV_RCVD_ABORT, &call->events))
966 mark = RXRPC_SKB_MARK_REMOTE_ABORT;
967 else
968 mark = RXRPC_SKB_MARK_BUSY;
969
970 _debug("post abort/busy");
971 rxrpc_clear_tx_window(call);
972 if (rxrpc_post_message(call, mark, ECONNABORTED, true) < 0)
973 goto no_mem;
974
975 clear_bit(RXRPC_CALL_EV_RCVD_BUSY, &call->events);
976 clear_bit(RXRPC_CALL_EV_RCVD_ABORT, &call->events);
977 goto kill_ACKs;
978 } 272 }
979 273
980 if (test_and_clear_bit(RXRPC_CALL_EV_RCVD_ACKALL, &call->events)) { 274 now = jiffies;
981 _debug("do implicit ackall"); 275 if (time_after_eq(now, call->expire_at)) {
982 rxrpc_clear_tx_window(call);
983 }
984
985 if (test_bit(RXRPC_CALL_EV_LIFE_TIMER, &call->events)) {
986 rxrpc_abort_call("EXP", call, 0, RX_CALL_TIMEOUT, ETIME); 276 rxrpc_abort_call("EXP", call, 0, RX_CALL_TIMEOUT, ETIME);
987 277 set_bit(RXRPC_CALL_EV_ABORT, &call->events);
988 _debug("post timeout");
989 if (rxrpc_post_message(call, RXRPC_SKB_MARK_LOCAL_ERROR,
990 ETIME, true) < 0)
991 goto no_mem;
992
993 clear_bit(RXRPC_CALL_EV_LIFE_TIMER, &call->events);
994 goto kill_ACKs;
995 } 278 }
996 279
997 /* deal with assorted inbound messages */ 280 if (test_and_clear_bit(RXRPC_CALL_EV_ACK, &call->events) ||
998 if (!skb_queue_empty(&call->rx_queue)) { 281 time_after_eq(now, call->ack_at)) {
999 ret = rxrpc_process_rx_queue(call, &abort_code); 282 call->ack_at = call->expire_at;
1000 switch (ret) { 283 if (call->ackr_reason) {
1001 case 0: 284 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ACK);
1002 case -EAGAIN: 285 goto recheck_state;
1003 break;
1004 case -ENOMEM:
1005 goto no_mem;
1006 case -EKEYEXPIRED:
1007 case -EKEYREJECTED:
1008 case -EPROTO:
1009 rxrpc_abort_call("PRO", call, 0, abort_code, -ret);
1010 goto kill_ACKs;
1011 } 286 }
1012 } 287 }
1013 288
1014 /* handle resending */ 289 if (test_and_clear_bit(RXRPC_CALL_EV_RESEND, &call->events) ||
1015 if (test_and_clear_bit(RXRPC_CALL_EV_RESEND_TIMER, &call->events)) 290 time_after_eq(now, call->resend_at)) {
1016 rxrpc_resend_timer(call);
1017 if (test_and_clear_bit(RXRPC_CALL_EV_RESEND, &call->events))
1018 rxrpc_resend(call); 291 rxrpc_resend(call);
1019 292 goto recheck_state;
1020 /* consider sending an ordinary ACK */
1021 if (test_bit(RXRPC_CALL_EV_ACK, &call->events)) {
1022 _debug("send ACK: window: %d - %d { %lx }",
1023 call->rx_data_eaten, call->ackr_win_top,
1024 call->ackr_window[0]);
1025
1026 if (call->state > RXRPC_CALL_SERVER_ACK_REQUEST &&
1027 call->ackr_reason != RXRPC_ACK_PING_RESPONSE) {
1028 /* ACK by sending reply DATA packet in this state */
1029 clear_bit(RXRPC_CALL_EV_ACK, &call->events);
1030 goto maybe_reschedule;
1031 }
1032
1033 genbit = RXRPC_CALL_EV_ACK;
1034
1035 acks = kzalloc(call->ackr_win_top - call->rx_data_eaten,
1036 GFP_NOFS);
1037 if (!acks)
1038 goto no_mem;
1039
1040 //hdr.flags = RXRPC_SLOW_START_OK;
1041 ack.bufferSpace = htons(8);
1042 ack.maxSkew = 0;
1043
1044 spin_lock_bh(&call->lock);
1045 ack.reason = call->ackr_reason;
1046 ack.serial = htonl(call->ackr_serial);
1047 ack.previousPacket = htonl(call->ackr_prev_seq);
1048 ack.firstPacket = htonl(call->rx_data_eaten + 1);
1049
1050 ack.nAcks = 0;
1051 for (loop = 0; loop < RXRPC_ACKR_WINDOW_ASZ; loop++) {
1052 nbit = loop * BITS_PER_LONG;
1053 for (bits = call->ackr_window[loop]; bits; bits >>= 1
1054 ) {
1055 _debug("- l=%d n=%d b=%lx", loop, nbit, bits);
1056 if (bits & 1) {
1057 acks[nbit] = RXRPC_ACK_TYPE_ACK;
1058 ack.nAcks = nbit + 1;
1059 }
1060 nbit++;
1061 }
1062 }
1063 call->ackr_reason = 0;
1064 spin_unlock_bh(&call->lock);
1065
1066 pad = 0;
1067
1068 iov[1].iov_base = &ack;
1069 iov[1].iov_len = sizeof(ack);
1070 iov[2].iov_base = acks;
1071 iov[2].iov_len = ack.nAcks;
1072 iov[3].iov_base = &pad;
1073 iov[3].iov_len = 3;
1074 iov[4].iov_base = &ackinfo;
1075 iov[4].iov_len = sizeof(ackinfo);
1076
1077 switch (ack.reason) {
1078 case RXRPC_ACK_REQUESTED:
1079 case RXRPC_ACK_DUPLICATE:
1080 case RXRPC_ACK_OUT_OF_SEQUENCE:
1081 case RXRPC_ACK_EXCEEDS_WINDOW:
1082 case RXRPC_ACK_NOSPACE:
1083 case RXRPC_ACK_PING:
1084 case RXRPC_ACK_PING_RESPONSE:
1085 goto send_ACK_with_skew;
1086 case RXRPC_ACK_DELAY:
1087 case RXRPC_ACK_IDLE:
1088 goto send_ACK;
1089 }
1090 } 293 }
1091 294
1092 /* handle completion of security negotiations on an incoming 295 rxrpc_set_timer(call);
1093 * connection */
1094 if (test_and_clear_bit(RXRPC_CALL_EV_SECURED, &call->events)) {
1095 _debug("secured");
1096 spin_lock_bh(&call->lock);
1097
1098 if (call->state == RXRPC_CALL_SERVER_SECURING) {
1099 struct rxrpc_sock *rx;
1100 _debug("securing");
1101 rcu_read_lock();
1102 rx = rcu_dereference(call->socket);
1103 if (rx) {
1104 write_lock(&rx->call_lock);
1105 if (!test_bit(RXRPC_CALL_RELEASED, &call->flags)) {
1106 _debug("not released");
1107 call->state = RXRPC_CALL_SERVER_ACCEPTING;
1108 list_move_tail(&call->accept_link,
1109 &rx->acceptq);
1110 }
1111 write_unlock(&rx->call_lock);
1112 }
1113 rcu_read_unlock();
1114 read_lock(&call->state_lock);
1115 if (call->state < RXRPC_CALL_COMPLETE)
1116 set_bit(RXRPC_CALL_EV_POST_ACCEPT, &call->events);
1117 read_unlock(&call->state_lock);
1118 }
1119
1120 spin_unlock_bh(&call->lock);
1121 if (!test_bit(RXRPC_CALL_EV_POST_ACCEPT, &call->events))
1122 goto maybe_reschedule;
1123 }
1124
1125 /* post a notification of an acceptable connection to the app */
1126 if (test_bit(RXRPC_CALL_EV_POST_ACCEPT, &call->events)) {
1127 _debug("post accept");
1128 if (rxrpc_post_message(call, RXRPC_SKB_MARK_NEW_CALL,
1129 0, false) < 0)
1130 goto no_mem;
1131 clear_bit(RXRPC_CALL_EV_POST_ACCEPT, &call->events);
1132 goto maybe_reschedule;
1133 }
1134
1135 /* handle incoming call acceptance */
1136 if (test_and_clear_bit(RXRPC_CALL_EV_ACCEPTED, &call->events)) {
1137 _debug("accepted");
1138 ASSERTCMP(call->rx_data_post, ==, 0);
1139 call->rx_data_post = 1;
1140 read_lock_bh(&call->state_lock);
1141 if (call->state < RXRPC_CALL_COMPLETE)
1142 set_bit(RXRPC_CALL_EV_DRAIN_RX_OOS, &call->events);
1143 read_unlock_bh(&call->state_lock);
1144 }
1145
1146 /* drain the out of sequence received packet queue into the packet Rx
1147 * queue */
1148 if (test_and_clear_bit(RXRPC_CALL_EV_DRAIN_RX_OOS, &call->events)) {
1149 while (call->rx_data_post == call->rx_first_oos)
1150 if (rxrpc_drain_rx_oos_queue(call) < 0)
1151 break;
1152 goto maybe_reschedule;
1153 }
1154 296
1155 /* other events may have been raised since we started checking */ 297 /* other events may have been raised since we started checking */
1156 goto maybe_reschedule; 298 if (call->events && call->state < RXRPC_CALL_COMPLETE) {
1157
1158send_ACK_with_skew:
1159 ack.maxSkew = htons(call->ackr_skew);
1160send_ACK:
1161 mtu = call->peer->if_mtu;
1162 mtu -= call->peer->hdrsize;
1163 ackinfo.maxMTU = htonl(mtu);
1164 ackinfo.rwind = htonl(rxrpc_rx_window_size);
1165
1166 /* permit the peer to send us jumbo packets if it wants to */
1167 ackinfo.rxMTU = htonl(rxrpc_rx_mtu);
1168 ackinfo.jumbo_max = htonl(rxrpc_rx_jumbo_max);
1169
1170 serial = atomic_inc_return(&call->conn->serial);
1171 whdr.serial = htonl(serial);
1172 _proto("Tx ACK %%%u { m=%hu f=#%u p=#%u s=%%%u r=%s n=%u }",
1173 serial,
1174 ntohs(ack.maxSkew),
1175 ntohl(ack.firstPacket),
1176 ntohl(ack.previousPacket),
1177 ntohl(ack.serial),
1178 rxrpc_acks(ack.reason),
1179 ack.nAcks);
1180
1181 del_timer_sync(&call->ack_timer);
1182 if (ack.nAcks > 0)
1183 set_bit(RXRPC_CALL_TX_SOFT_ACK, &call->flags);
1184 goto send_message_2;
1185
1186send_message:
1187 _debug("send message");
1188
1189 serial = atomic_inc_return(&call->conn->serial);
1190 whdr.serial = htonl(serial);
1191 _proto("Tx %s %%%u", rxrpc_pkts[whdr.type], serial);
1192send_message_2:
1193
1194 len = iov[0].iov_len;
1195 ioc = 1;
1196 if (iov[4].iov_len) {
1197 ioc = 5;
1198 len += iov[4].iov_len;
1199 len += iov[3].iov_len;
1200 len += iov[2].iov_len;
1201 len += iov[1].iov_len;
1202 } else if (iov[3].iov_len) {
1203 ioc = 4;
1204 len += iov[3].iov_len;
1205 len += iov[2].iov_len;
1206 len += iov[1].iov_len;
1207 } else if (iov[2].iov_len) {
1208 ioc = 3;
1209 len += iov[2].iov_len;
1210 len += iov[1].iov_len;
1211 } else if (iov[1].iov_len) {
1212 ioc = 2;
1213 len += iov[1].iov_len;
1214 }
1215
1216 ret = kernel_sendmsg(call->conn->params.local->socket,
1217 &msg, iov, ioc, len);
1218 if (ret < 0) {
1219 _debug("sendmsg failed: %d", ret);
1220 if (call->state < RXRPC_CALL_COMPLETE)
1221 requeue = true;
1222 goto error;
1223 }
1224
1225 switch (genbit) {
1226 case RXRPC_CALL_EV_ABORT:
1227 clear_bit(genbit, &call->events);
1228 clear_bit(RXRPC_CALL_EV_RCVD_ABORT, &call->events);
1229 goto kill_ACKs;
1230
1231 case RXRPC_CALL_EV_ACK_FINAL:
1232 rxrpc_call_completed(call);
1233 goto kill_ACKs;
1234
1235 default:
1236 clear_bit(genbit, &call->events);
1237 switch (call->state) {
1238 case RXRPC_CALL_CLIENT_AWAIT_REPLY:
1239 case RXRPC_CALL_CLIENT_RECV_REPLY:
1240 case RXRPC_CALL_SERVER_RECV_REQUEST:
1241 case RXRPC_CALL_SERVER_ACK_REQUEST:
1242 _debug("start ACK timer");
1243 rxrpc_propose_ACK(call, RXRPC_ACK_DELAY,
1244 call->ackr_skew, call->ackr_serial,
1245 false);
1246 default:
1247 break;
1248 }
1249 goto maybe_reschedule;
1250 }
1251
1252kill_ACKs:
1253 del_timer_sync(&call->ack_timer);
1254 clear_bit(RXRPC_CALL_EV_ACK, &call->events);
1255
1256maybe_reschedule:
1257 if (call->events || !skb_queue_empty(&call->rx_queue)) {
1258 if (call->state < RXRPC_CALL_COMPLETE)
1259 requeue = true;
1260 }
1261
1262error:
1263 kfree(acks);
1264
1265 if ((requeue || call->events) && !work_pending(&call->processor)) {
1266 _debug("jumpstart %x", call->conn->proto.cid);
1267 __rxrpc_queue_call(call); 299 __rxrpc_queue_call(call);
1268 } else { 300 goto out;
1269 rxrpc_put_call(call, rxrpc_call_put);
1270 } 301 }
1271 302
303out_put:
304 rxrpc_put_call(call, rxrpc_call_put);
305out:
1272 _leave(""); 306 _leave("");
1273 return;
1274
1275no_mem:
1276 _debug("out of memory");
1277 goto maybe_reschedule;
1278} 307}
diff --git a/net/rxrpc/call_object.c b/net/rxrpc/call_object.c
index d233adc9b5e5..18ab13f82f6e 100644
--- a/net/rxrpc/call_object.c
+++ b/net/rxrpc/call_object.c
@@ -30,7 +30,6 @@ const char *const rxrpc_call_states[NR__RXRPC_CALL_STATES] = {
30 [RXRPC_CALL_CLIENT_SEND_REQUEST] = "ClSndReq", 30 [RXRPC_CALL_CLIENT_SEND_REQUEST] = "ClSndReq",
31 [RXRPC_CALL_CLIENT_AWAIT_REPLY] = "ClAwtRpl", 31 [RXRPC_CALL_CLIENT_AWAIT_REPLY] = "ClAwtRpl",
32 [RXRPC_CALL_CLIENT_RECV_REPLY] = "ClRcvRpl", 32 [RXRPC_CALL_CLIENT_RECV_REPLY] = "ClRcvRpl",
33 [RXRPC_CALL_CLIENT_FINAL_ACK] = "ClFnlACK",
34 [RXRPC_CALL_SERVER_PREALLOC] = "SvPrealc", 33 [RXRPC_CALL_SERVER_PREALLOC] = "SvPrealc",
35 [RXRPC_CALL_SERVER_SECURING] = "SvSecure", 34 [RXRPC_CALL_SERVER_SECURING] = "SvSecure",
36 [RXRPC_CALL_SERVER_ACCEPTING] = "SvAccept", 35 [RXRPC_CALL_SERVER_ACCEPTING] = "SvAccept",
@@ -43,7 +42,6 @@ const char *const rxrpc_call_states[NR__RXRPC_CALL_STATES] = {
43 42
44const char *const rxrpc_call_completions[NR__RXRPC_CALL_COMPLETIONS] = { 43const char *const rxrpc_call_completions[NR__RXRPC_CALL_COMPLETIONS] = {
45 [RXRPC_CALL_SUCCEEDED] = "Complete", 44 [RXRPC_CALL_SUCCEEDED] = "Complete",
46 [RXRPC_CALL_SERVER_BUSY] = "SvBusy ",
47 [RXRPC_CALL_REMOTELY_ABORTED] = "RmtAbort", 45 [RXRPC_CALL_REMOTELY_ABORTED] = "RmtAbort",
48 [RXRPC_CALL_LOCALLY_ABORTED] = "LocAbort", 46 [RXRPC_CALL_LOCALLY_ABORTED] = "LocAbort",
49 [RXRPC_CALL_LOCAL_ERROR] = "LocError", 47 [RXRPC_CALL_LOCAL_ERROR] = "LocError",
@@ -57,10 +55,8 @@ const char rxrpc_call_traces[rxrpc_call__nr_trace][4] = {
57 [rxrpc_call_queued_ref] = "QUR", 55 [rxrpc_call_queued_ref] = "QUR",
58 [rxrpc_call_seen] = "SEE", 56 [rxrpc_call_seen] = "SEE",
59 [rxrpc_call_got] = "GOT", 57 [rxrpc_call_got] = "GOT",
60 [rxrpc_call_got_skb] = "Gsk",
61 [rxrpc_call_got_userid] = "Gus", 58 [rxrpc_call_got_userid] = "Gus",
62 [rxrpc_call_put] = "PUT", 59 [rxrpc_call_put] = "PUT",
63 [rxrpc_call_put_skb] = "Psk",
64 [rxrpc_call_put_userid] = "Pus", 60 [rxrpc_call_put_userid] = "Pus",
65 [rxrpc_call_put_noqueue] = "PNQ", 61 [rxrpc_call_put_noqueue] = "PNQ",
66}; 62};
@@ -69,9 +65,15 @@ struct kmem_cache *rxrpc_call_jar;
69LIST_HEAD(rxrpc_calls); 65LIST_HEAD(rxrpc_calls);
70DEFINE_RWLOCK(rxrpc_call_lock); 66DEFINE_RWLOCK(rxrpc_call_lock);
71 67
72static void rxrpc_call_life_expired(unsigned long _call); 68static void rxrpc_call_timer_expired(unsigned long _call)
73static void rxrpc_ack_time_expired(unsigned long _call); 69{
74static void rxrpc_resend_time_expired(unsigned long _call); 70 struct rxrpc_call *call = (struct rxrpc_call *)_call;
71
72 _enter("%d", call->debug_id);
73
74 if (call->state < RXRPC_CALL_COMPLETE)
75 rxrpc_queue_call(call);
76}
75 77
76/* 78/*
77 * find an extant server call 79 * find an extant server call
@@ -121,27 +123,24 @@ struct rxrpc_call *rxrpc_alloc_call(gfp_t gfp)
121 if (!call) 123 if (!call)
122 return NULL; 124 return NULL;
123 125
124 call->acks_winsz = 16; 126 call->rxtx_buffer = kcalloc(RXRPC_RXTX_BUFF_SIZE,
125 call->acks_window = kmalloc(call->acks_winsz * sizeof(unsigned long), 127 sizeof(struct sk_buff *),
126 gfp); 128 gfp);
127 if (!call->acks_window) { 129 if (!call->rxtx_buffer)
128 kmem_cache_free(rxrpc_call_jar, call); 130 goto nomem;
129 return NULL;
130 }
131 131
132 setup_timer(&call->lifetimer, &rxrpc_call_life_expired, 132 call->rxtx_annotations = kcalloc(RXRPC_RXTX_BUFF_SIZE, sizeof(u8), gfp);
133 (unsigned long) call); 133 if (!call->rxtx_annotations)
134 setup_timer(&call->ack_timer, &rxrpc_ack_time_expired, 134 goto nomem_2;
135 (unsigned long) call); 135
136 setup_timer(&call->resend_timer, &rxrpc_resend_time_expired, 136 setup_timer(&call->timer, rxrpc_call_timer_expired,
137 (unsigned long) call); 137 (unsigned long)call);
138 INIT_WORK(&call->processor, &rxrpc_process_call); 138 INIT_WORK(&call->processor, &rxrpc_process_call);
139 INIT_LIST_HEAD(&call->link); 139 INIT_LIST_HEAD(&call->link);
140 INIT_LIST_HEAD(&call->chan_wait_link); 140 INIT_LIST_HEAD(&call->chan_wait_link);
141 INIT_LIST_HEAD(&call->accept_link); 141 INIT_LIST_HEAD(&call->accept_link);
142 skb_queue_head_init(&call->rx_queue); 142 INIT_LIST_HEAD(&call->recvmsg_link);
143 skb_queue_head_init(&call->rx_oos_queue); 143 INIT_LIST_HEAD(&call->sock_link);
144 skb_queue_head_init(&call->knlrecv_queue);
145 init_waitqueue_head(&call->waitq); 144 init_waitqueue_head(&call->waitq);
146 spin_lock_init(&call->lock); 145 spin_lock_init(&call->lock);
147 rwlock_init(&call->state_lock); 146 rwlock_init(&call->state_lock);
@@ -150,63 +149,52 @@ struct rxrpc_call *rxrpc_alloc_call(gfp_t gfp)
150 149
151 memset(&call->sock_node, 0xed, sizeof(call->sock_node)); 150 memset(&call->sock_node, 0xed, sizeof(call->sock_node));
152 151
153 call->rx_data_expect = 1; 152 /* Leave space in the ring to handle a maxed-out jumbo packet */
154 call->rx_data_eaten = 0; 153 call->rx_winsize = RXRPC_RXTX_BUFF_SIZE - 1 - 46;
155 call->rx_first_oos = 0; 154 call->tx_winsize = 16;
156 call->ackr_win_top = call->rx_data_eaten + 1 + rxrpc_rx_window_size; 155 call->rx_expect_next = 1;
157 call->creation_jif = jiffies;
158 return call; 156 return call;
157
158nomem_2:
159 kfree(call->rxtx_buffer);
160nomem:
161 kmem_cache_free(rxrpc_call_jar, call);
162 return NULL;
159} 163}
160 164
161/* 165/*
162 * Allocate a new client call. 166 * Allocate a new client call.
163 */ 167 */
164static struct rxrpc_call *rxrpc_alloc_client_call(struct rxrpc_sock *rx, 168static struct rxrpc_call *rxrpc_alloc_client_call(struct sockaddr_rxrpc *srx,
165 struct sockaddr_rxrpc *srx,
166 gfp_t gfp) 169 gfp_t gfp)
167{ 170{
168 struct rxrpc_call *call; 171 struct rxrpc_call *call;
169 172
170 _enter(""); 173 _enter("");
171 174
172 ASSERT(rx->local != NULL);
173
174 call = rxrpc_alloc_call(gfp); 175 call = rxrpc_alloc_call(gfp);
175 if (!call) 176 if (!call)
176 return ERR_PTR(-ENOMEM); 177 return ERR_PTR(-ENOMEM);
177 call->state = RXRPC_CALL_CLIENT_AWAIT_CONN; 178 call->state = RXRPC_CALL_CLIENT_AWAIT_CONN;
178 call->rx_data_post = 1;
179 call->service_id = srx->srx_service; 179 call->service_id = srx->srx_service;
180 rcu_assign_pointer(call->socket, rx);
181 180
182 _leave(" = %p", call); 181 _leave(" = %p", call);
183 return call; 182 return call;
184} 183}
185 184
186/* 185/*
187 * Begin client call. 186 * Initiate the call ack/resend/expiry timer.
188 */ 187 */
189static int rxrpc_begin_client_call(struct rxrpc_call *call, 188static void rxrpc_start_call_timer(struct rxrpc_call *call)
190 struct rxrpc_conn_parameters *cp,
191 struct sockaddr_rxrpc *srx,
192 gfp_t gfp)
193{ 189{
194 int ret; 190 unsigned long expire_at;
195 191
196 /* Set up or get a connection record and set the protocol parameters, 192 expire_at = jiffies + rxrpc_max_call_lifetime;
197 * including channel number and call ID. 193 call->expire_at = expire_at;
198 */ 194 call->ack_at = expire_at;
199 ret = rxrpc_connect_call(call, cp, srx, gfp); 195 call->resend_at = expire_at;
200 if (ret < 0) 196 call->timer.expires = expire_at;
201 return ret; 197 add_timer(&call->timer);
202
203 spin_lock(&call->conn->params.peer->lock);
204 hlist_add_head(&call->error_link, &call->conn->params.peer->error_targets);
205 spin_unlock(&call->conn->params.peer->lock);
206
207 call->lifetimer.expires = jiffies + rxrpc_max_call_lifetime;
208 add_timer(&call->lifetimer);
209 return 0;
210} 198}
211 199
212/* 200/*
@@ -226,7 +214,7 @@ struct rxrpc_call *rxrpc_new_client_call(struct rxrpc_sock *rx,
226 214
227 _enter("%p,%lx", rx, user_call_ID); 215 _enter("%p,%lx", rx, user_call_ID);
228 216
229 call = rxrpc_alloc_client_call(rx, srx, gfp); 217 call = rxrpc_alloc_client_call(srx, gfp);
230 if (IS_ERR(call)) { 218 if (IS_ERR(call)) {
231 _leave(" = %ld", PTR_ERR(call)); 219 _leave(" = %ld", PTR_ERR(call));
232 return call; 220 return call;
@@ -255,19 +243,32 @@ struct rxrpc_call *rxrpc_new_client_call(struct rxrpc_sock *rx,
255 goto found_user_ID_now_present; 243 goto found_user_ID_now_present;
256 } 244 }
257 245
246 rcu_assign_pointer(call->socket, rx);
258 rxrpc_get_call(call, rxrpc_call_got_userid); 247 rxrpc_get_call(call, rxrpc_call_got_userid);
259 rb_link_node(&call->sock_node, parent, pp); 248 rb_link_node(&call->sock_node, parent, pp);
260 rb_insert_color(&call->sock_node, &rx->calls); 249 rb_insert_color(&call->sock_node, &rx->calls);
250 list_add(&call->sock_link, &rx->sock_calls);
251
261 write_unlock(&rx->call_lock); 252 write_unlock(&rx->call_lock);
262 253
263 write_lock_bh(&rxrpc_call_lock); 254 write_lock(&rxrpc_call_lock);
264 list_add_tail(&call->link, &rxrpc_calls); 255 list_add_tail(&call->link, &rxrpc_calls);
265 write_unlock_bh(&rxrpc_call_lock); 256 write_unlock(&rxrpc_call_lock);
266 257
267 ret = rxrpc_begin_client_call(call, cp, srx, gfp); 258 /* Set up or get a connection record and set the protocol parameters,
259 * including channel number and call ID.
260 */
261 ret = rxrpc_connect_call(call, cp, srx, gfp);
268 if (ret < 0) 262 if (ret < 0)
269 goto error; 263 goto error;
270 264
265 spin_lock_bh(&call->conn->params.peer->lock);
266 hlist_add_head(&call->error_link,
267 &call->conn->params.peer->error_targets);
268 spin_unlock_bh(&call->conn->params.peer->lock);
269
270 rxrpc_start_call_timer(call);
271
271 _net("CALL new %d on CONN %d", call->debug_id, call->conn->debug_id); 272 _net("CALL new %d on CONN %d", call->debug_id, call->conn->debug_id);
272 273
273 _leave(" = %p [new]", call); 274 _leave(" = %p [new]", call);
@@ -279,9 +280,9 @@ error:
279 write_unlock(&rx->call_lock); 280 write_unlock(&rx->call_lock);
280 rxrpc_put_call(call, rxrpc_call_put_userid); 281 rxrpc_put_call(call, rxrpc_call_put_userid);
281 282
282 write_lock_bh(&rxrpc_call_lock); 283 write_lock(&rxrpc_call_lock);
283 list_del_init(&call->link); 284 list_del_init(&call->link);
284 write_unlock_bh(&rxrpc_call_lock); 285 write_unlock(&rxrpc_call_lock);
285 286
286error_out: 287error_out:
287 __rxrpc_set_call_completion(call, RXRPC_CALL_LOCAL_ERROR, 288 __rxrpc_set_call_completion(call, RXRPC_CALL_LOCAL_ERROR,
@@ -303,142 +304,46 @@ found_user_ID_now_present:
303} 304}
304 305
305/* 306/*
306 * set up an incoming call 307 * Set up an incoming call. call->conn points to the connection.
307 * - called in process context with IRQs enabled 308 * This is called in BH context and isn't allowed to fail.
308 */ 309 */
309struct rxrpc_call *rxrpc_incoming_call(struct rxrpc_sock *rx, 310void rxrpc_incoming_call(struct rxrpc_sock *rx,
310 struct rxrpc_connection *conn, 311 struct rxrpc_call *call,
311 struct sk_buff *skb) 312 struct sk_buff *skb)
312{ 313{
314 struct rxrpc_connection *conn = call->conn;
313 struct rxrpc_skb_priv *sp = rxrpc_skb(skb); 315 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
314 struct rxrpc_call *call, *candidate; 316 u32 chan;
315 const void *here = __builtin_return_address(0);
316 u32 call_id, chan;
317
318 _enter(",%d", conn->debug_id);
319
320 ASSERT(rx != NULL);
321
322 candidate = rxrpc_alloc_call(GFP_NOIO);
323 if (!candidate)
324 return ERR_PTR(-EBUSY);
325 317
326 trace_rxrpc_call(candidate, rxrpc_call_new_service, 318 _enter(",%d", call->conn->debug_id);
327 atomic_read(&candidate->usage), here, NULL);
328 319
329 chan = sp->hdr.cid & RXRPC_CHANNELMASK; 320 rcu_assign_pointer(call->socket, rx);
330 candidate->conn = conn; 321 call->call_id = sp->hdr.callNumber;
331 candidate->peer = conn->params.peer; 322 call->service_id = sp->hdr.serviceId;
332 candidate->cid = sp->hdr.cid; 323 call->cid = sp->hdr.cid;
333 candidate->call_id = sp->hdr.callNumber; 324 call->state = RXRPC_CALL_SERVER_ACCEPTING;
334 candidate->security_ix = sp->hdr.securityIndex; 325 if (sp->hdr.securityIndex > 0)
335 candidate->rx_data_post = 0; 326 call->state = RXRPC_CALL_SERVER_SECURING;
336 candidate->state = RXRPC_CALL_SERVER_ACCEPTING; 327
337 candidate->flags |= (1 << RXRPC_CALL_IS_SERVICE); 328 /* Set the channel for this call. We don't get channel_lock as we're
338 if (conn->security_ix > 0) 329 * only defending against the data_ready handler (which we're called
339 candidate->state = RXRPC_CALL_SERVER_SECURING; 330 * from) and the RESPONSE packet parser (which is only really
340 rcu_assign_pointer(candidate->socket, rx); 331 * interested in call_counter and can cope with a disagreement with the
341 332 * call pointer).
342 spin_lock(&conn->channel_lock);
343
344 /* set the channel for this call */
345 call = rcu_dereference_protected(conn->channels[chan].call,
346 lockdep_is_held(&conn->channel_lock));
347
348 _debug("channel[%u] is %p", candidate->cid & RXRPC_CHANNELMASK, call);
349 if (call && call->call_id == sp->hdr.callNumber) {
350 /* already set; must've been a duplicate packet */
351 _debug("extant call [%d]", call->state);
352 ASSERTCMP(call->conn, ==, conn);
353
354 read_lock(&call->state_lock);
355 switch (call->state) {
356 case RXRPC_CALL_LOCALLY_ABORTED:
357 if (!test_and_set_bit(RXRPC_CALL_EV_ABORT, &call->events))
358 rxrpc_queue_call(call);
359 case RXRPC_CALL_REMOTELY_ABORTED:
360 read_unlock(&call->state_lock);
361 goto aborted_call;
362 default:
363 rxrpc_get_call(call, rxrpc_call_got);
364 read_unlock(&call->state_lock);
365 goto extant_call;
366 }
367 }
368
369 if (call) {
370 /* it seems the channel is still in use from the previous call
371 * - ditch the old binding if its call is now complete */
372 _debug("CALL: %u { %s }",
373 call->debug_id, rxrpc_call_states[call->state]);
374
375 if (call->state == RXRPC_CALL_COMPLETE) {
376 __rxrpc_disconnect_call(conn, call);
377 } else {
378 spin_unlock(&conn->channel_lock);
379 kmem_cache_free(rxrpc_call_jar, candidate);
380 _leave(" = -EBUSY");
381 return ERR_PTR(-EBUSY);
382 }
383 }
384
385 /* check the call number isn't duplicate */
386 _debug("check dup");
387 call_id = sp->hdr.callNumber;
388
389 /* We just ignore calls prior to the current call ID. Terminated calls
390 * are handled via the connection.
391 */ 333 */
392 if (call_id <= conn->channels[chan].call_counter) 334 chan = sp->hdr.cid & RXRPC_CHANNELMASK;
393 goto old_call; /* TODO: Just drop packet */ 335 conn->channels[chan].call_counter = call->call_id;
394 336 conn->channels[chan].call_id = call->call_id;
395 /* Temporary: Mirror the backlog prealloc ref (TODO: use prealloc) */
396 rxrpc_get_call(candidate, rxrpc_call_got);
397
398 /* make the call available */
399 _debug("new call");
400 call = candidate;
401 candidate = NULL;
402 conn->channels[chan].call_counter = call_id;
403 rcu_assign_pointer(conn->channels[chan].call, call); 337 rcu_assign_pointer(conn->channels[chan].call, call);
404 rxrpc_get_connection(conn);
405 rxrpc_get_peer(call->peer);
406 spin_unlock(&conn->channel_lock);
407 338
408 spin_lock(&conn->params.peer->lock); 339 spin_lock(&conn->params.peer->lock);
409 hlist_add_head(&call->error_link, &conn->params.peer->error_targets); 340 hlist_add_head(&call->error_link, &conn->params.peer->error_targets);
410 spin_unlock(&conn->params.peer->lock); 341 spin_unlock(&conn->params.peer->lock);
411 342
412 write_lock_bh(&rxrpc_call_lock);
413 list_add_tail(&call->link, &rxrpc_calls);
414 write_unlock_bh(&rxrpc_call_lock);
415
416 call->service_id = conn->params.service_id;
417
418 _net("CALL incoming %d on CONN %d", call->debug_id, call->conn->debug_id); 343 _net("CALL incoming %d on CONN %d", call->debug_id, call->conn->debug_id);
419 344
420 call->lifetimer.expires = jiffies + rxrpc_max_call_lifetime; 345 rxrpc_start_call_timer(call);
421 add_timer(&call->lifetimer); 346 _leave("");
422 _leave(" = %p {%d} [new]", call, call->debug_id);
423 return call;
424
425extant_call:
426 spin_unlock(&conn->channel_lock);
427 kmem_cache_free(rxrpc_call_jar, candidate);
428 _leave(" = %p {%d} [extant]", call, call ? call->debug_id : -1);
429 return call;
430
431aborted_call:
432 spin_unlock(&conn->channel_lock);
433 kmem_cache_free(rxrpc_call_jar, candidate);
434 _leave(" = -ECONNABORTED");
435 return ERR_PTR(-ECONNABORTED);
436
437old_call:
438 spin_unlock(&conn->channel_lock);
439 kmem_cache_free(rxrpc_call_jar, candidate);
440 _leave(" = -ECONNRESET [old]");
441 return ERR_PTR(-ECONNRESET);
442} 347}
443 348
444/* 349/*
@@ -497,25 +402,17 @@ void rxrpc_get_call(struct rxrpc_call *call, enum rxrpc_call_trace op)
497} 402}
498 403
499/* 404/*
500 * Note the addition of a ref on a call for a socket buffer. 405 * Detach a call from its owning socket.
501 */ 406 */
502void rxrpc_get_call_for_skb(struct rxrpc_call *call, struct sk_buff *skb) 407void rxrpc_release_call(struct rxrpc_sock *rx, struct rxrpc_call *call)
503{ 408{
504 const void *here = __builtin_return_address(0); 409 struct rxrpc_connection *conn = call->conn;
505 int n = atomic_inc_return(&call->usage); 410 bool put = false;
411 int i;
506 412
507 trace_rxrpc_call(call, rxrpc_call_got_skb, n, here, skb); 413 _enter("{%d,%d}", call->debug_id, atomic_read(&call->usage));
508}
509 414
510/* 415 ASSERTCMP(call->state, ==, RXRPC_CALL_COMPLETE);
511 * detach a call from a socket and set up for release
512 */
513void rxrpc_release_call(struct rxrpc_sock *rx, struct rxrpc_call *call)
514{
515 _enter("{%d,%d,%d,%d}",
516 call->debug_id, atomic_read(&call->usage),
517 atomic_read(&call->ackr_not_idle),
518 call->rx_first_oos);
519 416
520 rxrpc_see_call(call); 417 rxrpc_see_call(call);
521 418
@@ -524,80 +421,46 @@ void rxrpc_release_call(struct rxrpc_sock *rx, struct rxrpc_call *call)
524 BUG(); 421 BUG();
525 spin_unlock_bh(&call->lock); 422 spin_unlock_bh(&call->lock);
526 423
527 /* dissociate from the socket 424 del_timer_sync(&call->timer);
528 * - the socket's ref on the call is passed to the death timer
529 */
530 _debug("RELEASE CALL %p (%d)", call, call->debug_id);
531 425
532 if (call->peer) { 426 /* Make sure we don't get any more notifications */
533 spin_lock(&call->peer->lock); 427 write_lock_bh(&rx->recvmsg_lock);
534 hlist_del_init(&call->error_link);
535 spin_unlock(&call->peer->lock);
536 }
537 428
538 write_lock_bh(&rx->call_lock); 429 if (!list_empty(&call->recvmsg_link)) {
539 if (!list_empty(&call->accept_link)) {
540 _debug("unlinking once-pending call %p { e=%lx f=%lx }", 430 _debug("unlinking once-pending call %p { e=%lx f=%lx }",
541 call, call->events, call->flags); 431 call, call->events, call->flags);
542 ASSERT(!test_bit(RXRPC_CALL_HAS_USERID, &call->flags)); 432 list_del(&call->recvmsg_link);
543 list_del_init(&call->accept_link); 433 put = true;
544 sk_acceptq_removed(&rx->sk); 434 }
545 } else if (test_bit(RXRPC_CALL_HAS_USERID, &call->flags)) { 435
436 /* list_empty() must return false in rxrpc_notify_socket() */
437 call->recvmsg_link.next = NULL;
438 call->recvmsg_link.prev = NULL;
439
440 write_unlock_bh(&rx->recvmsg_lock);
441 if (put)
442 rxrpc_put_call(call, rxrpc_call_put);
443
444 write_lock(&rx->call_lock);
445
446 if (test_and_clear_bit(RXRPC_CALL_HAS_USERID, &call->flags)) {
546 rb_erase(&call->sock_node, &rx->calls); 447 rb_erase(&call->sock_node, &rx->calls);
547 memset(&call->sock_node, 0xdd, sizeof(call->sock_node)); 448 memset(&call->sock_node, 0xdd, sizeof(call->sock_node));
548 clear_bit(RXRPC_CALL_HAS_USERID, &call->flags);
549 rxrpc_put_call(call, rxrpc_call_put_userid); 449 rxrpc_put_call(call, rxrpc_call_put_userid);
550 } 450 }
551 write_unlock_bh(&rx->call_lock);
552
553 /* free up the channel for reuse */
554 if (call->state == RXRPC_CALL_CLIENT_FINAL_ACK) {
555 clear_bit(RXRPC_CALL_EV_ACK_FINAL, &call->events);
556 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ACK);
557 rxrpc_call_completed(call);
558 } else {
559 write_lock_bh(&call->state_lock);
560
561 if (call->state < RXRPC_CALL_COMPLETE) {
562 _debug("+++ ABORTING STATE %d +++\n", call->state);
563 __rxrpc_abort_call("SKT", call, 0, RX_CALL_DEAD, ECONNRESET);
564 clear_bit(RXRPC_CALL_EV_ACK_FINAL, &call->events);
565 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ABORT);
566 }
567
568 write_unlock_bh(&call->state_lock);
569 }
570 451
571 if (call->conn) 452 list_del(&call->sock_link);
453 write_unlock(&rx->call_lock);
454
455 _debug("RELEASE CALL %p (%d CONN %p)", call, call->debug_id, conn);
456
457 if (conn)
572 rxrpc_disconnect_call(call); 458 rxrpc_disconnect_call(call);
573 459
574 /* clean up the Rx queue */ 460 for (i = 0; i < RXRPC_RXTX_BUFF_SIZE; i++) {
575 if (!skb_queue_empty(&call->rx_queue) || 461 rxrpc_free_skb(call->rxtx_buffer[i]);
576 !skb_queue_empty(&call->rx_oos_queue)) { 462 call->rxtx_buffer[i] = NULL;
577 struct rxrpc_skb_priv *sp;
578 struct sk_buff *skb;
579
580 _debug("purge Rx queues");
581
582 spin_lock_bh(&call->lock);
583 while ((skb = skb_dequeue(&call->rx_queue)) ||
584 (skb = skb_dequeue(&call->rx_oos_queue))) {
585 spin_unlock_bh(&call->lock);
586
587 sp = rxrpc_skb(skb);
588 _debug("- zap %s %%%u #%u",
589 rxrpc_pkts[sp->hdr.type],
590 sp->hdr.serial, sp->hdr.seq);
591 rxrpc_free_skb(skb);
592 spin_lock_bh(&call->lock);
593 }
594 spin_unlock_bh(&call->lock);
595 } 463 }
596 rxrpc_purge_queue(&call->knlrecv_queue);
597
598 del_timer_sync(&call->resend_timer);
599 del_timer_sync(&call->ack_timer);
600 del_timer_sync(&call->lifetimer);
601 464
602 /* We have to release the prealloc backlog ref */ 465 /* We have to release the prealloc backlog ref */
603 if (rxrpc_is_service_call(call)) 466 if (rxrpc_is_service_call(call))
@@ -611,28 +474,19 @@ void rxrpc_release_call(struct rxrpc_sock *rx, struct rxrpc_call *call)
611void rxrpc_release_calls_on_socket(struct rxrpc_sock *rx) 474void rxrpc_release_calls_on_socket(struct rxrpc_sock *rx)
612{ 475{
613 struct rxrpc_call *call; 476 struct rxrpc_call *call;
614 struct rb_node *p;
615 477
616 _enter("%p", rx); 478 _enter("%p", rx);
617 479
618 read_lock_bh(&rx->call_lock); 480 while (!list_empty(&rx->sock_calls)) {
619 481 call = list_entry(rx->sock_calls.next,
620 /* kill the not-yet-accepted incoming calls */ 482 struct rxrpc_call, sock_link);
621 list_for_each_entry(call, &rx->secureq, accept_link) { 483 rxrpc_get_call(call, rxrpc_call_got);
622 rxrpc_release_call(rx, call); 484 rxrpc_abort_call("SKT", call, 0, RX_CALL_DEAD, ECONNRESET);
623 } 485 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ABORT);
624
625 list_for_each_entry(call, &rx->acceptq, accept_link) {
626 rxrpc_release_call(rx, call);
627 }
628
629 /* mark all the calls as no longer wanting incoming packets */
630 for (p = rb_first(&rx->calls); p; p = rb_next(p)) {
631 call = rb_entry(p, struct rxrpc_call, sock_node);
632 rxrpc_release_call(rx, call); 486 rxrpc_release_call(rx, call);
487 rxrpc_put_call(call, rxrpc_call_put);
633 } 488 }
634 489
635 read_unlock_bh(&rx->call_lock);
636 _leave(""); 490 _leave("");
637} 491}
638 492
@@ -651,23 +505,12 @@ void rxrpc_put_call(struct rxrpc_call *call, enum rxrpc_call_trace op)
651 ASSERTCMP(n, >=, 0); 505 ASSERTCMP(n, >=, 0);
652 if (n == 0) { 506 if (n == 0) {
653 _debug("call %d dead", call->debug_id); 507 _debug("call %d dead", call->debug_id);
654 rxrpc_cleanup_call(call); 508 ASSERTCMP(call->state, ==, RXRPC_CALL_COMPLETE);
655 }
656}
657 509
658/* 510 write_lock(&rxrpc_call_lock);
659 * Release a call ref held by a socket buffer. 511 list_del_init(&call->link);
660 */ 512 write_unlock(&rxrpc_call_lock);
661void rxrpc_put_call_for_skb(struct rxrpc_call *call, struct sk_buff *skb)
662{
663 const void *here = __builtin_return_address(0);
664 int n;
665 513
666 n = atomic_dec_return(&call->usage);
667 trace_rxrpc_call(call, rxrpc_call_put_skb, n, here, skb);
668 ASSERTCMP(n, >=, 0);
669 if (n == 0) {
670 _debug("call %d dead", call->debug_id);
671 rxrpc_cleanup_call(call); 514 rxrpc_cleanup_call(call);
672 } 515 }
673} 516}
@@ -679,9 +522,9 @@ static void rxrpc_rcu_destroy_call(struct rcu_head *rcu)
679{ 522{
680 struct rxrpc_call *call = container_of(rcu, struct rxrpc_call, rcu); 523 struct rxrpc_call *call = container_of(rcu, struct rxrpc_call, rcu);
681 524
682 rxrpc_purge_queue(&call->rx_queue);
683 rxrpc_purge_queue(&call->knlrecv_queue);
684 rxrpc_put_peer(call->peer); 525 rxrpc_put_peer(call->peer);
526 kfree(call->rxtx_buffer);
527 kfree(call->rxtx_annotations);
685 kmem_cache_free(rxrpc_call_jar, call); 528 kmem_cache_free(rxrpc_call_jar, call);
686} 529}
687 530
@@ -690,49 +533,24 @@ static void rxrpc_rcu_destroy_call(struct rcu_head *rcu)
690 */ 533 */
691void rxrpc_cleanup_call(struct rxrpc_call *call) 534void rxrpc_cleanup_call(struct rxrpc_call *call)
692{ 535{
693 _net("DESTROY CALL %d", call->debug_id); 536 int i;
694 537
695 write_lock_bh(&rxrpc_call_lock); 538 _net("DESTROY CALL %d", call->debug_id);
696 list_del_init(&call->link);
697 write_unlock_bh(&rxrpc_call_lock);
698 539
699 memset(&call->sock_node, 0xcd, sizeof(call->sock_node)); 540 memset(&call->sock_node, 0xcd, sizeof(call->sock_node));
700 541
701 del_timer_sync(&call->lifetimer); 542 del_timer_sync(&call->timer);
702 del_timer_sync(&call->ack_timer);
703 del_timer_sync(&call->resend_timer);
704 543
705 ASSERTCMP(call->state, ==, RXRPC_CALL_COMPLETE); 544 ASSERTCMP(call->state, ==, RXRPC_CALL_COMPLETE);
706 ASSERT(test_bit(RXRPC_CALL_RELEASED, &call->flags)); 545 ASSERT(test_bit(RXRPC_CALL_RELEASED, &call->flags));
707 ASSERT(!work_pending(&call->processor));
708 ASSERTCMP(call->conn, ==, NULL); 546 ASSERTCMP(call->conn, ==, NULL);
709 547
710 if (call->acks_window) { 548 /* Clean up the Rx/Tx buffer */
711 _debug("kill Tx window %d", 549 for (i = 0; i < RXRPC_RXTX_BUFF_SIZE; i++)
712 CIRC_CNT(call->acks_head, call->acks_tail, 550 rxrpc_free_skb(call->rxtx_buffer[i]);
713 call->acks_winsz));
714 smp_mb();
715 while (CIRC_CNT(call->acks_head, call->acks_tail,
716 call->acks_winsz) > 0) {
717 struct rxrpc_skb_priv *sp;
718 unsigned long _skb;
719
720 _skb = call->acks_window[call->acks_tail] & ~1;
721 sp = rxrpc_skb((struct sk_buff *)_skb);
722 _debug("+++ clear Tx %u", sp->hdr.seq);
723 rxrpc_free_skb((struct sk_buff *)_skb);
724 call->acks_tail =
725 (call->acks_tail + 1) & (call->acks_winsz - 1);
726 }
727
728 kfree(call->acks_window);
729 }
730 551
731 rxrpc_free_skb(call->tx_pending); 552 rxrpc_free_skb(call->tx_pending);
732 553
733 rxrpc_purge_queue(&call->rx_queue);
734 ASSERT(skb_queue_empty(&call->rx_oos_queue));
735 rxrpc_purge_queue(&call->knlrecv_queue);
736 call_rcu(&call->rcu, rxrpc_rcu_destroy_call); 554 call_rcu(&call->rcu, rxrpc_rcu_destroy_call);
737} 555}
738 556
@@ -747,8 +565,8 @@ void __exit rxrpc_destroy_all_calls(void)
747 565
748 if (list_empty(&rxrpc_calls)) 566 if (list_empty(&rxrpc_calls))
749 return; 567 return;
750 568
751 write_lock_bh(&rxrpc_call_lock); 569 write_lock(&rxrpc_call_lock);
752 570
753 while (!list_empty(&rxrpc_calls)) { 571 while (!list_empty(&rxrpc_calls)) {
754 call = list_entry(rxrpc_calls.next, struct rxrpc_call, link); 572 call = list_entry(rxrpc_calls.next, struct rxrpc_call, link);
@@ -757,74 +575,15 @@ void __exit rxrpc_destroy_all_calls(void)
757 rxrpc_see_call(call); 575 rxrpc_see_call(call);
758 list_del_init(&call->link); 576 list_del_init(&call->link);
759 577
760 pr_err("Call %p still in use (%d,%d,%s,%lx,%lx)!\n", 578 pr_err("Call %p still in use (%d,%s,%lx,%lx)!\n",
761 call, atomic_read(&call->usage), 579 call, atomic_read(&call->usage),
762 atomic_read(&call->ackr_not_idle),
763 rxrpc_call_states[call->state], 580 rxrpc_call_states[call->state],
764 call->flags, call->events); 581 call->flags, call->events);
765 if (!skb_queue_empty(&call->rx_queue))
766 pr_err("Rx queue occupied\n");
767 if (!skb_queue_empty(&call->rx_oos_queue))
768 pr_err("OOS queue occupied\n");
769 582
770 write_unlock_bh(&rxrpc_call_lock); 583 write_unlock(&rxrpc_call_lock);
771 cond_resched(); 584 cond_resched();
772 write_lock_bh(&rxrpc_call_lock); 585 write_lock(&rxrpc_call_lock);
773 } 586 }
774 587
775 write_unlock_bh(&rxrpc_call_lock); 588 write_unlock(&rxrpc_call_lock);
776 _leave("");
777}
778
779/*
780 * handle call lifetime being exceeded
781 */
782static void rxrpc_call_life_expired(unsigned long _call)
783{
784 struct rxrpc_call *call = (struct rxrpc_call *) _call;
785
786 _enter("{%d}", call->debug_id);
787
788 rxrpc_see_call(call);
789 if (call->state >= RXRPC_CALL_COMPLETE)
790 return;
791
792 set_bit(RXRPC_CALL_EV_LIFE_TIMER, &call->events);
793 rxrpc_queue_call(call);
794}
795
796/*
797 * handle resend timer expiry
798 * - may not take call->state_lock as this can deadlock against del_timer_sync()
799 */
800static void rxrpc_resend_time_expired(unsigned long _call)
801{
802 struct rxrpc_call *call = (struct rxrpc_call *) _call;
803
804 _enter("{%d}", call->debug_id);
805
806 rxrpc_see_call(call);
807 if (call->state >= RXRPC_CALL_COMPLETE)
808 return;
809
810 clear_bit(RXRPC_CALL_RUN_RTIMER, &call->flags);
811 if (!test_and_set_bit(RXRPC_CALL_EV_RESEND_TIMER, &call->events))
812 rxrpc_queue_call(call);
813}
814
815/*
816 * handle ACK timer expiry
817 */
818static void rxrpc_ack_time_expired(unsigned long _call)
819{
820 struct rxrpc_call *call = (struct rxrpc_call *) _call;
821
822 _enter("{%d}", call->debug_id);
823
824 rxrpc_see_call(call);
825 if (call->state >= RXRPC_CALL_COMPLETE)
826 return;
827
828 if (!test_and_set_bit(RXRPC_CALL_EV_ACK, &call->events))
829 rxrpc_queue_call(call);
830} 589}
diff --git a/net/rxrpc/conn_event.c b/net/rxrpc/conn_event.c
index 8c7938ba6a84..0691007cfc02 100644
--- a/net/rxrpc/conn_event.c
+++ b/net/rxrpc/conn_event.c
@@ -15,10 +15,6 @@
15#include <linux/net.h> 15#include <linux/net.h>
16#include <linux/skbuff.h> 16#include <linux/skbuff.h>
17#include <linux/errqueue.h> 17#include <linux/errqueue.h>
18#include <linux/udp.h>
19#include <linux/in.h>
20#include <linux/in6.h>
21#include <linux/icmp.h>
22#include <net/sock.h> 18#include <net/sock.h>
23#include <net/af_rxrpc.h> 19#include <net/af_rxrpc.h>
24#include <net/ip.h> 20#include <net/ip.h>
@@ -140,16 +136,10 @@ static void rxrpc_abort_calls(struct rxrpc_connection *conn,
140 u32 abort_code, int error) 136 u32 abort_code, int error)
141{ 137{
142 struct rxrpc_call *call; 138 struct rxrpc_call *call;
143 bool queue; 139 int i;
144 int i, bit;
145 140
146 _enter("{%d},%x", conn->debug_id, abort_code); 141 _enter("{%d},%x", conn->debug_id, abort_code);
147 142
148 if (compl == RXRPC_CALL_LOCALLY_ABORTED)
149 bit = RXRPC_CALL_EV_CONN_ABORT;
150 else
151 bit = RXRPC_CALL_EV_RCVD_ABORT;
152
153 spin_lock(&conn->channel_lock); 143 spin_lock(&conn->channel_lock);
154 144
155 for (i = 0; i < RXRPC_MAXCALLS; i++) { 145 for (i = 0; i < RXRPC_MAXCALLS; i++) {
@@ -157,22 +147,13 @@ static void rxrpc_abort_calls(struct rxrpc_connection *conn,
157 conn->channels[i].call, 147 conn->channels[i].call,
158 lockdep_is_held(&conn->channel_lock)); 148 lockdep_is_held(&conn->channel_lock));
159 if (call) { 149 if (call) {
160 rxrpc_see_call(call);
161 if (compl == RXRPC_CALL_LOCALLY_ABORTED) 150 if (compl == RXRPC_CALL_LOCALLY_ABORTED)
162 trace_rxrpc_abort("CON", call->cid, 151 trace_rxrpc_abort("CON", call->cid,
163 call->call_id, 0, 152 call->call_id, 0,
164 abort_code, error); 153 abort_code, error);
165 154 if (rxrpc_set_call_completion(call, compl,
166 write_lock_bh(&call->state_lock); 155 abort_code, error))
167 if (rxrpc_set_call_completion(call, compl, abort_code, 156 rxrpc_notify_socket(call);
168 error)) {
169 set_bit(bit, &call->events);
170 queue = true;
171 }
172 write_unlock_bh(&call->state_lock);
173 if (queue)
174 rxrpc_queue_call(call);
175
176 } 157 }
177 } 158 }
178 159
@@ -251,17 +232,18 @@ static int rxrpc_abort_connection(struct rxrpc_connection *conn,
251 232
252/* 233/*
253 * mark a call as being on a now-secured channel 234 * mark a call as being on a now-secured channel
254 * - must be called with softirqs disabled 235 * - must be called with BH's disabled.
255 */ 236 */
256static void rxrpc_call_is_secure(struct rxrpc_call *call) 237static void rxrpc_call_is_secure(struct rxrpc_call *call)
257{ 238{
258 _enter("%p", call); 239 _enter("%p", call);
259 if (call) { 240 if (call) {
260 read_lock(&call->state_lock); 241 write_lock_bh(&call->state_lock);
261 if (call->state < RXRPC_CALL_COMPLETE && 242 if (call->state == RXRPC_CALL_SERVER_SECURING) {
262 !test_and_set_bit(RXRPC_CALL_EV_SECURED, &call->events)) 243 call->state = RXRPC_CALL_SERVER_ACCEPTING;
263 rxrpc_queue_call(call); 244 rxrpc_notify_socket(call);
264 read_unlock(&call->state_lock); 245 }
246 write_unlock_bh(&call->state_lock);
265 } 247 }
266} 248}
267 249
@@ -278,7 +260,7 @@ static int rxrpc_process_event(struct rxrpc_connection *conn,
278 int loop, ret; 260 int loop, ret;
279 261
280 if (conn->state >= RXRPC_CONN_REMOTELY_ABORTED) { 262 if (conn->state >= RXRPC_CONN_REMOTELY_ABORTED) {
281 kleave(" = -ECONNABORTED [%u]", conn->state); 263 _leave(" = -ECONNABORTED [%u]", conn->state);
282 return -ECONNABORTED; 264 return -ECONNABORTED;
283 } 265 }
284 266
@@ -291,14 +273,14 @@ static int rxrpc_process_event(struct rxrpc_connection *conn,
291 return 0; 273 return 0;
292 274
293 case RXRPC_PACKET_TYPE_ABORT: 275 case RXRPC_PACKET_TYPE_ABORT:
294 if (skb_copy_bits(skb, 0, &wtmp, sizeof(wtmp)) < 0) 276 if (skb_copy_bits(skb, sp->offset, &wtmp, sizeof(wtmp)) < 0)
295 return -EPROTO; 277 return -EPROTO;
296 abort_code = ntohl(wtmp); 278 abort_code = ntohl(wtmp);
297 _proto("Rx ABORT %%%u { ac=%d }", sp->hdr.serial, abort_code); 279 _proto("Rx ABORT %%%u { ac=%d }", sp->hdr.serial, abort_code);
298 280
299 conn->state = RXRPC_CONN_REMOTELY_ABORTED; 281 conn->state = RXRPC_CONN_REMOTELY_ABORTED;
300 rxrpc_abort_calls(conn, 0, RXRPC_CALL_REMOTELY_ABORTED, 282 rxrpc_abort_calls(conn, RXRPC_CALL_REMOTELY_ABORTED,
301 abort_code); 283 abort_code, ECONNABORTED);
302 return -ECONNABORTED; 284 return -ECONNABORTED;
303 285
304 case RXRPC_PACKET_TYPE_CHALLENGE: 286 case RXRPC_PACKET_TYPE_CHALLENGE:
@@ -323,14 +305,16 @@ static int rxrpc_process_event(struct rxrpc_connection *conn,
323 305
324 if (conn->state == RXRPC_CONN_SERVICE_CHALLENGING) { 306 if (conn->state == RXRPC_CONN_SERVICE_CHALLENGING) {
325 conn->state = RXRPC_CONN_SERVICE; 307 conn->state = RXRPC_CONN_SERVICE;
308 spin_unlock(&conn->state_lock);
326 for (loop = 0; loop < RXRPC_MAXCALLS; loop++) 309 for (loop = 0; loop < RXRPC_MAXCALLS; loop++)
327 rxrpc_call_is_secure( 310 rxrpc_call_is_secure(
328 rcu_dereference_protected( 311 rcu_dereference_protected(
329 conn->channels[loop].call, 312 conn->channels[loop].call,
330 lockdep_is_held(&conn->channel_lock))); 313 lockdep_is_held(&conn->channel_lock)));
314 } else {
315 spin_unlock(&conn->state_lock);
331 } 316 }
332 317
333 spin_unlock(&conn->state_lock);
334 spin_unlock(&conn->channel_lock); 318 spin_unlock(&conn->channel_lock);
335 return 0; 319 return 0;
336 320
@@ -433,88 +417,3 @@ protocol_error:
433 _leave(" [EPROTO]"); 417 _leave(" [EPROTO]");
434 goto out; 418 goto out;
435} 419}
436
437/*
438 * put a packet up for transport-level abort
439 */
440void rxrpc_reject_packet(struct rxrpc_local *local, struct sk_buff *skb)
441{
442 CHECK_SLAB_OKAY(&local->usage);
443
444 skb_queue_tail(&local->reject_queue, skb);
445 rxrpc_queue_local(local);
446}
447
448/*
449 * reject packets through the local endpoint
450 */
451void rxrpc_reject_packets(struct rxrpc_local *local)
452{
453 union {
454 struct sockaddr sa;
455 struct sockaddr_in sin;
456 } sa;
457 struct rxrpc_skb_priv *sp;
458 struct rxrpc_wire_header whdr;
459 struct sk_buff *skb;
460 struct msghdr msg;
461 struct kvec iov[2];
462 size_t size;
463 __be32 code;
464
465 _enter("%d", local->debug_id);
466
467 iov[0].iov_base = &whdr;
468 iov[0].iov_len = sizeof(whdr);
469 iov[1].iov_base = &code;
470 iov[1].iov_len = sizeof(code);
471 size = sizeof(whdr) + sizeof(code);
472
473 msg.msg_name = &sa;
474 msg.msg_control = NULL;
475 msg.msg_controllen = 0;
476 msg.msg_flags = 0;
477
478 memset(&sa, 0, sizeof(sa));
479 sa.sa.sa_family = local->srx.transport.family;
480 switch (sa.sa.sa_family) {
481 case AF_INET:
482 msg.msg_namelen = sizeof(sa.sin);
483 break;
484 default:
485 msg.msg_namelen = 0;
486 break;
487 }
488
489 memset(&whdr, 0, sizeof(whdr));
490 whdr.type = RXRPC_PACKET_TYPE_ABORT;
491
492 while ((skb = skb_dequeue(&local->reject_queue))) {
493 rxrpc_see_skb(skb);
494 sp = rxrpc_skb(skb);
495 switch (sa.sa.sa_family) {
496 case AF_INET:
497 sa.sin.sin_port = udp_hdr(skb)->source;
498 sa.sin.sin_addr.s_addr = ip_hdr(skb)->saddr;
499 code = htonl(skb->priority);
500
501 whdr.epoch = htonl(sp->hdr.epoch);
502 whdr.cid = htonl(sp->hdr.cid);
503 whdr.callNumber = htonl(sp->hdr.callNumber);
504 whdr.serviceId = htons(sp->hdr.serviceId);
505 whdr.flags = sp->hdr.flags;
506 whdr.flags ^= RXRPC_CLIENT_INITIATED;
507 whdr.flags &= RXRPC_CLIENT_INITIATED;
508
509 kernel_sendmsg(local->socket, &msg, iov, 2, size);
510 break;
511
512 default:
513 break;
514 }
515
516 rxrpc_free_skb(skb);
517 }
518
519 _leave("");
520}
diff --git a/net/rxrpc/conn_object.c b/net/rxrpc/conn_object.c
index 8da82e3aa00e..ffa9addb97b2 100644
--- a/net/rxrpc/conn_object.c
+++ b/net/rxrpc/conn_object.c
@@ -169,7 +169,7 @@ void __rxrpc_disconnect_call(struct rxrpc_connection *conn,
169 chan->last_abort = call->abort_code; 169 chan->last_abort = call->abort_code;
170 chan->last_type = RXRPC_PACKET_TYPE_ABORT; 170 chan->last_type = RXRPC_PACKET_TYPE_ABORT;
171 } else { 171 } else {
172 chan->last_seq = call->rx_data_eaten; 172 chan->last_seq = call->rx_hard_ack;
173 chan->last_type = RXRPC_PACKET_TYPE_ACK; 173 chan->last_type = RXRPC_PACKET_TYPE_ACK;
174 } 174 }
175 /* Sync with rxrpc_conn_retransmit(). */ 175 /* Sync with rxrpc_conn_retransmit(). */
@@ -191,6 +191,10 @@ void rxrpc_disconnect_call(struct rxrpc_call *call)
191{ 191{
192 struct rxrpc_connection *conn = call->conn; 192 struct rxrpc_connection *conn = call->conn;
193 193
194 spin_lock_bh(&conn->params.peer->lock);
195 hlist_del_init(&call->error_link);
196 spin_unlock_bh(&conn->params.peer->lock);
197
194 if (rxrpc_is_client_call(call)) 198 if (rxrpc_is_client_call(call))
195 return rxrpc_disconnect_client_call(call); 199 return rxrpc_disconnect_client_call(call);
196 200
diff --git a/net/rxrpc/conn_service.c b/net/rxrpc/conn_service.c
index 189338a60457..83d54da4ce8b 100644
--- a/net/rxrpc/conn_service.c
+++ b/net/rxrpc/conn_service.c
@@ -65,9 +65,8 @@ done:
65 * Insert a service connection into a peer's tree, thereby making it a target 65 * Insert a service connection into a peer's tree, thereby making it a target
66 * for incoming packets. 66 * for incoming packets.
67 */ 67 */
68static struct rxrpc_connection * 68static void rxrpc_publish_service_conn(struct rxrpc_peer *peer,
69rxrpc_publish_service_conn(struct rxrpc_peer *peer, 69 struct rxrpc_connection *conn)
70 struct rxrpc_connection *conn)
71{ 70{
72 struct rxrpc_connection *cursor = NULL; 71 struct rxrpc_connection *cursor = NULL;
73 struct rxrpc_conn_proto k = conn->proto; 72 struct rxrpc_conn_proto k = conn->proto;
@@ -96,7 +95,7 @@ conn_published:
96 set_bit(RXRPC_CONN_IN_SERVICE_CONNS, &conn->flags); 95 set_bit(RXRPC_CONN_IN_SERVICE_CONNS, &conn->flags);
97 write_sequnlock_bh(&peer->service_conn_lock); 96 write_sequnlock_bh(&peer->service_conn_lock);
98 _leave(" = %d [new]", conn->debug_id); 97 _leave(" = %d [new]", conn->debug_id);
99 return conn; 98 return;
100 99
101found_extant_conn: 100found_extant_conn:
102 if (atomic_read(&cursor->usage) == 0) 101 if (atomic_read(&cursor->usage) == 0)
@@ -143,106 +142,30 @@ struct rxrpc_connection *rxrpc_prealloc_service_connection(gfp_t gfp)
143} 142}
144 143
145/* 144/*
146 * get a record of an incoming connection 145 * Set up an incoming connection. This is called in BH context with the RCU
146 * read lock held.
147 */ 147 */
148struct rxrpc_connection *rxrpc_incoming_connection(struct rxrpc_local *local, 148void rxrpc_new_incoming_connection(struct rxrpc_connection *conn,
149 struct sockaddr_rxrpc *srx, 149 struct sk_buff *skb)
150 struct sk_buff *skb)
151{ 150{
152 struct rxrpc_connection *conn;
153 struct rxrpc_skb_priv *sp = rxrpc_skb(skb); 151 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
154 struct rxrpc_peer *peer;
155 const char *new = "old";
156 152
157 _enter(""); 153 _enter("");
158 154
159 peer = rxrpc_lookup_peer(local, srx, GFP_NOIO);
160 if (!peer) {
161 _debug("no peer");
162 return ERR_PTR(-EBUSY);
163 }
164
165 ASSERT(sp->hdr.flags & RXRPC_CLIENT_INITIATED);
166
167 rcu_read_lock();
168 peer = rxrpc_lookup_peer_rcu(local, srx);
169 if (peer) {
170 conn = rxrpc_find_service_conn_rcu(peer, skb);
171 if (conn) {
172 if (sp->hdr.securityIndex != conn->security_ix)
173 goto security_mismatch_rcu;
174 if (rxrpc_get_connection_maybe(conn))
175 goto found_extant_connection_rcu;
176
177 /* The conn has expired but we can't remove it without
178 * the appropriate lock, so we attempt to replace it
179 * when we have a new candidate.
180 */
181 }
182
183 if (!rxrpc_get_peer_maybe(peer))
184 peer = NULL;
185 }
186 rcu_read_unlock();
187
188 if (!peer) {
189 peer = rxrpc_lookup_peer(local, srx, GFP_NOIO);
190 if (!peer)
191 goto enomem;
192 }
193
194 /* We don't have a matching record yet. */
195 conn = rxrpc_alloc_connection(GFP_NOIO);
196 if (!conn)
197 goto enomem_peer;
198
199 conn->proto.epoch = sp->hdr.epoch; 155 conn->proto.epoch = sp->hdr.epoch;
200 conn->proto.cid = sp->hdr.cid & RXRPC_CIDMASK; 156 conn->proto.cid = sp->hdr.cid & RXRPC_CIDMASK;
201 conn->params.local = local;
202 conn->params.peer = peer;
203 conn->params.service_id = sp->hdr.serviceId; 157 conn->params.service_id = sp->hdr.serviceId;
204 conn->security_ix = sp->hdr.securityIndex; 158 conn->security_ix = sp->hdr.securityIndex;
205 conn->out_clientflag = 0; 159 conn->out_clientflag = 0;
206 conn->state = RXRPC_CONN_SERVICE; 160 if (conn->security_ix)
207 if (conn->params.service_id)
208 conn->state = RXRPC_CONN_SERVICE_UNSECURED; 161 conn->state = RXRPC_CONN_SERVICE_UNSECURED;
209 162 else
210 rxrpc_get_local(local); 163 conn->state = RXRPC_CONN_SERVICE;
211
212 /* We maintain an extra ref on the connection whilst it is on
213 * the rxrpc_connections list.
214 */
215 atomic_set(&conn->usage, 2);
216
217 write_lock(&rxrpc_connection_lock);
218 list_add_tail(&conn->link, &rxrpc_connections);
219 list_add_tail(&conn->proc_link, &rxrpc_connection_proc_list);
220 write_unlock(&rxrpc_connection_lock);
221 164
222 /* Make the connection a target for incoming packets. */ 165 /* Make the connection a target for incoming packets. */
223 rxrpc_publish_service_conn(peer, conn); 166 rxrpc_publish_service_conn(conn->params.peer, conn);
224
225 new = "new";
226
227success:
228 _net("CONNECTION %s %d {%x}", new, conn->debug_id, conn->proto.cid);
229 _leave(" = %p {u=%d}", conn, atomic_read(&conn->usage));
230 return conn;
231
232found_extant_connection_rcu:
233 rcu_read_unlock();
234 goto success;
235
236security_mismatch_rcu:
237 rcu_read_unlock();
238 _leave(" = -EKEYREJECTED");
239 return ERR_PTR(-EKEYREJECTED);
240 167
241enomem_peer: 168 _net("CONNECTION new %d {%x}", conn->debug_id, conn->proto.cid);
242 rxrpc_put_peer(peer);
243enomem:
244 _leave(" = -ENOMEM");
245 return ERR_PTR(-ENOMEM);
246} 169}
247 170
248/* 171/*
diff --git a/net/rxrpc/input.c b/net/rxrpc/input.c
index 5906579060cd..afeba98004b1 100644
--- a/net/rxrpc/input.c
+++ b/net/rxrpc/input.c
@@ -1,6 +1,6 @@
1/* RxRPC packet reception 1/* RxRPC packet reception
2 * 2 *
3 * Copyright (C) 2007 Red Hat, Inc. All Rights Reserved. 3 * Copyright (C) 2007, 2016 Red Hat, Inc. All Rights Reserved.
4 * Written by David Howells (dhowells@redhat.com) 4 * Written by David Howells (dhowells@redhat.com)
5 * 5 *
6 * This program is free software; you can redistribute it and/or 6 * This program is free software; you can redistribute it and/or
@@ -27,549 +27,547 @@
27#include <net/net_namespace.h> 27#include <net/net_namespace.h>
28#include "ar-internal.h" 28#include "ar-internal.h"
29 29
30static void rxrpc_proto_abort(const char *why,
31 struct rxrpc_call *call, rxrpc_seq_t seq)
32{
33 if (rxrpc_abort_call(why, call, seq, RX_PROTOCOL_ERROR, EBADMSG)) {
34 set_bit(RXRPC_CALL_EV_ABORT, &call->events);
35 rxrpc_queue_call(call);
36 }
37}
38
30/* 39/*
31 * queue a packet for recvmsg to pass to userspace 40 * Apply a hard ACK by advancing the Tx window.
32 * - the caller must hold a lock on call->lock
33 * - must not be called with interrupts disabled (sk_filter() disables BH's)
34 * - eats the packet whether successful or not
35 * - there must be just one reference to the packet, which the caller passes to
36 * this function
37 */ 41 */
38int rxrpc_queue_rcv_skb(struct rxrpc_call *call, struct sk_buff *skb, 42static void rxrpc_rotate_tx_window(struct rxrpc_call *call, rxrpc_seq_t to)
39 bool force, bool terminal)
40{ 43{
41 struct rxrpc_skb_priv *sp; 44 struct sk_buff *skb, *list = NULL;
42 struct rxrpc_sock *rx; 45 int ix;
43 struct sock *sk;
44 int ret;
45 46
46 _enter(",,%d,%d", force, terminal); 47 spin_lock(&call->lock);
47 48
48 ASSERT(!irqs_disabled()); 49 while (before(call->tx_hard_ack, to)) {
50 call->tx_hard_ack++;
51 ix = call->tx_hard_ack & RXRPC_RXTX_BUFF_MASK;
52 skb = call->rxtx_buffer[ix];
53 rxrpc_see_skb(skb);
54 call->rxtx_buffer[ix] = NULL;
55 call->rxtx_annotations[ix] = 0;
56 skb->next = list;
57 list = skb;
58 }
49 59
50 sp = rxrpc_skb(skb); 60 spin_unlock(&call->lock);
51 ASSERTCMP(sp->call, ==, call);
52 61
53 /* if we've already posted the terminal message for a call, then we 62 while (list) {
54 * don't post any more */ 63 skb = list;
55 if (test_bit(RXRPC_CALL_TERMINAL_MSG, &call->flags)) { 64 list = skb->next;
56 _debug("already terminated"); 65 skb->next = NULL;
57 ASSERTCMP(call->state, >=, RXRPC_CALL_COMPLETE);
58 rxrpc_free_skb(skb); 66 rxrpc_free_skb(skb);
59 return 0;
60 } 67 }
68}
61 69
62 /* The socket may go away under us */ 70/*
63 ret = 0; 71 * End the transmission phase of a call.
64 rcu_read_lock(); 72 *
65 rx = rcu_dereference(call->socket); 73 * This occurs when we get an ACKALL packet, the first DATA packet of a reply,
66 if (!rx) 74 * or a final ACK packet.
67 goto out; 75 */
68 sk = &rx->sk; 76static bool rxrpc_end_tx_phase(struct rxrpc_call *call, const char *abort_why)
69 if (sock_flag(sk, SOCK_DEAD)) 77{
70 goto out; 78 _enter("");
71 79
72 if (!force) { 80 switch (call->state) {
73 /* cast skb->rcvbuf to unsigned... It's pointless, but 81 case RXRPC_CALL_CLIENT_RECV_REPLY:
74 * reduces number of warnings when compiling with -W 82 return true;
75 * --ANK */ 83 case RXRPC_CALL_CLIENT_AWAIT_REPLY:
76// ret = -ENOBUFS; 84 case RXRPC_CALL_SERVER_AWAIT_ACK:
77// if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >= 85 break;
78// (unsigned int) sk->sk_rcvbuf) 86 default:
79// goto out; 87 rxrpc_proto_abort(abort_why, call, call->tx_top);
80 88 return false;
81 ret = sk_filter(sk, skb);
82 if (ret < 0)
83 goto out;
84 } 89 }
85 90
86 spin_lock_bh(&sk->sk_receive_queue.lock); 91 rxrpc_rotate_tx_window(call, call->tx_top);
87 if (!test_bit(RXRPC_CALL_TERMINAL_MSG, &call->flags) &&
88 !test_bit(RXRPC_CALL_RELEASED, &call->flags) &&
89 sk->sk_state != RXRPC_CLOSE) {
90 skb->destructor = rxrpc_packet_destructor;
91 skb->dev = NULL;
92 skb->sk = sk;
93 atomic_add(skb->truesize, &sk->sk_rmem_alloc);
94
95 if (terminal) {
96 _debug("<<<< TERMINAL MESSAGE >>>>");
97 set_bit(RXRPC_CALL_TERMINAL_MSG, &call->flags);
98 }
99 92
100 /* allow interception by a kernel service */ 93 write_lock(&call->state_lock);
101 if (skb->mark == RXRPC_SKB_MARK_NEW_CALL &&
102 rx->notify_new_call) {
103 spin_unlock_bh(&sk->sk_receive_queue.lock);
104 skb_queue_tail(&call->knlrecv_queue, skb);
105 rx->notify_new_call(&rx->sk, NULL, 0);
106 } else if (call->notify_rx) {
107 spin_unlock_bh(&sk->sk_receive_queue.lock);
108 skb_queue_tail(&call->knlrecv_queue, skb);
109 call->notify_rx(&rx->sk, call, call->user_call_ID);
110 } else {
111 _net("post skb %p", skb);
112 __skb_queue_tail(&sk->sk_receive_queue, skb);
113 spin_unlock_bh(&sk->sk_receive_queue.lock);
114 94
115 sk->sk_data_ready(sk); 95 switch (call->state) {
116 } 96 default:
117 skb = NULL; 97 break;
118 } else { 98 case RXRPC_CALL_CLIENT_AWAIT_REPLY:
119 spin_unlock_bh(&sk->sk_receive_queue.lock); 99 call->state = RXRPC_CALL_CLIENT_RECV_REPLY;
100 break;
101 case RXRPC_CALL_SERVER_AWAIT_ACK:
102 __rxrpc_call_completed(call);
103 rxrpc_notify_socket(call);
104 break;
120 } 105 }
121 ret = 0;
122 106
123out: 107 write_unlock(&call->state_lock);
124 rxrpc_free_skb(skb); 108 _leave(" = ok");
125 rcu_read_unlock(); 109 return true;
110}
111
112/*
113 * Scan a jumbo packet to validate its structure and to work out how many
114 * subpackets it contains.
115 *
116 * A jumbo packet is a collection of consecutive packets glued together with
117 * little headers between that indicate how to change the initial header for
118 * each subpacket.
119 *
120 * RXRPC_JUMBO_PACKET must be set on all but the last subpacket - and all but
121 * the last are RXRPC_JUMBO_DATALEN in size. The last subpacket may be of any
122 * size.
123 */
124static bool rxrpc_validate_jumbo(struct sk_buff *skb)
125{
126 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
127 unsigned int offset = sp->offset;
128 unsigned int len = skb->data_len;
129 int nr_jumbo = 1;
130 u8 flags = sp->hdr.flags;
131
132 do {
133 nr_jumbo++;
134 if (len - offset < RXRPC_JUMBO_SUBPKTLEN)
135 goto protocol_error;
136 if (flags & RXRPC_LAST_PACKET)
137 goto protocol_error;
138 offset += RXRPC_JUMBO_DATALEN;
139 if (skb_copy_bits(skb, offset, &flags, 1) < 0)
140 goto protocol_error;
141 offset += sizeof(struct rxrpc_jumbo_header);
142 } while (flags & RXRPC_JUMBO_PACKET);
143
144 sp->nr_jumbo = nr_jumbo;
145 return true;
126 146
127 _leave(" = %d", ret); 147protocol_error:
128 return ret; 148 return false;
129} 149}
130 150
131/* 151/*
132 * process a DATA packet, posting the packet to the appropriate queue 152 * Handle reception of a duplicate packet.
133 * - eats the packet if successful 153 *
154 * We have to take care to avoid an attack here whereby we're given a series of
155 * jumbograms, each with a sequence number one before the preceding one and
156 * filled up to maximum UDP size. If they never send us the first packet in
157 * the sequence, they can cause us to have to hold on to around 2MiB of kernel
158 * space until the call times out.
159 *
160 * We limit the space usage by only accepting three duplicate jumbo packets per
161 * call. After that, we tell the other side we're no longer accepting jumbos
162 * (that information is encoded in the ACK packet).
134 */ 163 */
135static int rxrpc_fast_process_data(struct rxrpc_call *call, 164static void rxrpc_input_dup_data(struct rxrpc_call *call, rxrpc_seq_t seq,
136 struct sk_buff *skb, u32 seq) 165 u8 annotation, bool *_jumbo_dup)
137{ 166{
138 struct rxrpc_skb_priv *sp; 167 /* Discard normal packets that are duplicates. */
139 bool terminal; 168 if (annotation == 0)
140 int ret, ackbit, ack; 169 return;
141 u32 serial;
142 u16 skew;
143 u8 flags;
144 170
145 _enter("{%u,%u},,{%u}", call->rx_data_post, call->rx_first_oos, seq); 171 /* Skip jumbo subpackets that are duplicates. When we've had three or
172 * more partially duplicate jumbo packets, we refuse to take any more
173 * jumbos for this call.
174 */
175 if (!*_jumbo_dup) {
176 call->nr_jumbo_dup++;
177 *_jumbo_dup = true;
178 }
179}
146 180
147 sp = rxrpc_skb(skb); 181/*
148 ASSERTCMP(sp->call, ==, NULL); 182 * Process a DATA packet, adding the packet to the Rx ring.
149 flags = sp->hdr.flags; 183 */
150 serial = sp->hdr.serial; 184static void rxrpc_input_data(struct rxrpc_call *call, struct sk_buff *skb,
151 skew = skb->priority; 185 u16 skew)
186{
187 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
188 unsigned int offset = sp->offset;
189 unsigned int ix;
190 rxrpc_serial_t serial = sp->hdr.serial, ack_serial = 0;
191 rxrpc_seq_t seq = sp->hdr.seq, hard_ack;
192 bool immediate_ack = false, jumbo_dup = false, queued;
193 u16 len;
194 u8 ack = 0, flags, annotation = 0;
152 195
153 spin_lock(&call->lock); 196 _enter("{%u,%u},{%u,%u}",
197 call->rx_hard_ack, call->rx_top, skb->data_len, seq);
154 198
155 if (call->state > RXRPC_CALL_COMPLETE) 199 _proto("Rx DATA %%%u { #%u f=%02x }",
156 goto discard; 200 sp->hdr.serial, seq, sp->hdr.flags);
157 201
158 ASSERTCMP(call->rx_data_expect, >=, call->rx_data_post); 202 if (call->state >= RXRPC_CALL_COMPLETE)
159 ASSERTCMP(call->rx_data_post, >=, call->rx_data_recv); 203 return;
160 ASSERTCMP(call->rx_data_recv, >=, call->rx_data_eaten);
161 204
162 if (seq < call->rx_data_post) { 205 /* Received data implicitly ACKs all of the request packets we sent
163 _debug("dup #%u [-%u]", seq, call->rx_data_post); 206 * when we're acting as a client.
164 ack = RXRPC_ACK_DUPLICATE; 207 */
165 ret = -ENOBUFS; 208 if (call->state == RXRPC_CALL_CLIENT_AWAIT_REPLY &&
166 goto discard_and_ack; 209 !rxrpc_end_tx_phase(call, "ETD"))
167 } 210 return;
168 211
169 /* we may already have the packet in the out of sequence queue */ 212 call->ackr_prev_seq = seq;
170 ackbit = seq - (call->rx_data_eaten + 1);
171 ASSERTCMP(ackbit, >=, 0);
172 if (__test_and_set_bit(ackbit, call->ackr_window)) {
173 _debug("dup oos #%u [%u,%u]",
174 seq, call->rx_data_eaten, call->rx_data_post);
175 ack = RXRPC_ACK_DUPLICATE;
176 goto discard_and_ack;
177 }
178 213
179 if (seq >= call->ackr_win_top) { 214 hard_ack = READ_ONCE(call->rx_hard_ack);
180 _debug("exceed #%u [%u]", seq, call->ackr_win_top); 215 if (after(seq, hard_ack + call->rx_winsize)) {
181 __clear_bit(ackbit, call->ackr_window);
182 ack = RXRPC_ACK_EXCEEDS_WINDOW; 216 ack = RXRPC_ACK_EXCEEDS_WINDOW;
183 goto discard_and_ack; 217 ack_serial = serial;
218 goto ack;
184 } 219 }
185 220
186 if (seq == call->rx_data_expect) { 221 flags = sp->hdr.flags;
187 clear_bit(RXRPC_CALL_EXPECT_OOS, &call->flags); 222 if (flags & RXRPC_JUMBO_PACKET) {
188 call->rx_data_expect++; 223 if (call->nr_jumbo_dup > 3) {
189 } else if (seq > call->rx_data_expect) { 224 ack = RXRPC_ACK_NOSPACE;
190 _debug("oos #%u [%u]", seq, call->rx_data_expect); 225 ack_serial = serial;
191 call->rx_data_expect = seq + 1; 226 goto ack;
192 if (test_and_set_bit(RXRPC_CALL_EXPECT_OOS, &call->flags)) {
193 ack = RXRPC_ACK_OUT_OF_SEQUENCE;
194 goto enqueue_and_ack;
195 } 227 }
196 goto enqueue_packet; 228 annotation = 1;
197 } 229 }
198 230
199 if (seq != call->rx_data_post) { 231next_subpacket:
200 _debug("ahead #%u [%u]", seq, call->rx_data_post); 232 queued = false;
201 goto enqueue_packet; 233 ix = seq & RXRPC_RXTX_BUFF_MASK;
234 len = skb->data_len;
235 if (flags & RXRPC_JUMBO_PACKET)
236 len = RXRPC_JUMBO_DATALEN;
237
238 if (flags & RXRPC_LAST_PACKET) {
239 if (test_and_set_bit(RXRPC_CALL_RX_LAST, &call->flags) &&
240 seq != call->rx_top)
241 return rxrpc_proto_abort("LSN", call, seq);
242 } else {
243 if (test_bit(RXRPC_CALL_RX_LAST, &call->flags) &&
244 after_eq(seq, call->rx_top))
245 return rxrpc_proto_abort("LSA", call, seq);
202 } 246 }
203 247
204 if (test_bit(RXRPC_CALL_RCVD_LAST, &call->flags)) 248 if (before_eq(seq, hard_ack)) {
205 goto protocol_error; 249 ack = RXRPC_ACK_DUPLICATE;
206 250 ack_serial = serial;
207 /* if the packet need security things doing to it, then it goes down 251 goto skip;
208 * the slow path */ 252 }
209 if (call->security_ix) 253
210 goto enqueue_packet; 254 if (flags & RXRPC_REQUEST_ACK && !ack) {
211 255 ack = RXRPC_ACK_REQUESTED;
212 sp->call = call; 256 ack_serial = serial;
213 rxrpc_get_call_for_skb(call, skb); 257 }
214 terminal = ((flags & RXRPC_LAST_PACKET) && 258
215 !(flags & RXRPC_CLIENT_INITIATED)); 259 if (call->rxtx_buffer[ix]) {
216 ret = rxrpc_queue_rcv_skb(call, skb, false, terminal); 260 rxrpc_input_dup_data(call, seq, annotation, &jumbo_dup);
217 if (ret < 0) { 261 if (ack != RXRPC_ACK_DUPLICATE) {
218 if (ret == -ENOMEM || ret == -ENOBUFS) { 262 ack = RXRPC_ACK_DUPLICATE;
219 __clear_bit(ackbit, call->ackr_window); 263 ack_serial = serial;
220 ack = RXRPC_ACK_NOSPACE;
221 goto discard_and_ack;
222 } 264 }
223 goto out; 265 immediate_ack = true;
266 goto skip;
224 } 267 }
225 268
226 skb = NULL; 269 /* Queue the packet. We use a couple of memory barriers here as need
227 sp = NULL; 270 * to make sure that rx_top is perceived to be set after the buffer
228 271 * pointer and that the buffer pointer is set after the annotation and
229 _debug("post #%u", seq); 272 * the skb data.
230 ASSERTCMP(call->rx_data_post, ==, seq); 273 *
231 call->rx_data_post++; 274 * Barriers against rxrpc_recvmsg_data() and rxrpc_rotate_rx_window()
232 275 * and also rxrpc_fill_out_ack().
233 if (flags & RXRPC_LAST_PACKET) 276 */
234 set_bit(RXRPC_CALL_RCVD_LAST, &call->flags); 277 rxrpc_get_skb(skb);
235 278 call->rxtx_annotations[ix] = annotation;
236 /* if we've reached an out of sequence packet then we need to drain 279 smp_wmb();
237 * that queue into the socket Rx queue now */ 280 call->rxtx_buffer[ix] = skb;
238 if (call->rx_data_post == call->rx_first_oos) { 281 if (after(seq, call->rx_top))
239 _debug("drain rx oos now"); 282 smp_store_release(&call->rx_top, seq);
240 read_lock(&call->state_lock); 283 queued = true;
241 if (call->state < RXRPC_CALL_COMPLETE && 284
242 !test_and_set_bit(RXRPC_CALL_EV_DRAIN_RX_OOS, &call->events)) 285 if (after_eq(seq, call->rx_expect_next)) {
243 rxrpc_queue_call(call); 286 if (after(seq, call->rx_expect_next)) {
244 read_unlock(&call->state_lock); 287 _net("OOS %u > %u", seq, call->rx_expect_next);
288 ack = RXRPC_ACK_OUT_OF_SEQUENCE;
289 ack_serial = serial;
290 }
291 call->rx_expect_next = seq + 1;
245 } 292 }
246 293
247 spin_unlock(&call->lock); 294skip:
248 atomic_inc(&call->ackr_not_idle); 295 offset += len;
249 rxrpc_propose_ACK(call, RXRPC_ACK_DELAY, skew, serial, false); 296 if (flags & RXRPC_JUMBO_PACKET) {
250 _leave(" = 0 [posted]"); 297 if (skb_copy_bits(skb, offset, &flags, 1) < 0)
251 return 0; 298 return rxrpc_proto_abort("XJF", call, seq);
299 offset += sizeof(struct rxrpc_jumbo_header);
300 seq++;
301 serial++;
302 annotation++;
303 if (flags & RXRPC_JUMBO_PACKET)
304 annotation |= RXRPC_RX_ANNO_JLAST;
305
306 _proto("Rx DATA Jumbo %%%u", serial);
307 goto next_subpacket;
308 }
252 309
253protocol_error: 310 if (queued && flags & RXRPC_LAST_PACKET && !ack) {
254 ret = -EBADMSG; 311 ack = RXRPC_ACK_DELAY;
255out: 312 ack_serial = serial;
256 spin_unlock(&call->lock); 313 }
257 _leave(" = %d", ret);
258 return ret;
259 314
260discard_and_ack: 315ack:
261 _debug("discard and ACK packet %p", skb); 316 if (ack)
262 __rxrpc_propose_ACK(call, ack, skew, serial, true); 317 rxrpc_propose_ACK(call, ack, skew, ack_serial,
263discard: 318 immediate_ack, true);
264 spin_unlock(&call->lock);
265 rxrpc_free_skb(skb);
266 _leave(" = 0 [discarded]");
267 return 0;
268 319
269enqueue_and_ack: 320 if (sp->hdr.seq == READ_ONCE(call->rx_hard_ack) + 1)
270 __rxrpc_propose_ACK(call, ack, skew, serial, true); 321 rxrpc_notify_socket(call);
271enqueue_packet: 322 _leave(" [queued]");
272 _net("defer skb %p", skb);
273 spin_unlock(&call->lock);
274 skb_queue_tail(&call->rx_queue, skb);
275 atomic_inc(&call->ackr_not_idle);
276 read_lock(&call->state_lock);
277 if (call->state < RXRPC_CALL_COMPLETE)
278 rxrpc_queue_call(call);
279 read_unlock(&call->state_lock);
280 _leave(" = 0 [queued]");
281 return 0;
282} 323}
283 324
284/* 325/*
285 * assume an implicit ACKALL of the transmission phase of a client socket upon 326 * Process the extra information that may be appended to an ACK packet
286 * reception of the first reply packet
287 */ 327 */
288static void rxrpc_assume_implicit_ackall(struct rxrpc_call *call, u32 serial) 328static void rxrpc_input_ackinfo(struct rxrpc_call *call, struct sk_buff *skb,
329 struct rxrpc_ackinfo *ackinfo)
289{ 330{
290 write_lock_bh(&call->state_lock); 331 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
291 332 struct rxrpc_peer *peer;
292 switch (call->state) { 333 unsigned int mtu;
293 case RXRPC_CALL_CLIENT_AWAIT_REPLY: 334
294 call->state = RXRPC_CALL_CLIENT_RECV_REPLY; 335 _proto("Rx ACK %%%u Info { rx=%u max=%u rwin=%u jm=%u }",
295 call->acks_latest = serial; 336 sp->hdr.serial,
296 337 ntohl(ackinfo->rxMTU), ntohl(ackinfo->maxMTU),
297 _debug("implicit ACKALL %%%u", call->acks_latest); 338 ntohl(ackinfo->rwind), ntohl(ackinfo->jumbo_max));
298 set_bit(RXRPC_CALL_EV_RCVD_ACKALL, &call->events); 339
299 write_unlock_bh(&call->state_lock); 340 if (call->tx_winsize > ntohl(ackinfo->rwind))
341 call->tx_winsize = ntohl(ackinfo->rwind);
342
343 mtu = min(ntohl(ackinfo->rxMTU), ntohl(ackinfo->maxMTU));
344
345 peer = call->peer;
346 if (mtu < peer->maxdata) {
347 spin_lock_bh(&peer->lock);
348 peer->maxdata = mtu;
349 peer->mtu = mtu + peer->hdrsize;
350 spin_unlock_bh(&peer->lock);
351 _net("Net MTU %u (maxdata %u)", peer->mtu, peer->maxdata);
352 }
353}
300 354
301 if (try_to_del_timer_sync(&call->resend_timer) >= 0) { 355/*
302 clear_bit(RXRPC_CALL_EV_RESEND_TIMER, &call->events); 356 * Process individual soft ACKs.
303 clear_bit(RXRPC_CALL_EV_RESEND, &call->events); 357 *
304 clear_bit(RXRPC_CALL_RUN_RTIMER, &call->flags); 358 * Each ACK in the array corresponds to one packet and can be either an ACK or
359 * a NAK. If we get find an explicitly NAK'd packet we resend immediately;
360 * packets that lie beyond the end of the ACK list are scheduled for resend by
361 * the timer on the basis that the peer might just not have processed them at
362 * the time the ACK was sent.
363 */
364static void rxrpc_input_soft_acks(struct rxrpc_call *call, u8 *acks,
365 rxrpc_seq_t seq, int nr_acks)
366{
367 bool resend = false;
368 int ix;
369
370 for (; nr_acks > 0; nr_acks--, seq++) {
371 ix = seq & RXRPC_RXTX_BUFF_MASK;
372 switch (*acks) {
373 case RXRPC_ACK_TYPE_ACK:
374 call->rxtx_annotations[ix] = RXRPC_TX_ANNO_ACK;
375 break;
376 case RXRPC_ACK_TYPE_NACK:
377 if (call->rxtx_annotations[ix] == RXRPC_TX_ANNO_NAK)
378 continue;
379 call->rxtx_annotations[ix] = RXRPC_TX_ANNO_NAK;
380 resend = true;
381 break;
382 default:
383 return rxrpc_proto_abort("SFT", call, 0);
305 } 384 }
306 break;
307
308 default:
309 write_unlock_bh(&call->state_lock);
310 break;
311 } 385 }
386
387 if (resend &&
388 !test_and_set_bit(RXRPC_CALL_EV_RESEND, &call->events))
389 rxrpc_queue_call(call);
312} 390}
313 391
314/* 392/*
315 * post an incoming packet to the nominated call to deal with 393 * Process an ACK packet.
316 * - must get rid of the sk_buff, either by freeing it or by queuing it 394 *
395 * ack.firstPacket is the sequence number of the first soft-ACK'd/NAK'd packet
396 * in the ACK array. Anything before that is hard-ACK'd and may be discarded.
397 *
398 * A hard-ACK means that a packet has been processed and may be discarded; a
399 * soft-ACK means that the packet may be discarded and retransmission
400 * requested. A phase is complete when all packets are hard-ACK'd.
317 */ 401 */
318void rxrpc_fast_process_packet(struct rxrpc_call *call, struct sk_buff *skb) 402static void rxrpc_input_ack(struct rxrpc_call *call, struct sk_buff *skb,
403 u16 skew)
319{ 404{
320 struct rxrpc_skb_priv *sp = rxrpc_skb(skb); 405 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
321 __be32 wtmp; 406 union {
322 u32 abort_code; 407 struct rxrpc_ackpacket ack;
323 408 struct rxrpc_ackinfo info;
324 _enter("%p,%p", call, skb); 409 u8 acks[RXRPC_MAXACKS];
325 410 } buf;
326 ASSERT(!irqs_disabled()); 411 rxrpc_seq_t first_soft_ack, hard_ack;
327 412 int nr_acks, offset;
328#if 0 // INJECT RX ERROR 413
329 if (sp->hdr.type == RXRPC_PACKET_TYPE_DATA) { 414 _enter("");
330 static int skip = 0; 415
331 if (++skip == 3) { 416 if (skb_copy_bits(skb, sp->offset, &buf.ack, sizeof(buf.ack)) < 0) {
332 printk("DROPPED 3RD PACKET!!!!!!!!!!!!!\n"); 417 _debug("extraction failure");
333 skip = 0; 418 return rxrpc_proto_abort("XAK", call, 0);
334 goto free_packet;
335 }
336 } 419 }
337#endif 420 sp->offset += sizeof(buf.ack);
338 421
339 /* request ACK generation for any ACK or DATA packet that requests 422 first_soft_ack = ntohl(buf.ack.firstPacket);
340 * it */ 423 hard_ack = first_soft_ack - 1;
341 if (sp->hdr.flags & RXRPC_REQUEST_ACK) { 424 nr_acks = buf.ack.nAcks;
342 _proto("ACK Requested on %%%u", sp->hdr.serial); 425
426 _proto("Rx ACK %%%u { m=%hu f=#%u p=#%u s=%%%u r=%s n=%u }",
427 sp->hdr.serial,
428 ntohs(buf.ack.maxSkew),
429 first_soft_ack,
430 ntohl(buf.ack.previousPacket),
431 ntohl(buf.ack.serial),
432 rxrpc_acks(buf.ack.reason),
433 buf.ack.nAcks);
434
435 if (buf.ack.reason == RXRPC_ACK_PING) {
436 _proto("Rx ACK %%%u PING Request", sp->hdr.serial);
437 rxrpc_propose_ACK(call, RXRPC_ACK_PING_RESPONSE,
438 skew, sp->hdr.serial, true, true);
439 } else if (sp->hdr.flags & RXRPC_REQUEST_ACK) {
343 rxrpc_propose_ACK(call, RXRPC_ACK_REQUESTED, 440 rxrpc_propose_ACK(call, RXRPC_ACK_REQUESTED,
344 skb->priority, sp->hdr.serial, false); 441 skew, sp->hdr.serial, true, true);
345 } 442 }
346 443
347 switch (sp->hdr.type) { 444 offset = sp->offset + nr_acks + 3;
348 case RXRPC_PACKET_TYPE_ABORT: 445 if (skb->data_len >= offset + sizeof(buf.info)) {
349 _debug("abort"); 446 if (skb_copy_bits(skb, offset, &buf.info, sizeof(buf.info)) < 0)
350 447 return rxrpc_proto_abort("XAI", call, 0);
351 if (skb_copy_bits(skb, 0, &wtmp, sizeof(wtmp)) < 0) 448 rxrpc_input_ackinfo(call, skb, &buf.info);
352 goto protocol_error; 449 }
353
354 abort_code = ntohl(wtmp);
355 _proto("Rx ABORT %%%u { %x }", sp->hdr.serial, abort_code);
356
357 if (__rxrpc_set_call_completion(call,
358 RXRPC_CALL_REMOTELY_ABORTED,
359 abort_code, ECONNABORTED)) {
360 set_bit(RXRPC_CALL_EV_RCVD_ABORT, &call->events);
361 rxrpc_queue_call(call);
362 }
363 goto free_packet;
364
365 case RXRPC_PACKET_TYPE_BUSY:
366 _proto("Rx BUSY %%%u", sp->hdr.serial);
367
368 if (rxrpc_is_service_call(call))
369 goto protocol_error;
370 450
371 write_lock_bh(&call->state_lock); 451 if (first_soft_ack == 0)
372 switch (call->state) { 452 return rxrpc_proto_abort("AK0", call, 0);
373 case RXRPC_CALL_CLIENT_SEND_REQUEST:
374 __rxrpc_set_call_completion(call,
375 RXRPC_CALL_SERVER_BUSY,
376 0, EBUSY);
377 set_bit(RXRPC_CALL_EV_RCVD_BUSY, &call->events);
378 rxrpc_queue_call(call);
379 case RXRPC_CALL_SERVER_BUSY:
380 goto free_packet_unlock;
381 default:
382 goto protocol_error_locked;
383 }
384 453
454 /* Ignore ACKs unless we are or have just been transmitting. */
455 switch (call->state) {
456 case RXRPC_CALL_CLIENT_SEND_REQUEST:
457 case RXRPC_CALL_CLIENT_AWAIT_REPLY:
458 case RXRPC_CALL_SERVER_SEND_REPLY:
459 case RXRPC_CALL_SERVER_AWAIT_ACK:
460 break;
385 default: 461 default:
386 _proto("Rx %s %%%u", rxrpc_pkts[sp->hdr.type], sp->hdr.serial); 462 return;
387 goto protocol_error; 463 }
388
389 case RXRPC_PACKET_TYPE_DATA:
390 _proto("Rx DATA %%%u { #%u }", sp->hdr.serial, sp->hdr.seq);
391
392 if (sp->hdr.seq == 0)
393 goto protocol_error;
394
395 call->ackr_prev_seq = sp->hdr.seq;
396 464
397 /* received data implicitly ACKs all of the request packets we 465 /* Discard any out-of-order or duplicate ACKs. */
398 * sent when we're acting as a client */ 466 if ((int)sp->hdr.serial - (int)call->acks_latest <= 0) {
399 if (call->state == RXRPC_CALL_CLIENT_AWAIT_REPLY) 467 _debug("discard ACK %d <= %d",
400 rxrpc_assume_implicit_ackall(call, sp->hdr.serial); 468 sp->hdr.serial, call->acks_latest);
469 return;
470 }
471 call->acks_latest = sp->hdr.serial;
401 472
402 switch (rxrpc_fast_process_data(call, skb, sp->hdr.seq)) { 473 if (test_bit(RXRPC_CALL_TX_LAST, &call->flags) &&
403 case 0: 474 hard_ack == call->tx_top) {
404 skb = NULL; 475 rxrpc_end_tx_phase(call, "ETA");
405 goto done; 476 return;
477 }
406 478
407 default: 479 if (before(hard_ack, call->tx_hard_ack) ||
408 BUG(); 480 after(hard_ack, call->tx_top))
481 return rxrpc_proto_abort("AKW", call, 0);
409 482
410 /* data packet received beyond the last packet */ 483 if (after(hard_ack, call->tx_hard_ack))
411 case -EBADMSG: 484 rxrpc_rotate_tx_window(call, hard_ack);
412 goto protocol_error;
413 }
414 485
415 case RXRPC_PACKET_TYPE_ACKALL: 486 if (after(first_soft_ack, call->tx_top))
416 case RXRPC_PACKET_TYPE_ACK: 487 return;
417 /* ACK processing is done in process context */
418 read_lock_bh(&call->state_lock);
419 if (call->state < RXRPC_CALL_COMPLETE) {
420 skb_queue_tail(&call->rx_queue, skb);
421 rxrpc_queue_call(call);
422 skb = NULL;
423 }
424 read_unlock_bh(&call->state_lock);
425 goto free_packet;
426 }
427 488
428protocol_error: 489 if (nr_acks > call->tx_top - first_soft_ack + 1)
429 _debug("protocol error"); 490 nr_acks = first_soft_ack - call->tx_top + 1;
430 write_lock_bh(&call->state_lock); 491 if (skb_copy_bits(skb, sp->offset, buf.acks, nr_acks) < 0)
431protocol_error_locked: 492 return rxrpc_proto_abort("XSA", call, 0);
432 if (__rxrpc_abort_call("FPR", call, 0, RX_PROTOCOL_ERROR, EPROTO)) 493 rxrpc_input_soft_acks(call, buf.acks, first_soft_ack, nr_acks);
433 rxrpc_queue_call(call);
434free_packet_unlock:
435 write_unlock_bh(&call->state_lock);
436free_packet:
437 rxrpc_free_skb(skb);
438done:
439 _leave("");
440} 494}
441 495
442/* 496/*
443 * split up a jumbo data packet 497 * Process an ACKALL packet.
444 */ 498 */
445static void rxrpc_process_jumbo_packet(struct rxrpc_call *call, 499static void rxrpc_input_ackall(struct rxrpc_call *call, struct sk_buff *skb)
446 struct sk_buff *jumbo)
447{ 500{
448 struct rxrpc_jumbo_header jhdr; 501 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
449 struct rxrpc_skb_priv *sp;
450 struct sk_buff *part;
451
452 _enter(",{%u,%u}", jumbo->data_len, jumbo->len);
453
454 sp = rxrpc_skb(jumbo);
455
456 do {
457 sp->hdr.flags &= ~RXRPC_JUMBO_PACKET;
458
459 /* make a clone to represent the first subpacket in what's left
460 * of the jumbo packet */
461 part = skb_clone(jumbo, GFP_ATOMIC);
462 if (!part) {
463 /* simply ditch the tail in the event of ENOMEM */
464 pskb_trim(jumbo, RXRPC_JUMBO_DATALEN);
465 break;
466 }
467 rxrpc_new_skb(part);
468
469 pskb_trim(part, RXRPC_JUMBO_DATALEN);
470
471 if (!pskb_pull(jumbo, RXRPC_JUMBO_DATALEN))
472 goto protocol_error;
473 502
474 if (skb_copy_bits(jumbo, 0, &jhdr, sizeof(jhdr)) < 0) 503 _proto("Rx ACKALL %%%u", sp->hdr.serial);
475 goto protocol_error;
476 if (!pskb_pull(jumbo, sizeof(jhdr)))
477 BUG();
478 504
479 sp->hdr.seq += 1; 505 rxrpc_end_tx_phase(call, "ETL");
480 sp->hdr.serial += 1; 506}
481 sp->hdr.flags = jhdr.flags;
482 sp->hdr._rsvd = ntohs(jhdr._rsvd);
483 507
484 _proto("Rx DATA Jumbo %%%u", sp->hdr.serial - 1); 508/*
509 * Process an ABORT packet.
510 */
511static void rxrpc_input_abort(struct rxrpc_call *call, struct sk_buff *skb)
512{
513 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
514 __be32 wtmp;
515 u32 abort_code = RX_CALL_DEAD;
485 516
486 rxrpc_fast_process_packet(call, part); 517 _enter("");
487 part = NULL;
488 518
489 } while (sp->hdr.flags & RXRPC_JUMBO_PACKET); 519 if (skb->len >= 4 &&
520 skb_copy_bits(skb, sp->offset, &wtmp, sizeof(wtmp)) >= 0)
521 abort_code = ntohl(wtmp);
490 522
491 rxrpc_fast_process_packet(call, jumbo); 523 _proto("Rx ABORT %%%u { %x }", sp->hdr.serial, abort_code);
492 _leave("");
493 return;
494 524
495protocol_error: 525 if (rxrpc_set_call_completion(call, RXRPC_CALL_REMOTELY_ABORTED,
496 _debug("protocol error"); 526 abort_code, ECONNABORTED))
497 rxrpc_free_skb(part); 527 rxrpc_notify_socket(call);
498 if (rxrpc_abort_call("PJP", call, sp->hdr.seq,
499 RX_PROTOCOL_ERROR, EPROTO))
500 rxrpc_queue_call(call);
501 rxrpc_free_skb(jumbo);
502 _leave("");
503} 528}
504 529
505/* 530/*
506 * post an incoming packet to the appropriate call/socket to deal with 531 * Process an incoming call packet.
507 * - must get rid of the sk_buff, either by freeing it or by queuing it
508 */ 532 */
509static void rxrpc_post_packet_to_call(struct rxrpc_connection *conn, 533static void rxrpc_input_call_packet(struct rxrpc_call *call,
510 struct rxrpc_call *call, 534 struct sk_buff *skb, u16 skew)
511 struct sk_buff *skb)
512{ 535{
513 struct rxrpc_skb_priv *sp; 536 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
514 537
515 _enter("%p,%p", call, skb); 538 _enter("%p,%p", call, skb);
516 539
517 sp = rxrpc_skb(skb); 540 switch (sp->hdr.type) {
518 541 case RXRPC_PACKET_TYPE_DATA:
519 _debug("extant call [%d]", call->state); 542 rxrpc_input_data(call, skb, skew);
520 543 break;
521 read_lock(&call->state_lock);
522 switch (call->state) {
523 case RXRPC_CALL_COMPLETE:
524 switch (call->completion) {
525 case RXRPC_CALL_LOCALLY_ABORTED:
526 if (!test_and_set_bit(RXRPC_CALL_EV_ABORT,
527 &call->events)) {
528 rxrpc_queue_call(call);
529 goto free_unlock;
530 }
531 default:
532 goto dead_call;
533 case RXRPC_CALL_SUCCEEDED:
534 if (rxrpc_is_service_call(call))
535 goto dead_call;
536 goto resend_final_ack;
537 }
538
539 case RXRPC_CALL_CLIENT_FINAL_ACK:
540 goto resend_final_ack;
541 544
542 default: 545 case RXRPC_PACKET_TYPE_ACK:
546 rxrpc_input_ack(call, skb, skew);
543 break; 547 break;
544 }
545 548
546 read_unlock(&call->state_lock); 549 case RXRPC_PACKET_TYPE_BUSY:
550 _proto("Rx BUSY %%%u", sp->hdr.serial);
547 551
548 if (sp->hdr.type == RXRPC_PACKET_TYPE_DATA && 552 /* Just ignore BUSY packets from the server; the retry and
549 sp->hdr.flags & RXRPC_JUMBO_PACKET) 553 * lifespan timers will take care of business. BUSY packets
550 rxrpc_process_jumbo_packet(call, skb); 554 * from the client don't make sense.
551 else 555 */
552 rxrpc_fast_process_packet(call, skb); 556 break;
553 557
554 goto done; 558 case RXRPC_PACKET_TYPE_ABORT:
559 rxrpc_input_abort(call, skb);
560 break;
555 561
556resend_final_ack: 562 case RXRPC_PACKET_TYPE_ACKALL:
557 _debug("final ack again"); 563 rxrpc_input_ackall(call, skb);
558 set_bit(RXRPC_CALL_EV_ACK_FINAL, &call->events); 564 break;
559 rxrpc_queue_call(call);
560 goto free_unlock;
561 565
562dead_call: 566 default:
563 if (sp->hdr.type != RXRPC_PACKET_TYPE_ABORT) { 567 _proto("Rx %s %%%u", rxrpc_pkts[sp->hdr.type], sp->hdr.serial);
564 skb->priority = RX_CALL_DEAD; 568 break;
565 rxrpc_reject_packet(conn->params.local, skb);
566 goto unlock;
567 } 569 }
568free_unlock: 570
569 rxrpc_free_skb(skb);
570unlock:
571 read_unlock(&call->state_lock);
572done:
573 _leave(""); 571 _leave("");
574} 572}
575 573
@@ -601,6 +599,17 @@ static void rxrpc_post_packet_to_local(struct rxrpc_local *local,
601} 599}
602 600
603/* 601/*
602 * put a packet up for transport-level abort
603 */
604static void rxrpc_reject_packet(struct rxrpc_local *local, struct sk_buff *skb)
605{
606 CHECK_SLAB_OKAY(&local->usage);
607
608 skb_queue_tail(&local->reject_queue, skb);
609 rxrpc_queue_local(local);
610}
611
612/*
604 * Extract the wire header from a packet and translate the byte order. 613 * Extract the wire header from a packet and translate the byte order.
605 */ 614 */
606static noinline 615static noinline
@@ -611,8 +620,6 @@ int rxrpc_extract_header(struct rxrpc_skb_priv *sp, struct sk_buff *skb)
611 /* dig out the RxRPC connection details */ 620 /* dig out the RxRPC connection details */
612 if (skb_copy_bits(skb, 0, &whdr, sizeof(whdr)) < 0) 621 if (skb_copy_bits(skb, 0, &whdr, sizeof(whdr)) < 0)
613 return -EBADMSG; 622 return -EBADMSG;
614 if (!pskb_pull(skb, sizeof(whdr)))
615 BUG();
616 623
617 memset(sp, 0, sizeof(*sp)); 624 memset(sp, 0, sizeof(*sp));
618 sp->hdr.epoch = ntohl(whdr.epoch); 625 sp->hdr.epoch = ntohl(whdr.epoch);
@@ -626,6 +633,7 @@ int rxrpc_extract_header(struct rxrpc_skb_priv *sp, struct sk_buff *skb)
626 sp->hdr.securityIndex = whdr.securityIndex; 633 sp->hdr.securityIndex = whdr.securityIndex;
627 sp->hdr._rsvd = ntohs(whdr._rsvd); 634 sp->hdr._rsvd = ntohs(whdr._rsvd);
628 sp->hdr.serviceId = ntohs(whdr.serviceId); 635 sp->hdr.serviceId = ntohs(whdr.serviceId);
636 sp->offset = sizeof(whdr);
629 return 0; 637 return 0;
630} 638}
631 639
@@ -637,19 +645,22 @@ int rxrpc_extract_header(struct rxrpc_skb_priv *sp, struct sk_buff *skb)
637 * shut down and the local endpoint from going away, thus sk_user_data will not 645 * shut down and the local endpoint from going away, thus sk_user_data will not
638 * be cleared until this function returns. 646 * be cleared until this function returns.
639 */ 647 */
640void rxrpc_data_ready(struct sock *sk) 648void rxrpc_data_ready(struct sock *udp_sk)
641{ 649{
642 struct rxrpc_connection *conn; 650 struct rxrpc_connection *conn;
651 struct rxrpc_channel *chan;
652 struct rxrpc_call *call;
643 struct rxrpc_skb_priv *sp; 653 struct rxrpc_skb_priv *sp;
644 struct rxrpc_local *local = sk->sk_user_data; 654 struct rxrpc_local *local = udp_sk->sk_user_data;
645 struct sk_buff *skb; 655 struct sk_buff *skb;
656 unsigned int channel;
646 int ret, skew; 657 int ret, skew;
647 658
648 _enter("%p", sk); 659 _enter("%p", udp_sk);
649 660
650 ASSERT(!irqs_disabled()); 661 ASSERT(!irqs_disabled());
651 662
652 skb = skb_recv_datagram(sk, 0, 1, &ret); 663 skb = skb_recv_datagram(udp_sk, 0, 1, &ret);
653 if (!skb) { 664 if (!skb) {
654 if (ret == -EAGAIN) 665 if (ret == -EAGAIN)
655 return; 666 return;
@@ -695,111 +706,122 @@ void rxrpc_data_ready(struct sock *sk)
695 goto bad_message; 706 goto bad_message;
696 } 707 }
697 708
698 if (sp->hdr.type == RXRPC_PACKET_TYPE_VERSION) { 709 switch (sp->hdr.type) {
710 case RXRPC_PACKET_TYPE_VERSION:
699 rxrpc_post_packet_to_local(local, skb); 711 rxrpc_post_packet_to_local(local, skb);
700 goto out; 712 goto out;
701 }
702 713
703 if (sp->hdr.type == RXRPC_PACKET_TYPE_DATA && 714 case RXRPC_PACKET_TYPE_BUSY:
704 (sp->hdr.callNumber == 0 || sp->hdr.seq == 0)) 715 if (sp->hdr.flags & RXRPC_CLIENT_INITIATED)
705 goto bad_message; 716 goto discard;
717
718 case RXRPC_PACKET_TYPE_DATA:
719 if (sp->hdr.callNumber == 0)
720 goto bad_message;
721 if (sp->hdr.flags & RXRPC_JUMBO_PACKET &&
722 !rxrpc_validate_jumbo(skb))
723 goto bad_message;
724 break;
725 }
706 726
707 rcu_read_lock(); 727 rcu_read_lock();
708 728
709 conn = rxrpc_find_connection_rcu(local, skb); 729 conn = rxrpc_find_connection_rcu(local, skb);
710 if (!conn) { 730 if (conn) {
711 skb->priority = 0; 731 if (sp->hdr.securityIndex != conn->security_ix)
712 goto cant_route_call; 732 goto wrong_security;
713 }
714 733
715 /* Note the serial number skew here */ 734 if (sp->hdr.callNumber == 0) {
716 skew = (int)sp->hdr.serial - (int)conn->hi_serial; 735 /* Connection-level packet */
717 if (skew >= 0) { 736 _debug("CONN %p {%d}", conn, conn->debug_id);
718 if (skew > 0) 737 rxrpc_post_packet_to_conn(conn, skb);
719 conn->hi_serial = sp->hdr.serial; 738 goto out_unlock;
720 skb->priority = 0; 739 }
721 } else { 740
722 skew = -skew; 741 /* Note the serial number skew here */
723 skb->priority = min(skew, 65535); 742 skew = (int)sp->hdr.serial - (int)conn->hi_serial;
724 } 743 if (skew >= 0) {
744 if (skew > 0)
745 conn->hi_serial = sp->hdr.serial;
746 } else {
747 skew = -skew;
748 skew = min(skew, 65535);
749 }
725 750
726 if (sp->hdr.callNumber == 0) {
727 /* Connection-level packet */
728 _debug("CONN %p {%d}", conn, conn->debug_id);
729 rxrpc_post_packet_to_conn(conn, skb);
730 goto out_unlock;
731 } else {
732 /* Call-bound packets are routed by connection channel. */ 751 /* Call-bound packets are routed by connection channel. */
733 unsigned int channel = sp->hdr.cid & RXRPC_CHANNELMASK; 752 channel = sp->hdr.cid & RXRPC_CHANNELMASK;
734 struct rxrpc_channel *chan = &conn->channels[channel]; 753 chan = &conn->channels[channel];
735 struct rxrpc_call *call;
736 754
737 /* Ignore really old calls */ 755 /* Ignore really old calls */
738 if (sp->hdr.callNumber < chan->last_call) 756 if (sp->hdr.callNumber < chan->last_call)
739 goto discard_unlock; 757 goto discard_unlock;
740 758
741 if (sp->hdr.callNumber == chan->last_call) { 759 if (sp->hdr.callNumber == chan->last_call) {
742 /* For the previous service call, if completed 760 /* For the previous service call, if completed successfully, we
743 * successfully, we discard all further packets. 761 * discard all further packets.
744 */ 762 */
745 if (rxrpc_conn_is_service(conn) && 763 if (rxrpc_conn_is_service(conn) &&
746 (chan->last_type == RXRPC_PACKET_TYPE_ACK || 764 (chan->last_type == RXRPC_PACKET_TYPE_ACK ||
747 sp->hdr.type == RXRPC_PACKET_TYPE_ABORT)) 765 sp->hdr.type == RXRPC_PACKET_TYPE_ABORT))
748 goto discard_unlock; 766 goto discard_unlock;
749 767
750 /* But otherwise we need to retransmit the final packet 768 /* But otherwise we need to retransmit the final packet from
751 * from data cached in the connection record. 769 * data cached in the connection record.
752 */ 770 */
753 rxrpc_post_packet_to_conn(conn, skb); 771 rxrpc_post_packet_to_conn(conn, skb);
754 goto out_unlock; 772 goto out_unlock;
755 } 773 }
756 774
757 call = rcu_dereference(chan->call); 775 call = rcu_dereference(chan->call);
758 if (!call || atomic_read(&call->usage) == 0) 776 } else {
759 goto cant_route_call; 777 skew = 0;
778 call = NULL;
779 }
760 780
761 rxrpc_see_call(call); 781 if (!call || atomic_read(&call->usage) == 0) {
762 rxrpc_post_packet_to_call(conn, call, skb); 782 if (!(sp->hdr.type & RXRPC_CLIENT_INITIATED) ||
763 goto out_unlock; 783 sp->hdr.callNumber == 0 ||
784 sp->hdr.type != RXRPC_PACKET_TYPE_DATA)
785 goto bad_message_unlock;
786 if (sp->hdr.seq != 1)
787 goto discard_unlock;
788 call = rxrpc_new_incoming_call(local, conn, skb);
789 if (!call) {
790 rcu_read_unlock();
791 goto reject_packet;
792 }
764 } 793 }
765 794
795 rxrpc_input_call_packet(call, skb, skew);
796 goto discard_unlock;
797
766discard_unlock: 798discard_unlock:
767 rxrpc_free_skb(skb);
768out_unlock:
769 rcu_read_unlock(); 799 rcu_read_unlock();
800discard:
801 rxrpc_free_skb(skb);
770out: 802out:
771 trace_rxrpc_rx_done(0, 0); 803 trace_rxrpc_rx_done(0, 0);
772 return; 804 return;
773 805
774cant_route_call: 806out_unlock:
775 rcu_read_unlock(); 807 rcu_read_unlock();
808 goto out;
776 809
777 _debug("can't route call"); 810wrong_security:
778 if (sp->hdr.flags & RXRPC_CLIENT_INITIATED && 811 rcu_read_unlock();
779 sp->hdr.type == RXRPC_PACKET_TYPE_DATA) { 812 trace_rxrpc_abort("SEC", sp->hdr.cid, sp->hdr.callNumber, sp->hdr.seq,
780 if (sp->hdr.seq == 1) { 813 RXKADINCONSISTENCY, EBADMSG);
781 _debug("first packet"); 814 skb->priority = RXKADINCONSISTENCY;
782 skb_queue_tail(&local->accept_queue, skb); 815 goto post_abort;
783 rxrpc_queue_work(&local->processor);
784 _leave(" [incoming]");
785 goto out;
786 }
787 skb->priority = RX_INVALID_OPERATION;
788 } else {
789 skb->priority = RX_CALL_DEAD;
790 }
791
792 if (sp->hdr.type != RXRPC_PACKET_TYPE_ABORT) {
793 _debug("reject type %d",sp->hdr.type);
794 goto reject_packet;
795 } else {
796 rxrpc_free_skb(skb);
797 }
798 _leave(" [no call]");
799 return;
800 816
817bad_message_unlock:
818 rcu_read_unlock();
801bad_message: 819bad_message:
820 trace_rxrpc_abort("BAD", sp->hdr.cid, sp->hdr.callNumber, sp->hdr.seq,
821 RX_PROTOCOL_ERROR, EBADMSG);
802 skb->priority = RX_PROTOCOL_ERROR; 822 skb->priority = RX_PROTOCOL_ERROR;
823post_abort:
824 skb->mark = RXRPC_SKB_MARK_LOCAL_ABORT;
803reject_packet: 825reject_packet:
804 trace_rxrpc_rx_done(skb->mark, skb->priority); 826 trace_rxrpc_rx_done(skb->mark, skb->priority);
805 rxrpc_reject_packet(local, skb); 827 rxrpc_reject_packet(local, skb);
diff --git a/net/rxrpc/insecure.c b/net/rxrpc/insecure.c
index a4aba0246731..7d4375e557e6 100644
--- a/net/rxrpc/insecure.c
+++ b/net/rxrpc/insecure.c
@@ -30,14 +30,18 @@ static int none_secure_packet(struct rxrpc_call *call,
30 return 0; 30 return 0;
31} 31}
32 32
33static int none_verify_packet(struct rxrpc_call *call, 33static int none_verify_packet(struct rxrpc_call *call, struct sk_buff *skb,
34 struct sk_buff *skb, 34 unsigned int offset, unsigned int len,
35 rxrpc_seq_t seq, 35 rxrpc_seq_t seq, u16 expected_cksum)
36 u16 expected_cksum)
37{ 36{
38 return 0; 37 return 0;
39} 38}
40 39
40static void none_locate_data(struct rxrpc_call *call, struct sk_buff *skb,
41 unsigned int *_offset, unsigned int *_len)
42{
43}
44
41static int none_respond_to_challenge(struct rxrpc_connection *conn, 45static int none_respond_to_challenge(struct rxrpc_connection *conn,
42 struct sk_buff *skb, 46 struct sk_buff *skb,
43 u32 *_abort_code) 47 u32 *_abort_code)
@@ -79,6 +83,7 @@ const struct rxrpc_security rxrpc_no_security = {
79 .prime_packet_security = none_prime_packet_security, 83 .prime_packet_security = none_prime_packet_security,
80 .secure_packet = none_secure_packet, 84 .secure_packet = none_secure_packet,
81 .verify_packet = none_verify_packet, 85 .verify_packet = none_verify_packet,
86 .locate_data = none_locate_data,
82 .respond_to_challenge = none_respond_to_challenge, 87 .respond_to_challenge = none_respond_to_challenge,
83 .verify_response = none_verify_response, 88 .verify_response = none_verify_response,
84 .clear = none_clear, 89 .clear = none_clear,
diff --git a/net/rxrpc/local_event.c b/net/rxrpc/local_event.c
index bcc6593b4cdb..cdd58e6e9fbd 100644
--- a/net/rxrpc/local_event.c
+++ b/net/rxrpc/local_event.c
@@ -98,7 +98,7 @@ void rxrpc_process_local_events(struct rxrpc_local *local)
98 98
99 switch (sp->hdr.type) { 99 switch (sp->hdr.type) {
100 case RXRPC_PACKET_TYPE_VERSION: 100 case RXRPC_PACKET_TYPE_VERSION:
101 if (skb_copy_bits(skb, 0, &v, 1) < 0) 101 if (skb_copy_bits(skb, sp->offset, &v, 1) < 0)
102 return; 102 return;
103 _proto("Rx VERSION { %02x }", v); 103 _proto("Rx VERSION { %02x }", v);
104 if (v == 0) 104 if (v == 0)
diff --git a/net/rxrpc/local_object.c b/net/rxrpc/local_object.c
index 610916f4ae34..782b9adf67cb 100644
--- a/net/rxrpc/local_object.c
+++ b/net/rxrpc/local_object.c
@@ -77,7 +77,6 @@ static struct rxrpc_local *rxrpc_alloc_local(const struct sockaddr_rxrpc *srx)
77 INIT_WORK(&local->processor, rxrpc_local_processor); 77 INIT_WORK(&local->processor, rxrpc_local_processor);
78 INIT_HLIST_HEAD(&local->services); 78 INIT_HLIST_HEAD(&local->services);
79 init_rwsem(&local->defrag_sem); 79 init_rwsem(&local->defrag_sem);
80 skb_queue_head_init(&local->accept_queue);
81 skb_queue_head_init(&local->reject_queue); 80 skb_queue_head_init(&local->reject_queue);
82 skb_queue_head_init(&local->event_queue); 81 skb_queue_head_init(&local->event_queue);
83 local->client_conns = RB_ROOT; 82 local->client_conns = RB_ROOT;
@@ -308,7 +307,6 @@ static void rxrpc_local_destroyer(struct rxrpc_local *local)
308 /* At this point, there should be no more packets coming in to the 307 /* At this point, there should be no more packets coming in to the
309 * local endpoint. 308 * local endpoint.
310 */ 309 */
311 rxrpc_purge_queue(&local->accept_queue);
312 rxrpc_purge_queue(&local->reject_queue); 310 rxrpc_purge_queue(&local->reject_queue);
313 rxrpc_purge_queue(&local->event_queue); 311 rxrpc_purge_queue(&local->event_queue);
314 312
@@ -332,11 +330,6 @@ static void rxrpc_local_processor(struct work_struct *work)
332 if (atomic_read(&local->usage) == 0) 330 if (atomic_read(&local->usage) == 0)
333 return rxrpc_local_destroyer(local); 331 return rxrpc_local_destroyer(local);
334 332
335 if (!skb_queue_empty(&local->accept_queue)) {
336 rxrpc_accept_incoming_calls(local);
337 again = true;
338 }
339
340 if (!skb_queue_empty(&local->reject_queue)) { 333 if (!skb_queue_empty(&local->reject_queue)) {
341 rxrpc_reject_packets(local); 334 rxrpc_reject_packets(local);
342 again = true; 335 again = true;
diff --git a/net/rxrpc/misc.c b/net/rxrpc/misc.c
index 39e7cc37c392..fd096f742e4b 100644
--- a/net/rxrpc/misc.c
+++ b/net/rxrpc/misc.c
@@ -50,7 +50,7 @@ unsigned int rxrpc_idle_ack_delay = 0.5 * HZ;
50 * limit is hit, we should generate an EXCEEDS_WINDOW ACK and discard further 50 * limit is hit, we should generate an EXCEEDS_WINDOW ACK and discard further
51 * packets. 51 * packets.
52 */ 52 */
53unsigned int rxrpc_rx_window_size = 32; 53unsigned int rxrpc_rx_window_size = RXRPC_RXTX_BUFF_SIZE - 46;
54 54
55/* 55/*
56 * Maximum Rx MTU size. This indicates to the sender the size of jumbo packet 56 * Maximum Rx MTU size. This indicates to the sender the size of jumbo packet
diff --git a/net/rxrpc/output.c b/net/rxrpc/output.c
index 8756d74fd74b..719a4c23f09d 100644
--- a/net/rxrpc/output.c
+++ b/net/rxrpc/output.c
@@ -15,6 +15,8 @@
15#include <linux/gfp.h> 15#include <linux/gfp.h>
16#include <linux/skbuff.h> 16#include <linux/skbuff.h>
17#include <linux/export.h> 17#include <linux/export.h>
18#include <linux/udp.h>
19#include <linux/ip.h>
18#include <net/sock.h> 20#include <net/sock.h>
19#include <net/af_rxrpc.h> 21#include <net/af_rxrpc.h>
20#include "ar-internal.h" 22#include "ar-internal.h"
@@ -38,20 +40,38 @@ struct rxrpc_pkt_buffer {
38static size_t rxrpc_fill_out_ack(struct rxrpc_call *call, 40static size_t rxrpc_fill_out_ack(struct rxrpc_call *call,
39 struct rxrpc_pkt_buffer *pkt) 41 struct rxrpc_pkt_buffer *pkt)
40{ 42{
43 rxrpc_seq_t hard_ack, top, seq;
44 int ix;
41 u32 mtu, jmax; 45 u32 mtu, jmax;
42 u8 *ackp = pkt->acks; 46 u8 *ackp = pkt->acks;
43 47
48 /* Barrier against rxrpc_input_data(). */
49 hard_ack = READ_ONCE(call->rx_hard_ack);
50 top = smp_load_acquire(&call->rx_top);
51
44 pkt->ack.bufferSpace = htons(8); 52 pkt->ack.bufferSpace = htons(8);
45 pkt->ack.maxSkew = htons(0); 53 pkt->ack.maxSkew = htons(call->ackr_skew);
46 pkt->ack.firstPacket = htonl(call->rx_data_eaten + 1); 54 pkt->ack.firstPacket = htonl(hard_ack + 1);
47 pkt->ack.previousPacket = htonl(call->ackr_prev_seq); 55 pkt->ack.previousPacket = htonl(call->ackr_prev_seq);
48 pkt->ack.serial = htonl(call->ackr_serial); 56 pkt->ack.serial = htonl(call->ackr_serial);
49 pkt->ack.reason = RXRPC_ACK_IDLE; 57 pkt->ack.reason = call->ackr_reason;
50 pkt->ack.nAcks = 0; 58 pkt->ack.nAcks = top - hard_ack;
59
60 if (after(top, hard_ack)) {
61 seq = hard_ack + 1;
62 do {
63 ix = seq & RXRPC_RXTX_BUFF_MASK;
64 if (call->rxtx_buffer[ix])
65 *ackp++ = RXRPC_ACK_TYPE_ACK;
66 else
67 *ackp++ = RXRPC_ACK_TYPE_NACK;
68 seq++;
69 } while (before_eq(seq, top));
70 }
51 71
52 mtu = call->peer->if_mtu; 72 mtu = call->conn->params.peer->if_mtu;
53 mtu -= call->peer->hdrsize; 73 mtu -= call->conn->params.peer->hdrsize;
54 jmax = rxrpc_rx_jumbo_max; 74 jmax = (call->nr_jumbo_dup > 3) ? 1 : rxrpc_rx_jumbo_max;
55 pkt->ackinfo.rxMTU = htonl(rxrpc_rx_mtu); 75 pkt->ackinfo.rxMTU = htonl(rxrpc_rx_mtu);
56 pkt->ackinfo.maxMTU = htonl(mtu); 76 pkt->ackinfo.maxMTU = htonl(mtu);
57 pkt->ackinfo.rwind = htonl(rxrpc_rx_window_size); 77 pkt->ackinfo.rwind = htonl(rxrpc_rx_window_size);
@@ -60,11 +80,11 @@ static size_t rxrpc_fill_out_ack(struct rxrpc_call *call,
60 *ackp++ = 0; 80 *ackp++ = 0;
61 *ackp++ = 0; 81 *ackp++ = 0;
62 *ackp++ = 0; 82 *ackp++ = 0;
63 return 3; 83 return top - hard_ack + 3;
64} 84}
65 85
66/* 86/*
67 * Send a final ACK or ABORT call packet. 87 * Send an ACK or ABORT call packet.
68 */ 88 */
69int rxrpc_send_call_packet(struct rxrpc_call *call, u8 type) 89int rxrpc_send_call_packet(struct rxrpc_call *call, u8 type)
70{ 90{
@@ -158,6 +178,19 @@ int rxrpc_send_call_packet(struct rxrpc_call *call, u8 type)
158 ret = kernel_sendmsg(conn->params.local->socket, 178 ret = kernel_sendmsg(conn->params.local->socket,
159 &msg, iov, ioc, len); 179 &msg, iov, ioc, len);
160 180
181 if (ret < 0 && call->state < RXRPC_CALL_COMPLETE) {
182 switch (pkt->whdr.type) {
183 case RXRPC_PACKET_TYPE_ACK:
184 rxrpc_propose_ACK(call, pkt->ack.reason,
185 ntohs(pkt->ack.maxSkew),
186 ntohl(pkt->ack.serial),
187 true, true);
188 break;
189 case RXRPC_PACKET_TYPE_ABORT:
190 break;
191 }
192 }
193
161out: 194out:
162 rxrpc_put_connection(conn); 195 rxrpc_put_connection(conn);
163 kfree(pkt); 196 kfree(pkt);
@@ -233,3 +266,77 @@ send_fragmentable:
233 _leave(" = %d [frag %u]", ret, conn->params.peer->maxdata); 266 _leave(" = %d [frag %u]", ret, conn->params.peer->maxdata);
234 return ret; 267 return ret;
235} 268}
269
270/*
271 * reject packets through the local endpoint
272 */
273void rxrpc_reject_packets(struct rxrpc_local *local)
274{
275 union {
276 struct sockaddr sa;
277 struct sockaddr_in sin;
278 } sa;
279 struct rxrpc_skb_priv *sp;
280 struct rxrpc_wire_header whdr;
281 struct sk_buff *skb;
282 struct msghdr msg;
283 struct kvec iov[2];
284 size_t size;
285 __be32 code;
286
287 _enter("%d", local->debug_id);
288
289 iov[0].iov_base = &whdr;
290 iov[0].iov_len = sizeof(whdr);
291 iov[1].iov_base = &code;
292 iov[1].iov_len = sizeof(code);
293 size = sizeof(whdr) + sizeof(code);
294
295 msg.msg_name = &sa;
296 msg.msg_control = NULL;
297 msg.msg_controllen = 0;
298 msg.msg_flags = 0;
299
300 memset(&sa, 0, sizeof(sa));
301 sa.sa.sa_family = local->srx.transport.family;
302 switch (sa.sa.sa_family) {
303 case AF_INET:
304 msg.msg_namelen = sizeof(sa.sin);
305 break;
306 default:
307 msg.msg_namelen = 0;
308 break;
309 }
310
311 memset(&whdr, 0, sizeof(whdr));
312 whdr.type = RXRPC_PACKET_TYPE_ABORT;
313
314 while ((skb = skb_dequeue(&local->reject_queue))) {
315 rxrpc_see_skb(skb);
316 sp = rxrpc_skb(skb);
317 switch (sa.sa.sa_family) {
318 case AF_INET:
319 sa.sin.sin_port = udp_hdr(skb)->source;
320 sa.sin.sin_addr.s_addr = ip_hdr(skb)->saddr;
321 code = htonl(skb->priority);
322
323 whdr.epoch = htonl(sp->hdr.epoch);
324 whdr.cid = htonl(sp->hdr.cid);
325 whdr.callNumber = htonl(sp->hdr.callNumber);
326 whdr.serviceId = htons(sp->hdr.serviceId);
327 whdr.flags = sp->hdr.flags;
328 whdr.flags ^= RXRPC_CLIENT_INITIATED;
329 whdr.flags &= RXRPC_CLIENT_INITIATED;
330
331 kernel_sendmsg(local->socket, &msg, iov, 2, size);
332 break;
333
334 default:
335 break;
336 }
337
338 rxrpc_free_skb(skb);
339 }
340
341 _leave("");
342}
diff --git a/net/rxrpc/peer_event.c b/net/rxrpc/peer_event.c
index 27b9ecad007e..c8948936c6fc 100644
--- a/net/rxrpc/peer_event.c
+++ b/net/rxrpc/peer_event.c
@@ -129,15 +129,14 @@ void rxrpc_error_report(struct sock *sk)
129 _leave("UDP socket errqueue empty"); 129 _leave("UDP socket errqueue empty");
130 return; 130 return;
131 } 131 }
132 rxrpc_new_skb(skb);
132 serr = SKB_EXT_ERR(skb); 133 serr = SKB_EXT_ERR(skb);
133 if (!skb->len && serr->ee.ee_origin == SO_EE_ORIGIN_TIMESTAMPING) { 134 if (!skb->len && serr->ee.ee_origin == SO_EE_ORIGIN_TIMESTAMPING) {
134 _leave("UDP empty message"); 135 _leave("UDP empty message");
135 kfree_skb(skb); 136 rxrpc_free_skb(skb);
136 return; 137 return;
137 } 138 }
138 139
139 rxrpc_new_skb(skb);
140
141 rcu_read_lock(); 140 rcu_read_lock();
142 peer = rxrpc_lookup_peer_icmp_rcu(local, skb); 141 peer = rxrpc_lookup_peer_icmp_rcu(local, skb);
143 if (peer && !rxrpc_get_peer_maybe(peer)) 142 if (peer && !rxrpc_get_peer_maybe(peer))
@@ -249,7 +248,6 @@ void rxrpc_peer_error_distributor(struct work_struct *work)
249 container_of(work, struct rxrpc_peer, error_distributor); 248 container_of(work, struct rxrpc_peer, error_distributor);
250 struct rxrpc_call *call; 249 struct rxrpc_call *call;
251 enum rxrpc_call_completion compl; 250 enum rxrpc_call_completion compl;
252 bool queue;
253 int error; 251 int error;
254 252
255 _enter(""); 253 _enter("");
@@ -272,15 +270,8 @@ void rxrpc_peer_error_distributor(struct work_struct *work)
272 hlist_del_init(&call->error_link); 270 hlist_del_init(&call->error_link);
273 rxrpc_see_call(call); 271 rxrpc_see_call(call);
274 272
275 queue = false; 273 if (rxrpc_set_call_completion(call, compl, 0, error))
276 write_lock(&call->state_lock); 274 rxrpc_notify_socket(call);
277 if (__rxrpc_set_call_completion(call, compl, 0, error)) {
278 set_bit(RXRPC_CALL_EV_RCVD_ERROR, &call->events);
279 queue = true;
280 }
281 write_unlock(&call->state_lock);
282 if (queue)
283 rxrpc_queue_call(call);
284 } 275 }
285 276
286 spin_unlock_bh(&peer->lock); 277 spin_unlock_bh(&peer->lock);
diff --git a/net/rxrpc/peer_object.c b/net/rxrpc/peer_object.c
index aebc73ac16dc..2efe29a4c232 100644
--- a/net/rxrpc/peer_object.c
+++ b/net/rxrpc/peer_object.c
@@ -199,6 +199,32 @@ struct rxrpc_peer *rxrpc_alloc_peer(struct rxrpc_local *local, gfp_t gfp)
199} 199}
200 200
201/* 201/*
202 * Initialise peer record.
203 */
204static void rxrpc_init_peer(struct rxrpc_peer *peer, unsigned long hash_key)
205{
206 rxrpc_assess_MTU_size(peer);
207 peer->mtu = peer->if_mtu;
208
209 if (peer->srx.transport.family == AF_INET) {
210 peer->hdrsize = sizeof(struct iphdr);
211 switch (peer->srx.transport_type) {
212 case SOCK_DGRAM:
213 peer->hdrsize += sizeof(struct udphdr);
214 break;
215 default:
216 BUG();
217 break;
218 }
219 } else {
220 BUG();
221 }
222
223 peer->hdrsize += sizeof(struct rxrpc_wire_header);
224 peer->maxdata = peer->mtu - peer->hdrsize;
225}
226
227/*
202 * Set up a new peer. 228 * Set up a new peer.
203 */ 229 */
204static struct rxrpc_peer *rxrpc_create_peer(struct rxrpc_local *local, 230static struct rxrpc_peer *rxrpc_create_peer(struct rxrpc_local *local,
@@ -214,29 +240,39 @@ static struct rxrpc_peer *rxrpc_create_peer(struct rxrpc_local *local,
214 if (peer) { 240 if (peer) {
215 peer->hash_key = hash_key; 241 peer->hash_key = hash_key;
216 memcpy(&peer->srx, srx, sizeof(*srx)); 242 memcpy(&peer->srx, srx, sizeof(*srx));
243 rxrpc_init_peer(peer, hash_key);
244 }
217 245
218 rxrpc_assess_MTU_size(peer); 246 _leave(" = %p", peer);
219 peer->mtu = peer->if_mtu; 247 return peer;
220 248}
221 if (srx->transport.family == AF_INET) {
222 peer->hdrsize = sizeof(struct iphdr);
223 switch (srx->transport_type) {
224 case SOCK_DGRAM:
225 peer->hdrsize += sizeof(struct udphdr);
226 break;
227 default:
228 BUG();
229 break;
230 }
231 } else {
232 BUG();
233 }
234 249
235 peer->hdrsize += sizeof(struct rxrpc_wire_header); 250/*
236 peer->maxdata = peer->mtu - peer->hdrsize; 251 * Set up a new incoming peer. The address is prestored in the preallocated
252 * peer.
253 */
254struct rxrpc_peer *rxrpc_lookup_incoming_peer(struct rxrpc_local *local,
255 struct rxrpc_peer *prealloc)
256{
257 struct rxrpc_peer *peer;
258 unsigned long hash_key;
259
260 hash_key = rxrpc_peer_hash_key(local, &prealloc->srx);
261 prealloc->local = local;
262 rxrpc_init_peer(prealloc, hash_key);
263
264 spin_lock(&rxrpc_peer_hash_lock);
265
266 /* Need to check that we aren't racing with someone else */
267 peer = __rxrpc_lookup_peer_rcu(local, &prealloc->srx, hash_key);
268 if (peer && !rxrpc_get_peer_maybe(peer))
269 peer = NULL;
270 if (!peer) {
271 peer = prealloc;
272 hash_add_rcu(rxrpc_peer_hash, &peer->hash_link, hash_key);
237 } 273 }
238 274
239 _leave(" = %p", peer); 275 spin_unlock(&rxrpc_peer_hash_lock);
240 return peer; 276 return peer;
241} 277}
242 278
@@ -272,7 +308,7 @@ struct rxrpc_peer *rxrpc_lookup_peer(struct rxrpc_local *local,
272 return NULL; 308 return NULL;
273 } 309 }
274 310
275 spin_lock(&rxrpc_peer_hash_lock); 311 spin_lock_bh(&rxrpc_peer_hash_lock);
276 312
277 /* Need to check that we aren't racing with someone else */ 313 /* Need to check that we aren't racing with someone else */
278 peer = __rxrpc_lookup_peer_rcu(local, srx, hash_key); 314 peer = __rxrpc_lookup_peer_rcu(local, srx, hash_key);
@@ -282,7 +318,7 @@ struct rxrpc_peer *rxrpc_lookup_peer(struct rxrpc_local *local,
282 hash_add_rcu(rxrpc_peer_hash, 318 hash_add_rcu(rxrpc_peer_hash,
283 &candidate->hash_link, hash_key); 319 &candidate->hash_link, hash_key);
284 320
285 spin_unlock(&rxrpc_peer_hash_lock); 321 spin_unlock_bh(&rxrpc_peer_hash_lock);
286 322
287 if (peer) 323 if (peer)
288 kfree(candidate); 324 kfree(candidate);
@@ -307,9 +343,9 @@ void __rxrpc_put_peer(struct rxrpc_peer *peer)
307{ 343{
308 ASSERT(hlist_empty(&peer->error_targets)); 344 ASSERT(hlist_empty(&peer->error_targets));
309 345
310 spin_lock(&rxrpc_peer_hash_lock); 346 spin_lock_bh(&rxrpc_peer_hash_lock);
311 hash_del_rcu(&peer->hash_link); 347 hash_del_rcu(&peer->hash_link);
312 spin_unlock(&rxrpc_peer_hash_lock); 348 spin_unlock_bh(&rxrpc_peer_hash_lock);
313 349
314 kfree_rcu(peer, rcu); 350 kfree_rcu(peer, rcu);
315} 351}
diff --git a/net/rxrpc/recvmsg.c b/net/rxrpc/recvmsg.c
index 6876ffb3b410..20d0b5c6f81b 100644
--- a/net/rxrpc/recvmsg.c
+++ b/net/rxrpc/recvmsg.c
@@ -19,319 +19,479 @@
19#include "ar-internal.h" 19#include "ar-internal.h"
20 20
21/* 21/*
22 * receive a message from an RxRPC socket 22 * Post a call for attention by the socket or kernel service. Further
23 * - we need to be careful about two or more threads calling recvmsg 23 * notifications are suppressed by putting recvmsg_link on a dummy queue.
24 * simultaneously
25 */ 24 */
26int rxrpc_recvmsg(struct socket *sock, struct msghdr *msg, size_t len, 25void rxrpc_notify_socket(struct rxrpc_call *call)
27 int flags)
28{ 26{
29 struct rxrpc_skb_priv *sp; 27 struct rxrpc_sock *rx;
30 struct rxrpc_call *call = NULL, *continue_call = NULL; 28 struct sock *sk;
31 struct rxrpc_sock *rx = rxrpc_sk(sock->sk);
32 struct sk_buff *skb;
33 long timeo;
34 int copy, ret, ullen, offset, copied = 0;
35 u32 abort_code;
36 29
37 DEFINE_WAIT(wait); 30 _enter("%d", call->debug_id);
38 31
39 _enter(",,,%zu,%d", len, flags); 32 if (!list_empty(&call->recvmsg_link))
33 return;
34
35 rcu_read_lock();
36
37 rx = rcu_dereference(call->socket);
38 sk = &rx->sk;
39 if (rx && sk->sk_state < RXRPC_CLOSE) {
40 if (call->notify_rx) {
41 call->notify_rx(sk, call, call->user_call_ID);
42 } else {
43 write_lock_bh(&rx->recvmsg_lock);
44 if (list_empty(&call->recvmsg_link)) {
45 rxrpc_get_call(call, rxrpc_call_got);
46 list_add_tail(&call->recvmsg_link, &rx->recvmsg_q);
47 }
48 write_unlock_bh(&rx->recvmsg_lock);
40 49
41 if (flags & (MSG_OOB | MSG_TRUNC)) 50 if (!sock_flag(sk, SOCK_DEAD)) {
42 return -EOPNOTSUPP; 51 _debug("call %ps", sk->sk_data_ready);
52 sk->sk_data_ready(sk);
53 }
54 }
55 }
43 56
44 ullen = msg->msg_flags & MSG_CMSG_COMPAT ? 4 : sizeof(unsigned long); 57 rcu_read_unlock();
58 _leave("");
59}
45 60
46 timeo = sock_rcvtimeo(&rx->sk, flags & MSG_DONTWAIT); 61/*
47 msg->msg_flags |= MSG_MORE; 62 * Pass a call terminating message to userspace.
63 */
64static int rxrpc_recvmsg_term(struct rxrpc_call *call, struct msghdr *msg)
65{
66 u32 tmp = 0;
67 int ret;
48 68
49 lock_sock(&rx->sk); 69 switch (call->completion) {
70 case RXRPC_CALL_SUCCEEDED:
71 ret = 0;
72 if (rxrpc_is_service_call(call))
73 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_ACK, 0, &tmp);
74 break;
75 case RXRPC_CALL_REMOTELY_ABORTED:
76 tmp = call->abort_code;
77 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_ABORT, 4, &tmp);
78 break;
79 case RXRPC_CALL_LOCALLY_ABORTED:
80 tmp = call->abort_code;
81 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_ABORT, 4, &tmp);
82 break;
83 case RXRPC_CALL_NETWORK_ERROR:
84 tmp = call->error;
85 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_NET_ERROR, 4, &tmp);
86 break;
87 case RXRPC_CALL_LOCAL_ERROR:
88 tmp = call->error;
89 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_LOCAL_ERROR, 4, &tmp);
90 break;
91 default:
92 pr_err("Invalid terminal call state %u\n", call->state);
93 BUG();
94 break;
95 }
50 96
51 for (;;) { 97 return ret;
52 /* return immediately if a client socket has no outstanding 98}
53 * calls */
54 if (RB_EMPTY_ROOT(&rx->calls)) {
55 if (copied)
56 goto out;
57 if (rx->sk.sk_state != RXRPC_SERVER_LISTENING) {
58 release_sock(&rx->sk);
59 if (continue_call)
60 rxrpc_put_call(continue_call,
61 rxrpc_call_put);
62 return -ENODATA;
63 }
64 }
65 99
66 /* get the next message on the Rx queue */ 100/*
67 skb = skb_peek(&rx->sk.sk_receive_queue); 101 * Pass back notification of a new call. The call is added to the
68 if (!skb) { 102 * to-be-accepted list. This means that the next call to be accepted might not
69 /* nothing remains on the queue */ 103 * be the last call seen awaiting acceptance, but unless we leave this on the
70 if (copied && 104 * front of the queue and block all other messages until someone gives us a
71 (flags & MSG_PEEK || timeo == 0)) 105 * user_ID for it, there's not a lot we can do.
72 goto out; 106 */
107static int rxrpc_recvmsg_new_call(struct rxrpc_sock *rx,
108 struct rxrpc_call *call,
109 struct msghdr *msg, int flags)
110{
111 int tmp = 0, ret;
73 112
74 /* wait for a message to turn up */ 113 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_NEW_CALL, 0, &tmp);
75 release_sock(&rx->sk);
76 prepare_to_wait_exclusive(sk_sleep(&rx->sk), &wait,
77 TASK_INTERRUPTIBLE);
78 ret = sock_error(&rx->sk);
79 if (ret)
80 goto wait_error;
81
82 if (skb_queue_empty(&rx->sk.sk_receive_queue)) {
83 if (signal_pending(current))
84 goto wait_interrupted;
85 timeo = schedule_timeout(timeo);
86 }
87 finish_wait(sk_sleep(&rx->sk), &wait);
88 lock_sock(&rx->sk);
89 continue;
90 }
91 114
92 peek_next_packet: 115 if (ret == 0 && !(flags & MSG_PEEK)) {
93 rxrpc_see_skb(skb); 116 _debug("to be accepted");
94 sp = rxrpc_skb(skb); 117 write_lock_bh(&rx->recvmsg_lock);
95 call = sp->call; 118 list_del_init(&call->recvmsg_link);
96 ASSERT(call != NULL); 119 write_unlock_bh(&rx->recvmsg_lock);
97 rxrpc_see_call(call);
98
99 _debug("next pkt %s", rxrpc_pkts[sp->hdr.type]);
100
101 /* make sure we wait for the state to be updated in this call */
102 spin_lock_bh(&call->lock);
103 spin_unlock_bh(&call->lock);
104
105 if (test_bit(RXRPC_CALL_RELEASED, &call->flags)) {
106 _debug("packet from released call");
107 if (skb_dequeue(&rx->sk.sk_receive_queue) != skb)
108 BUG();
109 rxrpc_free_skb(skb);
110 continue;
111 }
112 120
113 /* determine whether to continue last data receive */ 121 write_lock(&rx->call_lock);
114 if (continue_call) { 122 list_add_tail(&call->accept_link, &rx->to_be_accepted);
115 _debug("maybe cont"); 123 write_unlock(&rx->call_lock);
116 if (call != continue_call || 124 }
117 skb->mark != RXRPC_SKB_MARK_DATA) {
118 release_sock(&rx->sk);
119 rxrpc_put_call(continue_call, rxrpc_call_put);
120 _leave(" = %d [noncont]", copied);
121 return copied;
122 }
123 }
124 125
125 rxrpc_get_call(call, rxrpc_call_got); 126 return ret;
127}
126 128
127 /* copy the peer address and timestamp */ 129/*
128 if (!continue_call) { 130 * End the packet reception phase.
129 if (msg->msg_name) { 131 */
130 size_t len = 132static void rxrpc_end_rx_phase(struct rxrpc_call *call)
131 sizeof(call->conn->params.peer->srx); 133{
132 memcpy(msg->msg_name, 134 _enter("%d,%s", call->debug_id, rxrpc_call_states[call->state]);
133 &call->conn->params.peer->srx, len);
134 msg->msg_namelen = len;
135 }
136 sock_recv_timestamp(msg, &rx->sk, skb);
137 }
138 135
139 /* receive the message */ 136 if (call->state == RXRPC_CALL_CLIENT_RECV_REPLY) {
140 if (skb->mark != RXRPC_SKB_MARK_DATA) 137 rxrpc_propose_ACK(call, RXRPC_ACK_IDLE, 0, 0, true, false);
141 goto receive_non_data_message; 138 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ACK);
139 } else {
140 rxrpc_propose_ACK(call, RXRPC_ACK_IDLE, 0, 0, false, false);
141 }
142 142
143 _debug("recvmsg DATA #%u { %d, %d }", 143 write_lock_bh(&call->state_lock);
144 sp->hdr.seq, skb->len, sp->offset);
145 144
146 if (!continue_call) { 145 switch (call->state) {
147 /* only set the control data once per recvmsg() */ 146 case RXRPC_CALL_CLIENT_RECV_REPLY:
148 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_USER_CALL_ID, 147 __rxrpc_call_completed(call);
149 ullen, &call->user_call_ID); 148 break;
150 if (ret < 0)
151 goto copy_error;
152 ASSERT(test_bit(RXRPC_CALL_HAS_USERID, &call->flags));
153 }
154 149
155 ASSERTCMP(sp->hdr.seq, >=, call->rx_data_recv); 150 case RXRPC_CALL_SERVER_RECV_REQUEST:
156 ASSERTCMP(sp->hdr.seq, <=, call->rx_data_recv + 1); 151 call->state = RXRPC_CALL_SERVER_ACK_REQUEST;
157 call->rx_data_recv = sp->hdr.seq; 152 break;
153 default:
154 break;
155 }
158 156
159 ASSERTCMP(sp->hdr.seq, >, call->rx_data_eaten); 157 write_unlock_bh(&call->state_lock);
158}
160 159
161 offset = sp->offset; 160/*
162 copy = skb->len - offset; 161 * Discard a packet we've used up and advance the Rx window by one.
163 if (copy > len - copied) 162 */
164 copy = len - copied; 163static void rxrpc_rotate_rx_window(struct rxrpc_call *call)
164{
165 struct sk_buff *skb;
166 rxrpc_seq_t hard_ack, top;
167 int ix;
168
169 _enter("%d", call->debug_id);
170
171 hard_ack = call->rx_hard_ack;
172 top = smp_load_acquire(&call->rx_top);
173 ASSERT(before(hard_ack, top));
174
175 hard_ack++;
176 ix = hard_ack & RXRPC_RXTX_BUFF_MASK;
177 skb = call->rxtx_buffer[ix];
178 rxrpc_see_skb(skb);
179 call->rxtx_buffer[ix] = NULL;
180 call->rxtx_annotations[ix] = 0;
181 /* Barrier against rxrpc_input_data(). */
182 smp_store_release(&call->rx_hard_ack, hard_ack);
165 183
166 ret = skb_copy_datagram_msg(skb, offset, msg, copy); 184 rxrpc_free_skb(skb);
167 185
186 _debug("%u,%u,%lx", hard_ack, top, call->flags);
187 if (hard_ack == top && test_bit(RXRPC_CALL_RX_LAST, &call->flags))
188 rxrpc_end_rx_phase(call);
189}
190
191/*
192 * Decrypt and verify a (sub)packet. The packet's length may be changed due to
193 * padding, but if this is the case, the packet length will be resident in the
194 * socket buffer. Note that we can't modify the master skb info as the skb may
195 * be the home to multiple subpackets.
196 */
197static int rxrpc_verify_packet(struct rxrpc_call *call, struct sk_buff *skb,
198 u8 annotation,
199 unsigned int offset, unsigned int len)
200{
201 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
202 rxrpc_seq_t seq = sp->hdr.seq;
203 u16 cksum = sp->hdr.cksum;
204
205 _enter("");
206
207 /* For all but the head jumbo subpacket, the security checksum is in a
208 * jumbo header immediately prior to the data.
209 */
210 if ((annotation & RXRPC_RX_ANNO_JUMBO) > 1) {
211 __be16 tmp;
212 if (skb_copy_bits(skb, offset - 2, &tmp, 2) < 0)
213 BUG();
214 cksum = ntohs(tmp);
215 seq += (annotation & RXRPC_RX_ANNO_JUMBO) - 1;
216 }
217
218 return call->conn->security->verify_packet(call, skb, offset, len,
219 seq, cksum);
220}
221
222/*
223 * Locate the data within a packet. This is complicated by:
224 *
225 * (1) An skb may contain a jumbo packet - so we have to find the appropriate
226 * subpacket.
227 *
228 * (2) The (sub)packets may be encrypted and, if so, the encrypted portion
229 * contains an extra header which includes the true length of the data,
230 * excluding any encrypted padding.
231 */
232static int rxrpc_locate_data(struct rxrpc_call *call, struct sk_buff *skb,
233 u8 *_annotation,
234 unsigned int *_offset, unsigned int *_len)
235{
236 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
237 unsigned int offset = *_offset;
238 unsigned int len = *_len;
239 int ret;
240 u8 annotation = *_annotation;
241
242 if (offset > 0)
243 return 0;
244
245 /* Locate the subpacket */
246 offset = sp->offset;
247 len = skb->len - sp->offset;
248 if ((annotation & RXRPC_RX_ANNO_JUMBO) > 0) {
249 offset += (((annotation & RXRPC_RX_ANNO_JUMBO) - 1) *
250 RXRPC_JUMBO_SUBPKTLEN);
251 len = (annotation & RXRPC_RX_ANNO_JLAST) ?
252 skb->len - offset : RXRPC_JUMBO_SUBPKTLEN;
253 }
254
255 if (!(annotation & RXRPC_RX_ANNO_VERIFIED)) {
256 ret = rxrpc_verify_packet(call, skb, annotation, offset, len);
168 if (ret < 0) 257 if (ret < 0)
169 goto copy_error; 258 return ret;
259 *_annotation |= RXRPC_RX_ANNO_VERIFIED;
260 }
170 261
171 /* handle piecemeal consumption of data packets */ 262 *_offset = offset;
172 _debug("copied %d+%d", copy, copied); 263 *_len = len;
264 call->conn->security->locate_data(call, skb, _offset, _len);
265 return 0;
266}
173 267
174 offset += copy; 268/*
175 copied += copy; 269 * Deliver messages to a call. This keeps processing packets until the buffer
270 * is filled and we find either more DATA (returns 0) or the end of the DATA
271 * (returns 1). If more packets are required, it returns -EAGAIN.
272 */
273static int rxrpc_recvmsg_data(struct socket *sock, struct rxrpc_call *call,
274 struct msghdr *msg, struct iov_iter *iter,
275 size_t len, int flags, size_t *_offset)
276{
277 struct rxrpc_skb_priv *sp;
278 struct sk_buff *skb;
279 rxrpc_seq_t hard_ack, top, seq;
280 size_t remain;
281 bool last;
282 unsigned int rx_pkt_offset, rx_pkt_len;
283 int ix, copy, ret = 0;
284
285 _enter("");
286
287 rx_pkt_offset = call->rx_pkt_offset;
288 rx_pkt_len = call->rx_pkt_len;
289
290 /* Barriers against rxrpc_input_data(). */
291 hard_ack = call->rx_hard_ack;
292 top = smp_load_acquire(&call->rx_top);
293 for (seq = hard_ack + 1; before_eq(seq, top); seq++) {
294 ix = seq & RXRPC_RXTX_BUFF_MASK;
295 skb = call->rxtx_buffer[ix];
296 if (!skb)
297 break;
298 smp_rmb();
299 rxrpc_see_skb(skb);
300 sp = rxrpc_skb(skb);
176 301
177 if (!(flags & MSG_PEEK)) 302 if (msg)
178 sp->offset = offset; 303 sock_recv_timestamp(msg, sock->sk, skb);
304
305 ret = rxrpc_locate_data(call, skb, &call->rxtx_annotations[ix],
306 &rx_pkt_offset, &rx_pkt_len);
307 _debug("recvmsg %x DATA #%u { %d, %d }",
308 sp->hdr.callNumber, seq, rx_pkt_offset, rx_pkt_len);
309
310 /* We have to handle short, empty and used-up DATA packets. */
311 remain = len - *_offset;
312 copy = rx_pkt_len;
313 if (copy > remain)
314 copy = remain;
315 if (copy > 0) {
316 ret = skb_copy_datagram_iter(skb, rx_pkt_offset, iter,
317 copy);
318 if (ret < 0)
319 goto out;
320
321 /* handle piecemeal consumption of data packets */
322 _debug("copied %d @%zu", copy, *_offset);
323
324 rx_pkt_offset += copy;
325 rx_pkt_len -= copy;
326 *_offset += copy;
327 }
179 328
180 if (sp->offset < skb->len) { 329 if (rx_pkt_len > 0) {
181 _debug("buffer full"); 330 _debug("buffer full");
182 ASSERTCMP(copied, ==, len); 331 ASSERTCMP(*_offset, ==, len);
183 break; 332 break;
184 } 333 }
185 334
186 /* we transferred the whole data packet */ 335 /* The whole packet has been transferred. */
336 last = sp->hdr.flags & RXRPC_LAST_PACKET;
187 if (!(flags & MSG_PEEK)) 337 if (!(flags & MSG_PEEK))
188 rxrpc_kernel_data_consumed(call, skb); 338 rxrpc_rotate_rx_window(call);
189 339 rx_pkt_offset = 0;
190 if (sp->hdr.flags & RXRPC_LAST_PACKET) { 340 rx_pkt_len = 0;
191 _debug("last");
192 if (rxrpc_conn_is_client(call->conn)) {
193 /* last byte of reply received */
194 ret = copied;
195 goto terminal_message;
196 }
197 341
198 /* last bit of request received */ 342 ASSERTIFCMP(last, seq, ==, top);
199 if (!(flags & MSG_PEEK)) { 343 }
200 _debug("eat packet");
201 if (skb_dequeue(&rx->sk.sk_receive_queue) !=
202 skb)
203 BUG();
204 rxrpc_free_skb(skb);
205 }
206 msg->msg_flags &= ~MSG_MORE;
207 break;
208 }
209 344
210 /* move on to the next data message */ 345 if (after(seq, top)) {
211 _debug("next"); 346 ret = -EAGAIN;
212 if (!continue_call) 347 if (test_bit(RXRPC_CALL_RX_LAST, &call->flags))
213 continue_call = sp->call; 348 ret = 1;
214 else 349 }
215 rxrpc_put_call(call, rxrpc_call_put); 350out:
216 call = NULL; 351 if (!(flags & MSG_PEEK)) {
217 352 call->rx_pkt_offset = rx_pkt_offset;
218 if (flags & MSG_PEEK) { 353 call->rx_pkt_len = rx_pkt_len;
219 _debug("peek next"); 354 }
220 skb = skb->next; 355 _leave(" = %d [%u/%u]", ret, seq, top);
221 if (skb == (struct sk_buff *) &rx->sk.sk_receive_queue) 356 return ret;
222 break; 357}
223 goto peek_next_packet;
224 }
225 358
226 _debug("eat packet"); 359/*
227 if (skb_dequeue(&rx->sk.sk_receive_queue) != skb) 360 * Receive a message from an RxRPC socket
228 BUG(); 361 * - we need to be careful about two or more threads calling recvmsg
229 rxrpc_free_skb(skb); 362 * simultaneously
363 */
364int rxrpc_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
365 int flags)
366{
367 struct rxrpc_call *call;
368 struct rxrpc_sock *rx = rxrpc_sk(sock->sk);
369 struct list_head *l;
370 size_t copied = 0;
371 long timeo;
372 int ret;
373
374 DEFINE_WAIT(wait);
375
376 _enter(",,,%zu,%d", len, flags);
377
378 if (flags & (MSG_OOB | MSG_TRUNC))
379 return -EOPNOTSUPP;
380
381 timeo = sock_rcvtimeo(&rx->sk, flags & MSG_DONTWAIT);
382
383try_again:
384 lock_sock(&rx->sk);
385
386 /* Return immediately if a client socket has no outstanding calls */
387 if (RB_EMPTY_ROOT(&rx->calls) &&
388 list_empty(&rx->recvmsg_q) &&
389 rx->sk.sk_state != RXRPC_SERVER_LISTENING) {
390 release_sock(&rx->sk);
391 return -ENODATA;
230 } 392 }
231 393
232 /* end of non-terminal data packet reception for the moment */ 394 if (list_empty(&rx->recvmsg_q)) {
233 _debug("end rcv data"); 395 ret = -EWOULDBLOCK;
234out: 396 if (timeo == 0)
235 release_sock(&rx->sk); 397 goto error_no_call;
236 if (call) 398
237 rxrpc_put_call(call, rxrpc_call_put); 399 release_sock(&rx->sk);
238 if (continue_call) 400
239 rxrpc_put_call(continue_call, rxrpc_call_put); 401 /* Wait for something to happen */
240 _leave(" = %d [data]", copied); 402 prepare_to_wait_exclusive(sk_sleep(&rx->sk), &wait,
241 return copied; 403 TASK_INTERRUPTIBLE);
242 404 ret = sock_error(&rx->sk);
243 /* handle non-DATA messages such as aborts, incoming connections and 405 if (ret)
244 * final ACKs */ 406 goto wait_error;
245receive_non_data_message: 407
246 _debug("non-data"); 408 if (list_empty(&rx->recvmsg_q)) {
247 409 if (signal_pending(current))
248 if (skb->mark == RXRPC_SKB_MARK_NEW_CALL) { 410 goto wait_interrupted;
249 _debug("RECV NEW CALL"); 411 timeo = schedule_timeout(timeo);
250 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_NEW_CALL, 0, &abort_code);
251 if (ret < 0)
252 goto copy_error;
253 if (!(flags & MSG_PEEK)) {
254 if (skb_dequeue(&rx->sk.sk_receive_queue) != skb)
255 BUG();
256 rxrpc_free_skb(skb);
257 } 412 }
258 goto out; 413 finish_wait(sk_sleep(&rx->sk), &wait);
414 goto try_again;
259 } 415 }
260 416
261 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_USER_CALL_ID, 417 /* Find the next call and dequeue it if we're not just peeking. If we
262 ullen, &call->user_call_ID); 418 * do dequeue it, that comes with a ref that we will need to release.
263 if (ret < 0) 419 */
264 goto copy_error; 420 write_lock_bh(&rx->recvmsg_lock);
265 ASSERT(test_bit(RXRPC_CALL_HAS_USERID, &call->flags)); 421 l = rx->recvmsg_q.next;
422 call = list_entry(l, struct rxrpc_call, recvmsg_link);
423 if (!(flags & MSG_PEEK))
424 list_del_init(&call->recvmsg_link);
425 else
426 rxrpc_get_call(call, rxrpc_call_got);
427 write_unlock_bh(&rx->recvmsg_lock);
266 428
267 switch (skb->mark) { 429 _debug("recvmsg call %p", call);
268 case RXRPC_SKB_MARK_DATA: 430
431 if (test_bit(RXRPC_CALL_RELEASED, &call->flags))
269 BUG(); 432 BUG();
270 case RXRPC_SKB_MARK_FINAL_ACK: 433
271 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_ACK, 0, &abort_code); 434 if (test_bit(RXRPC_CALL_HAS_USERID, &call->flags)) {
272 break; 435 if (flags & MSG_CMSG_COMPAT) {
273 case RXRPC_SKB_MARK_BUSY: 436 unsigned int id32 = call->user_call_ID;
274 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_BUSY, 0, &abort_code); 437
275 break; 438 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_USER_CALL_ID,
276 case RXRPC_SKB_MARK_REMOTE_ABORT: 439 sizeof(unsigned int), &id32);
277 abort_code = call->abort_code; 440 } else {
278 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_ABORT, 4, &abort_code); 441 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_USER_CALL_ID,
279 break; 442 sizeof(unsigned long),
280 case RXRPC_SKB_MARK_LOCAL_ABORT: 443 &call->user_call_ID);
281 abort_code = call->abort_code;
282 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_ABORT, 4, &abort_code);
283 if (call->error) {
284 abort_code = call->error;
285 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_LOCAL_ERROR, 4,
286 &abort_code);
287 } 444 }
445 if (ret < 0)
446 goto error;
447 }
448
449 if (msg->msg_name) {
450 size_t len = sizeof(call->conn->params.peer->srx);
451 memcpy(msg->msg_name, &call->conn->params.peer->srx, len);
452 msg->msg_namelen = len;
453 }
454
455 switch (call->state) {
456 case RXRPC_CALL_SERVER_ACCEPTING:
457 ret = rxrpc_recvmsg_new_call(rx, call, msg, flags);
288 break; 458 break;
289 case RXRPC_SKB_MARK_NET_ERROR: 459 case RXRPC_CALL_CLIENT_RECV_REPLY:
290 _debug("RECV NET ERROR %d", sp->error); 460 case RXRPC_CALL_SERVER_RECV_REQUEST:
291 abort_code = sp->error; 461 case RXRPC_CALL_SERVER_ACK_REQUEST:
292 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_NET_ERROR, 4, &abort_code); 462 ret = rxrpc_recvmsg_data(sock, call, msg, &msg->msg_iter, len,
293 break; 463 flags, &copied);
294 case RXRPC_SKB_MARK_LOCAL_ERROR: 464 if (ret == -EAGAIN)
295 _debug("RECV LOCAL ERROR %d", sp->error); 465 ret = 0;
296 abort_code = sp->error;
297 ret = put_cmsg(msg, SOL_RXRPC, RXRPC_LOCAL_ERROR, 4,
298 &abort_code);
299 break; 466 break;
300 default: 467 default:
301 pr_err("Unknown packet mark %u\n", skb->mark); 468 ret = 0;
302 BUG();
303 break; 469 break;
304 } 470 }
305 471
306 if (ret < 0) 472 if (ret < 0)
307 goto copy_error; 473 goto error;
308
309terminal_message:
310 _debug("terminal");
311 msg->msg_flags &= ~MSG_MORE;
312 msg->msg_flags |= MSG_EOR;
313 474
314 if (!(flags & MSG_PEEK)) { 475 if (call->state == RXRPC_CALL_COMPLETE) {
315 _net("free terminal skb %p", skb); 476 ret = rxrpc_recvmsg_term(call, msg);
316 if (skb_dequeue(&rx->sk.sk_receive_queue) != skb) 477 if (ret < 0)
317 BUG(); 478 goto error;
318 rxrpc_free_skb(skb); 479 if (!(flags & MSG_PEEK))
319 rxrpc_release_call(rx, call); 480 rxrpc_release_call(rx, call);
481 msg->msg_flags |= MSG_EOR;
482 ret = 1;
320 } 483 }
321 484
322 release_sock(&rx->sk); 485 if (ret == 0)
323 rxrpc_put_call(call, rxrpc_call_put); 486 msg->msg_flags |= MSG_MORE;
324 if (continue_call) 487 else
325 rxrpc_put_call(continue_call, rxrpc_call_put); 488 msg->msg_flags &= ~MSG_MORE;
326 _leave(" = %d", ret); 489 ret = copied;
327 return ret;
328 490
329copy_error: 491error:
330 _debug("copy error");
331 release_sock(&rx->sk);
332 rxrpc_put_call(call, rxrpc_call_put); 492 rxrpc_put_call(call, rxrpc_call_put);
333 if (continue_call) 493error_no_call:
334 rxrpc_put_call(continue_call, rxrpc_call_put); 494 release_sock(&rx->sk);
335 _leave(" = %d", ret); 495 _leave(" = %d", ret);
336 return ret; 496 return ret;
337 497
@@ -339,85 +499,8 @@ wait_interrupted:
339 ret = sock_intr_errno(timeo); 499 ret = sock_intr_errno(timeo);
340wait_error: 500wait_error:
341 finish_wait(sk_sleep(&rx->sk), &wait); 501 finish_wait(sk_sleep(&rx->sk), &wait);
342 if (continue_call) 502 release_sock(&rx->sk);
343 rxrpc_put_call(continue_call, rxrpc_call_put); 503 _leave(" = %d [wait]", ret);
344 if (copied)
345 copied = ret;
346 _leave(" = %d [waitfail %d]", copied, ret);
347 return copied;
348
349}
350
351/*
352 * Deliver messages to a call. This keeps processing packets until the buffer
353 * is filled and we find either more DATA (returns 0) or the end of the DATA
354 * (returns 1). If more packets are required, it returns -EAGAIN.
355 *
356 * TODO: Note that this is hacked in at the moment and will be replaced.
357 */
358static int temp_deliver_data(struct socket *sock, struct rxrpc_call *call,
359 struct iov_iter *iter, size_t size,
360 size_t *_offset)
361{
362 struct rxrpc_skb_priv *sp;
363 struct sk_buff *skb;
364 size_t remain;
365 int ret, copy;
366
367 _enter("%d", call->debug_id);
368
369next:
370 local_bh_disable();
371 skb = skb_dequeue(&call->knlrecv_queue);
372 local_bh_enable();
373 if (!skb) {
374 if (test_bit(RXRPC_CALL_RX_NO_MORE, &call->flags))
375 return 1;
376 _leave(" = -EAGAIN [empty]");
377 return -EAGAIN;
378 }
379
380 sp = rxrpc_skb(skb);
381 _debug("dequeued %p %u/%zu", skb, sp->offset, size);
382
383 switch (skb->mark) {
384 case RXRPC_SKB_MARK_DATA:
385 remain = size - *_offset;
386 if (remain > 0) {
387 copy = skb->len - sp->offset;
388 if (copy > remain)
389 copy = remain;
390 ret = skb_copy_datagram_iter(skb, sp->offset, iter,
391 copy);
392 if (ret < 0)
393 goto requeue_and_leave;
394
395 /* handle piecemeal consumption of data packets */
396 sp->offset += copy;
397 *_offset += copy;
398 }
399
400 if (sp->offset < skb->len)
401 goto partially_used_skb;
402
403 /* We consumed the whole packet */
404 ASSERTCMP(sp->offset, ==, skb->len);
405 if (sp->hdr.flags & RXRPC_LAST_PACKET)
406 set_bit(RXRPC_CALL_RX_NO_MORE, &call->flags);
407 rxrpc_kernel_data_consumed(call, skb);
408 rxrpc_free_skb(skb);
409 goto next;
410
411 default:
412 rxrpc_free_skb(skb);
413 goto next;
414 }
415
416partially_used_skb:
417 ASSERTCMP(*_offset, ==, size);
418 ret = 0;
419requeue_and_leave:
420 skb_queue_head(&call->knlrecv_queue, skb);
421 return ret; 504 return ret;
422} 505}
423 506
@@ -453,8 +536,9 @@ int rxrpc_kernel_recv_data(struct socket *sock, struct rxrpc_call *call,
453 struct kvec iov; 536 struct kvec iov;
454 int ret; 537 int ret;
455 538
456 _enter("{%d,%s},%zu,%d", 539 _enter("{%d,%s},%zu/%zu,%d",
457 call->debug_id, rxrpc_call_states[call->state], size, want_more); 540 call->debug_id, rxrpc_call_states[call->state],
541 *_offset, size, want_more);
458 542
459 ASSERTCMP(*_offset, <=, size); 543 ASSERTCMP(*_offset, <=, size);
460 ASSERTCMP(call->state, !=, RXRPC_CALL_SERVER_ACCEPTING); 544 ASSERTCMP(call->state, !=, RXRPC_CALL_SERVER_ACCEPTING);
@@ -469,7 +553,8 @@ int rxrpc_kernel_recv_data(struct socket *sock, struct rxrpc_call *call,
469 case RXRPC_CALL_CLIENT_RECV_REPLY: 553 case RXRPC_CALL_CLIENT_RECV_REPLY:
470 case RXRPC_CALL_SERVER_RECV_REQUEST: 554 case RXRPC_CALL_SERVER_RECV_REQUEST:
471 case RXRPC_CALL_SERVER_ACK_REQUEST: 555 case RXRPC_CALL_SERVER_ACK_REQUEST:
472 ret = temp_deliver_data(sock, call, &iter, size, _offset); 556 ret = rxrpc_recvmsg_data(sock, call, NULL, &iter, size, 0,
557 _offset);
473 if (ret < 0) 558 if (ret < 0)
474 goto out; 559 goto out;
475 560
@@ -494,7 +579,6 @@ int rxrpc_kernel_recv_data(struct socket *sock, struct rxrpc_call *call,
494 goto call_complete; 579 goto call_complete;
495 580
496 default: 581 default:
497 *_offset = 0;
498 ret = -EINPROGRESS; 582 ret = -EINPROGRESS;
499 goto out; 583 goto out;
500 } 584 }
diff --git a/net/rxrpc/rxkad.c b/net/rxrpc/rxkad.c
index 3777432df10b..ae392558829d 100644
--- a/net/rxrpc/rxkad.c
+++ b/net/rxrpc/rxkad.c
@@ -317,6 +317,7 @@ static int rxkad_secure_packet(struct rxrpc_call *call,
317 * decrypt partial encryption on a packet (level 1 security) 317 * decrypt partial encryption on a packet (level 1 security)
318 */ 318 */
319static int rxkad_verify_packet_1(struct rxrpc_call *call, struct sk_buff *skb, 319static int rxkad_verify_packet_1(struct rxrpc_call *call, struct sk_buff *skb,
320 unsigned int offset, unsigned int len,
320 rxrpc_seq_t seq) 321 rxrpc_seq_t seq)
321{ 322{
322 struct rxkad_level1_hdr sechdr; 323 struct rxkad_level1_hdr sechdr;
@@ -330,18 +331,20 @@ static int rxkad_verify_packet_1(struct rxrpc_call *call, struct sk_buff *skb,
330 331
331 _enter(""); 332 _enter("");
332 333
333 if (skb->len < 8) { 334 if (len < 8) {
334 rxrpc_abort_call("V1H", call, seq, RXKADSEALEDINCON, EPROTO); 335 rxrpc_abort_call("V1H", call, seq, RXKADSEALEDINCON, EPROTO);
335 goto protocol_error; 336 goto protocol_error;
336 } 337 }
337 338
338 /* we want to decrypt the skbuff in-place */ 339 /* Decrypt the skbuff in-place. TODO: We really want to decrypt
340 * directly into the target buffer.
341 */
339 nsg = skb_cow_data(skb, 0, &trailer); 342 nsg = skb_cow_data(skb, 0, &trailer);
340 if (nsg < 0 || nsg > 16) 343 if (nsg < 0 || nsg > 16)
341 goto nomem; 344 goto nomem;
342 345
343 sg_init_table(sg, nsg); 346 sg_init_table(sg, nsg);
344 skb_to_sgvec(skb, sg, 0, 8); 347 skb_to_sgvec(skb, sg, offset, 8);
345 348
346 /* start the decryption afresh */ 349 /* start the decryption afresh */
347 memset(&iv, 0, sizeof(iv)); 350 memset(&iv, 0, sizeof(iv));
@@ -353,12 +356,12 @@ static int rxkad_verify_packet_1(struct rxrpc_call *call, struct sk_buff *skb,
353 skcipher_request_zero(req); 356 skcipher_request_zero(req);
354 357
355 /* Extract the decrypted packet length */ 358 /* Extract the decrypted packet length */
356 if (skb_copy_bits(skb, 0, &sechdr, sizeof(sechdr)) < 0) { 359 if (skb_copy_bits(skb, offset, &sechdr, sizeof(sechdr)) < 0) {
357 rxrpc_abort_call("XV1", call, seq, RXKADDATALEN, EPROTO); 360 rxrpc_abort_call("XV1", call, seq, RXKADDATALEN, EPROTO);
358 goto protocol_error; 361 goto protocol_error;
359 } 362 }
360 if (!skb_pull(skb, sizeof(sechdr))) 363 offset += sizeof(sechdr);
361 BUG(); 364 len -= sizeof(sechdr);
362 365
363 buf = ntohl(sechdr.data_size); 366 buf = ntohl(sechdr.data_size);
364 data_size = buf & 0xffff; 367 data_size = buf & 0xffff;
@@ -371,18 +374,16 @@ static int rxkad_verify_packet_1(struct rxrpc_call *call, struct sk_buff *skb,
371 goto protocol_error; 374 goto protocol_error;
372 } 375 }
373 376
374 /* shorten the packet to remove the padding */ 377 if (data_size > len) {
375 if (data_size > skb->len) {
376 rxrpc_abort_call("V1L", call, seq, RXKADDATALEN, EPROTO); 378 rxrpc_abort_call("V1L", call, seq, RXKADDATALEN, EPROTO);
377 goto protocol_error; 379 goto protocol_error;
378 } 380 }
379 if (data_size < skb->len)
380 skb->len = data_size;
381 381
382 _leave(" = 0 [dlen=%x]", data_size); 382 _leave(" = 0 [dlen=%x]", data_size);
383 return 0; 383 return 0;
384 384
385protocol_error: 385protocol_error:
386 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ABORT);
386 _leave(" = -EPROTO"); 387 _leave(" = -EPROTO");
387 return -EPROTO; 388 return -EPROTO;
388 389
@@ -395,6 +396,7 @@ nomem:
395 * wholly decrypt a packet (level 2 security) 396 * wholly decrypt a packet (level 2 security)
396 */ 397 */
397static int rxkad_verify_packet_2(struct rxrpc_call *call, struct sk_buff *skb, 398static int rxkad_verify_packet_2(struct rxrpc_call *call, struct sk_buff *skb,
399 unsigned int offset, unsigned int len,
398 rxrpc_seq_t seq) 400 rxrpc_seq_t seq)
399{ 401{
400 const struct rxrpc_key_token *token; 402 const struct rxrpc_key_token *token;
@@ -409,12 +411,14 @@ static int rxkad_verify_packet_2(struct rxrpc_call *call, struct sk_buff *skb,
409 411
410 _enter(",{%d}", skb->len); 412 _enter(",{%d}", skb->len);
411 413
412 if (skb->len < 8) { 414 if (len < 8) {
413 rxrpc_abort_call("V2H", call, seq, RXKADSEALEDINCON, EPROTO); 415 rxrpc_abort_call("V2H", call, seq, RXKADSEALEDINCON, EPROTO);
414 goto protocol_error; 416 goto protocol_error;
415 } 417 }
416 418
417 /* we want to decrypt the skbuff in-place */ 419 /* Decrypt the skbuff in-place. TODO: We really want to decrypt
420 * directly into the target buffer.
421 */
418 nsg = skb_cow_data(skb, 0, &trailer); 422 nsg = skb_cow_data(skb, 0, &trailer);
419 if (nsg < 0) 423 if (nsg < 0)
420 goto nomem; 424 goto nomem;
@@ -427,7 +431,7 @@ static int rxkad_verify_packet_2(struct rxrpc_call *call, struct sk_buff *skb,
427 } 431 }
428 432
429 sg_init_table(sg, nsg); 433 sg_init_table(sg, nsg);
430 skb_to_sgvec(skb, sg, 0, skb->len); 434 skb_to_sgvec(skb, sg, offset, len);
431 435
432 /* decrypt from the session key */ 436 /* decrypt from the session key */
433 token = call->conn->params.key->payload.data[0]; 437 token = call->conn->params.key->payload.data[0];
@@ -435,19 +439,19 @@ static int rxkad_verify_packet_2(struct rxrpc_call *call, struct sk_buff *skb,
435 439
436 skcipher_request_set_tfm(req, call->conn->cipher); 440 skcipher_request_set_tfm(req, call->conn->cipher);
437 skcipher_request_set_callback(req, 0, NULL, NULL); 441 skcipher_request_set_callback(req, 0, NULL, NULL);
438 skcipher_request_set_crypt(req, sg, sg, skb->len, iv.x); 442 skcipher_request_set_crypt(req, sg, sg, len, iv.x);
439 crypto_skcipher_decrypt(req); 443 crypto_skcipher_decrypt(req);
440 skcipher_request_zero(req); 444 skcipher_request_zero(req);
441 if (sg != _sg) 445 if (sg != _sg)
442 kfree(sg); 446 kfree(sg);
443 447
444 /* Extract the decrypted packet length */ 448 /* Extract the decrypted packet length */
445 if (skb_copy_bits(skb, 0, &sechdr, sizeof(sechdr)) < 0) { 449 if (skb_copy_bits(skb, offset, &sechdr, sizeof(sechdr)) < 0) {
446 rxrpc_abort_call("XV2", call, seq, RXKADDATALEN, EPROTO); 450 rxrpc_abort_call("XV2", call, seq, RXKADDATALEN, EPROTO);
447 goto protocol_error; 451 goto protocol_error;
448 } 452 }
449 if (!skb_pull(skb, sizeof(sechdr))) 453 offset += sizeof(sechdr);
450 BUG(); 454 len -= sizeof(sechdr);
451 455
452 buf = ntohl(sechdr.data_size); 456 buf = ntohl(sechdr.data_size);
453 data_size = buf & 0xffff; 457 data_size = buf & 0xffff;
@@ -460,17 +464,16 @@ static int rxkad_verify_packet_2(struct rxrpc_call *call, struct sk_buff *skb,
460 goto protocol_error; 464 goto protocol_error;
461 } 465 }
462 466
463 if (data_size > skb->len) { 467 if (data_size > len) {
464 rxrpc_abort_call("V2L", call, seq, RXKADDATALEN, EPROTO); 468 rxrpc_abort_call("V2L", call, seq, RXKADDATALEN, EPROTO);
465 goto protocol_error; 469 goto protocol_error;
466 } 470 }
467 if (data_size < skb->len)
468 skb->len = data_size;
469 471
470 _leave(" = 0 [dlen=%x]", data_size); 472 _leave(" = 0 [dlen=%x]", data_size);
471 return 0; 473 return 0;
472 474
473protocol_error: 475protocol_error:
476 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ABORT);
474 _leave(" = -EPROTO"); 477 _leave(" = -EPROTO");
475 return -EPROTO; 478 return -EPROTO;
476 479
@@ -484,6 +487,7 @@ nomem:
484 * jumbo packet). 487 * jumbo packet).
485 */ 488 */
486static int rxkad_verify_packet(struct rxrpc_call *call, struct sk_buff *skb, 489static int rxkad_verify_packet(struct rxrpc_call *call, struct sk_buff *skb,
490 unsigned int offset, unsigned int len,
487 rxrpc_seq_t seq, u16 expected_cksum) 491 rxrpc_seq_t seq, u16 expected_cksum)
488{ 492{
489 SKCIPHER_REQUEST_ON_STACK(req, call->conn->cipher); 493 SKCIPHER_REQUEST_ON_STACK(req, call->conn->cipher);
@@ -521,6 +525,7 @@ static int rxkad_verify_packet(struct rxrpc_call *call, struct sk_buff *skb,
521 525
522 if (cksum != expected_cksum) { 526 if (cksum != expected_cksum) {
523 rxrpc_abort_call("VCK", call, seq, RXKADSEALEDINCON, EPROTO); 527 rxrpc_abort_call("VCK", call, seq, RXKADSEALEDINCON, EPROTO);
528 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ABORT);
524 _leave(" = -EPROTO [csum failed]"); 529 _leave(" = -EPROTO [csum failed]");
525 return -EPROTO; 530 return -EPROTO;
526 } 531 }
@@ -529,15 +534,61 @@ static int rxkad_verify_packet(struct rxrpc_call *call, struct sk_buff *skb,
529 case RXRPC_SECURITY_PLAIN: 534 case RXRPC_SECURITY_PLAIN:
530 return 0; 535 return 0;
531 case RXRPC_SECURITY_AUTH: 536 case RXRPC_SECURITY_AUTH:
532 return rxkad_verify_packet_1(call, skb, seq); 537 return rxkad_verify_packet_1(call, skb, offset, len, seq);
533 case RXRPC_SECURITY_ENCRYPT: 538 case RXRPC_SECURITY_ENCRYPT:
534 return rxkad_verify_packet_2(call, skb, seq); 539 return rxkad_verify_packet_2(call, skb, offset, len, seq);
535 default: 540 default:
536 return -ENOANO; 541 return -ENOANO;
537 } 542 }
538} 543}
539 544
540/* 545/*
546 * Locate the data contained in a packet that was partially encrypted.
547 */
548static void rxkad_locate_data_1(struct rxrpc_call *call, struct sk_buff *skb,
549 unsigned int *_offset, unsigned int *_len)
550{
551 struct rxkad_level1_hdr sechdr;
552
553 if (skb_copy_bits(skb, *_offset, &sechdr, sizeof(sechdr)) < 0)
554 BUG();
555 *_offset += sizeof(sechdr);
556 *_len = ntohl(sechdr.data_size) & 0xffff;
557}
558
559/*
560 * Locate the data contained in a packet that was completely encrypted.
561 */
562static void rxkad_locate_data_2(struct rxrpc_call *call, struct sk_buff *skb,
563 unsigned int *_offset, unsigned int *_len)
564{
565 struct rxkad_level2_hdr sechdr;
566
567 if (skb_copy_bits(skb, *_offset, &sechdr, sizeof(sechdr)) < 0)
568 BUG();
569 *_offset += sizeof(sechdr);
570 *_len = ntohl(sechdr.data_size) & 0xffff;
571}
572
573/*
574 * Locate the data contained in an already decrypted packet.
575 */
576static void rxkad_locate_data(struct rxrpc_call *call, struct sk_buff *skb,
577 unsigned int *_offset, unsigned int *_len)
578{
579 switch (call->conn->params.security_level) {
580 case RXRPC_SECURITY_AUTH:
581 rxkad_locate_data_1(call, skb, _offset, _len);
582 return;
583 case RXRPC_SECURITY_ENCRYPT:
584 rxkad_locate_data_2(call, skb, _offset, _len);
585 return;
586 default:
587 return;
588 }
589}
590
591/*
541 * issue a challenge 592 * issue a challenge
542 */ 593 */
543static int rxkad_issue_challenge(struct rxrpc_connection *conn) 594static int rxkad_issue_challenge(struct rxrpc_connection *conn)
@@ -704,7 +755,7 @@ static int rxkad_respond_to_challenge(struct rxrpc_connection *conn,
704 struct rxkad_challenge challenge; 755 struct rxkad_challenge challenge;
705 struct rxkad_response resp 756 struct rxkad_response resp
706 __attribute__((aligned(8))); /* must be aligned for crypto */ 757 __attribute__((aligned(8))); /* must be aligned for crypto */
707 struct rxrpc_skb_priv *sp; 758 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
708 u32 version, nonce, min_level, abort_code; 759 u32 version, nonce, min_level, abort_code;
709 int ret; 760 int ret;
710 761
@@ -722,8 +773,7 @@ static int rxkad_respond_to_challenge(struct rxrpc_connection *conn,
722 } 773 }
723 774
724 abort_code = RXKADPACKETSHORT; 775 abort_code = RXKADPACKETSHORT;
725 sp = rxrpc_skb(skb); 776 if (skb_copy_bits(skb, sp->offset, &challenge, sizeof(challenge)) < 0)
726 if (skb_copy_bits(skb, 0, &challenge, sizeof(challenge)) < 0)
727 goto protocol_error; 777 goto protocol_error;
728 778
729 version = ntohl(challenge.version); 779 version = ntohl(challenge.version);
@@ -969,7 +1019,7 @@ static int rxkad_verify_response(struct rxrpc_connection *conn,
969{ 1019{
970 struct rxkad_response response 1020 struct rxkad_response response
971 __attribute__((aligned(8))); /* must be aligned for crypto */ 1021 __attribute__((aligned(8))); /* must be aligned for crypto */
972 struct rxrpc_skb_priv *sp; 1022 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
973 struct rxrpc_crypt session_key; 1023 struct rxrpc_crypt session_key;
974 time_t expiry; 1024 time_t expiry;
975 void *ticket; 1025 void *ticket;
@@ -980,7 +1030,7 @@ static int rxkad_verify_response(struct rxrpc_connection *conn,
980 _enter("{%d,%x}", conn->debug_id, key_serial(conn->server_key)); 1030 _enter("{%d,%x}", conn->debug_id, key_serial(conn->server_key));
981 1031
982 abort_code = RXKADPACKETSHORT; 1032 abort_code = RXKADPACKETSHORT;
983 if (skb_copy_bits(skb, 0, &response, sizeof(response)) < 0) 1033 if (skb_copy_bits(skb, sp->offset, &response, sizeof(response)) < 0)
984 goto protocol_error; 1034 goto protocol_error;
985 if (!pskb_pull(skb, sizeof(response))) 1035 if (!pskb_pull(skb, sizeof(response)))
986 BUG(); 1036 BUG();
@@ -988,7 +1038,6 @@ static int rxkad_verify_response(struct rxrpc_connection *conn,
988 version = ntohl(response.version); 1038 version = ntohl(response.version);
989 ticket_len = ntohl(response.ticket_len); 1039 ticket_len = ntohl(response.ticket_len);
990 kvno = ntohl(response.kvno); 1040 kvno = ntohl(response.kvno);
991 sp = rxrpc_skb(skb);
992 _proto("Rx RESPONSE %%%u { v=%u kv=%u tl=%u }", 1041 _proto("Rx RESPONSE %%%u { v=%u kv=%u tl=%u }",
993 sp->hdr.serial, version, kvno, ticket_len); 1042 sp->hdr.serial, version, kvno, ticket_len);
994 1043
@@ -1010,7 +1059,7 @@ static int rxkad_verify_response(struct rxrpc_connection *conn,
1010 return -ENOMEM; 1059 return -ENOMEM;
1011 1060
1012 abort_code = RXKADPACKETSHORT; 1061 abort_code = RXKADPACKETSHORT;
1013 if (skb_copy_bits(skb, 0, ticket, ticket_len) < 0) 1062 if (skb_copy_bits(skb, sp->offset, ticket, ticket_len) < 0)
1014 goto protocol_error_free; 1063 goto protocol_error_free;
1015 1064
1016 ret = rxkad_decrypt_ticket(conn, ticket, ticket_len, &session_key, 1065 ret = rxkad_decrypt_ticket(conn, ticket, ticket_len, &session_key,
@@ -1135,6 +1184,7 @@ const struct rxrpc_security rxkad = {
1135 .prime_packet_security = rxkad_prime_packet_security, 1184 .prime_packet_security = rxkad_prime_packet_security,
1136 .secure_packet = rxkad_secure_packet, 1185 .secure_packet = rxkad_secure_packet,
1137 .verify_packet = rxkad_verify_packet, 1186 .verify_packet = rxkad_verify_packet,
1187 .locate_data = rxkad_locate_data,
1138 .issue_challenge = rxkad_issue_challenge, 1188 .issue_challenge = rxkad_issue_challenge,
1139 .respond_to_challenge = rxkad_respond_to_challenge, 1189 .respond_to_challenge = rxkad_respond_to_challenge,
1140 .verify_response = rxkad_verify_response, 1190 .verify_response = rxkad_verify_response,
diff --git a/net/rxrpc/security.c b/net/rxrpc/security.c
index 5d79d5a9c944..82d8134e9287 100644
--- a/net/rxrpc/security.c
+++ b/net/rxrpc/security.c
@@ -130,20 +130,20 @@ int rxrpc_init_server_conn_security(struct rxrpc_connection *conn)
130 } 130 }
131 131
132 /* find the service */ 132 /* find the service */
133 read_lock_bh(&local->services_lock); 133 read_lock(&local->services_lock);
134 hlist_for_each_entry(rx, &local->services, listen_link) { 134 hlist_for_each_entry(rx, &local->services, listen_link) {
135 if (rx->srx.srx_service == conn->params.service_id) 135 if (rx->srx.srx_service == conn->params.service_id)
136 goto found_service; 136 goto found_service;
137 } 137 }
138 138
139 /* the service appears to have died */ 139 /* the service appears to have died */
140 read_unlock_bh(&local->services_lock); 140 read_unlock(&local->services_lock);
141 _leave(" = -ENOENT"); 141 _leave(" = -ENOENT");
142 return -ENOENT; 142 return -ENOENT;
143 143
144found_service: 144found_service:
145 if (!rx->securities) { 145 if (!rx->securities) {
146 read_unlock_bh(&local->services_lock); 146 read_unlock(&local->services_lock);
147 _leave(" = -ENOKEY"); 147 _leave(" = -ENOKEY");
148 return -ENOKEY; 148 return -ENOKEY;
149 } 149 }
@@ -152,13 +152,13 @@ found_service:
152 kref = keyring_search(make_key_ref(rx->securities, 1UL), 152 kref = keyring_search(make_key_ref(rx->securities, 1UL),
153 &key_type_rxrpc_s, kdesc); 153 &key_type_rxrpc_s, kdesc);
154 if (IS_ERR(kref)) { 154 if (IS_ERR(kref)) {
155 read_unlock_bh(&local->services_lock); 155 read_unlock(&local->services_lock);
156 _leave(" = %ld [search]", PTR_ERR(kref)); 156 _leave(" = %ld [search]", PTR_ERR(kref));
157 return PTR_ERR(kref); 157 return PTR_ERR(kref);
158 } 158 }
159 159
160 key = key_ref_to_ptr(kref); 160 key = key_ref_to_ptr(kref);
161 read_unlock_bh(&local->services_lock); 161 read_unlock(&local->services_lock);
162 162
163 conn->server_key = key; 163 conn->server_key = key;
164 conn->security = sec; 164 conn->security = sec;
diff --git a/net/rxrpc/sendmsg.c b/net/rxrpc/sendmsg.c
index 9a4af992fcdf..cba236575073 100644
--- a/net/rxrpc/sendmsg.c
+++ b/net/rxrpc/sendmsg.c
@@ -15,7 +15,6 @@
15#include <linux/gfp.h> 15#include <linux/gfp.h>
16#include <linux/skbuff.h> 16#include <linux/skbuff.h>
17#include <linux/export.h> 17#include <linux/export.h>
18#include <linux/circ_buf.h>
19#include <net/sock.h> 18#include <net/sock.h>
20#include <net/af_rxrpc.h> 19#include <net/af_rxrpc.h>
21#include "ar-internal.h" 20#include "ar-internal.h"
@@ -38,19 +37,20 @@ static int rxrpc_wait_for_tx_window(struct rxrpc_sock *rx,
38 DECLARE_WAITQUEUE(myself, current); 37 DECLARE_WAITQUEUE(myself, current);
39 int ret; 38 int ret;
40 39
41 _enter(",{%d},%ld", 40 _enter(",{%u,%u,%u}",
42 CIRC_SPACE(call->acks_head, ACCESS_ONCE(call->acks_tail), 41 call->tx_hard_ack, call->tx_top, call->tx_winsize);
43 call->acks_winsz),
44 *timeo);
45 42
46 add_wait_queue(&call->waitq, &myself); 43 add_wait_queue(&call->waitq, &myself);
47 44
48 for (;;) { 45 for (;;) {
49 set_current_state(TASK_INTERRUPTIBLE); 46 set_current_state(TASK_INTERRUPTIBLE);
50 ret = 0; 47 ret = 0;
51 if (CIRC_SPACE(call->acks_head, ACCESS_ONCE(call->acks_tail), 48 if (call->tx_top - call->tx_hard_ack < call->tx_winsize)
52 call->acks_winsz) > 0)
53 break; 49 break;
50 if (call->state >= RXRPC_CALL_COMPLETE) {
51 ret = -call->error;
52 break;
53 }
54 if (signal_pending(current)) { 54 if (signal_pending(current)) {
55 ret = sock_intr_errno(*timeo); 55 ret = sock_intr_errno(*timeo);
56 break; 56 break;
@@ -68,36 +68,44 @@ static int rxrpc_wait_for_tx_window(struct rxrpc_sock *rx,
68} 68}
69 69
70/* 70/*
71 * attempt to schedule an instant Tx resend 71 * Schedule an instant Tx resend.
72 */ 72 */
73static inline void rxrpc_instant_resend(struct rxrpc_call *call) 73static inline void rxrpc_instant_resend(struct rxrpc_call *call, int ix)
74{ 74{
75 read_lock_bh(&call->state_lock); 75 spin_lock_bh(&call->lock);
76 if (try_to_del_timer_sync(&call->resend_timer) >= 0) { 76
77 clear_bit(RXRPC_CALL_RUN_RTIMER, &call->flags); 77 if (call->state < RXRPC_CALL_COMPLETE) {
78 if (call->state < RXRPC_CALL_COMPLETE && 78 call->rxtx_annotations[ix] = RXRPC_TX_ANNO_RETRANS;
79 !test_and_set_bit(RXRPC_CALL_EV_RESEND_TIMER, &call->events)) 79 if (!test_and_set_bit(RXRPC_CALL_EV_RESEND, &call->events))
80 rxrpc_queue_call(call); 80 rxrpc_queue_call(call);
81 } 81 }
82 read_unlock_bh(&call->state_lock); 82
83 spin_unlock_bh(&call->lock);
83} 84}
84 85
85/* 86/*
86 * queue a packet for transmission, set the resend timer and attempt 87 * Queue a DATA packet for transmission, set the resend timeout and send the
87 * to send the packet immediately 88 * packet immediately
88 */ 89 */
89static void rxrpc_queue_packet(struct rxrpc_call *call, struct sk_buff *skb, 90static void rxrpc_queue_packet(struct rxrpc_call *call, struct sk_buff *skb,
90 bool last) 91 bool last)
91{ 92{
92 struct rxrpc_skb_priv *sp = rxrpc_skb(skb); 93 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
93 int ret; 94 rxrpc_seq_t seq = sp->hdr.seq;
95 int ret, ix;
96
97 _net("queue skb %p [%d]", skb, seq);
94 98
95 _net("queue skb %p [%d]", skb, call->acks_head); 99 ASSERTCMP(seq, ==, call->tx_top + 1);
96 100
97 ASSERT(call->acks_window != NULL); 101 ix = seq & RXRPC_RXTX_BUFF_MASK;
98 call->acks_window[call->acks_head] = (unsigned long) skb; 102 rxrpc_get_skb(skb);
103 call->rxtx_annotations[ix] = RXRPC_TX_ANNO_UNACK;
99 smp_wmb(); 104 smp_wmb();
100 call->acks_head = (call->acks_head + 1) & (call->acks_winsz - 1); 105 call->rxtx_buffer[ix] = skb;
106 call->tx_top = seq;
107 if (last)
108 set_bit(RXRPC_CALL_TX_LAST, &call->flags);
101 109
102 if (last || call->state == RXRPC_CALL_SERVER_ACK_REQUEST) { 110 if (last || call->state == RXRPC_CALL_SERVER_ACK_REQUEST) {
103 _debug("________awaiting reply/ACK__________"); 111 _debug("________awaiting reply/ACK__________");
@@ -121,34 +129,17 @@ static void rxrpc_queue_packet(struct rxrpc_call *call, struct sk_buff *skb,
121 129
122 _proto("Tx DATA %%%u { #%u }", sp->hdr.serial, sp->hdr.seq); 130 _proto("Tx DATA %%%u { #%u }", sp->hdr.serial, sp->hdr.seq);
123 131
124 sp->need_resend = false; 132 if (seq == 1 && rxrpc_is_client_call(call))
125 sp->resend_at = jiffies + rxrpc_resend_timeout; 133 rxrpc_expose_client_call(call);
126 if (!test_and_set_bit(RXRPC_CALL_RUN_RTIMER, &call->flags)) {
127 _debug("run timer");
128 call->resend_timer.expires = sp->resend_at;
129 add_timer(&call->resend_timer);
130 }
131
132 /* attempt to cancel the rx-ACK timer, deferring reply transmission if
133 * we're ACK'ing the request phase of an incoming call */
134 ret = -EAGAIN;
135 if (try_to_del_timer_sync(&call->ack_timer) >= 0) {
136 /* the packet may be freed by rxrpc_process_call() before this
137 * returns */
138 if (rxrpc_is_client_call(call))
139 rxrpc_expose_client_call(call);
140 ret = rxrpc_send_data_packet(call->conn, skb);
141 _net("sent skb %p", skb);
142 } else {
143 _debug("failed to delete ACK timer");
144 }
145 134
135 sp->resend_at = jiffies + rxrpc_resend_timeout;
136 ret = rxrpc_send_data_packet(call->conn, skb);
146 if (ret < 0) { 137 if (ret < 0) {
147 _debug("need instant resend %d", ret); 138 _debug("need instant resend %d", ret);
148 sp->need_resend = true; 139 rxrpc_instant_resend(call, ix);
149 rxrpc_instant_resend(call);
150 } 140 }
151 141
142 rxrpc_free_skb(skb);
152 _leave(""); 143 _leave("");
153} 144}
154 145
@@ -212,9 +203,8 @@ static int rxrpc_send_data(struct rxrpc_sock *rx,
212 203
213 _debug("alloc"); 204 _debug("alloc");
214 205
215 if (CIRC_SPACE(call->acks_head, 206 if (call->tx_top - call->tx_hard_ack >=
216 ACCESS_ONCE(call->acks_tail), 207 call->tx_winsize) {
217 call->acks_winsz) <= 0) {
218 ret = -EAGAIN; 208 ret = -EAGAIN;
219 if (msg->msg_flags & MSG_DONTWAIT) 209 if (msg->msg_flags & MSG_DONTWAIT)
220 goto maybe_error; 210 goto maybe_error;
@@ -313,7 +303,7 @@ static int rxrpc_send_data(struct rxrpc_sock *rx,
313 memset(skb_put(skb, pad), 0, pad); 303 memset(skb_put(skb, pad), 0, pad);
314 } 304 }
315 305
316 seq = atomic_inc_return(&call->sequence); 306 seq = call->tx_top + 1;
317 307
318 sp->hdr.epoch = conn->proto.epoch; 308 sp->hdr.epoch = conn->proto.epoch;
319 sp->hdr.cid = call->cid; 309 sp->hdr.cid = call->cid;
@@ -329,9 +319,8 @@ static int rxrpc_send_data(struct rxrpc_sock *rx,
329 sp->hdr.flags = conn->out_clientflag; 319 sp->hdr.flags = conn->out_clientflag;
330 if (msg_data_left(msg) == 0 && !more) 320 if (msg_data_left(msg) == 0 && !more)
331 sp->hdr.flags |= RXRPC_LAST_PACKET; 321 sp->hdr.flags |= RXRPC_LAST_PACKET;
332 else if (CIRC_SPACE(call->acks_head, 322 else if (call->tx_top - call->tx_hard_ack <
333 ACCESS_ONCE(call->acks_tail), 323 call->tx_winsize)
334 call->acks_winsz) > 1)
335 sp->hdr.flags |= RXRPC_MORE_PACKETS; 324 sp->hdr.flags |= RXRPC_MORE_PACKETS;
336 if (more && seq & 1) 325 if (more && seq & 1)
337 sp->hdr.flags |= RXRPC_REQUEST_ACK; 326 sp->hdr.flags |= RXRPC_REQUEST_ACK;
@@ -358,7 +347,7 @@ out:
358call_terminated: 347call_terminated:
359 rxrpc_free_skb(skb); 348 rxrpc_free_skb(skb);
360 _leave(" = %d", -call->error); 349 _leave(" = %d", -call->error);
361 return ret; 350 return -call->error;
362 351
363maybe_error: 352maybe_error:
364 if (copied) 353 if (copied)
@@ -452,29 +441,6 @@ static int rxrpc_sendmsg_cmsg(struct msghdr *msg,
452} 441}
453 442
454/* 443/*
455 * abort a call, sending an ABORT packet to the peer
456 */
457static void rxrpc_send_abort(struct rxrpc_call *call, const char *why,
458 u32 abort_code, int error)
459{
460 if (call->state >= RXRPC_CALL_COMPLETE)
461 return;
462
463 write_lock_bh(&call->state_lock);
464
465 if (__rxrpc_abort_call(why, call, 0, abort_code, error)) {
466 del_timer_sync(&call->resend_timer);
467 del_timer_sync(&call->ack_timer);
468 clear_bit(RXRPC_CALL_EV_RESEND_TIMER, &call->events);
469 clear_bit(RXRPC_CALL_EV_ACK, &call->events);
470 clear_bit(RXRPC_CALL_RUN_RTIMER, &call->flags);
471 rxrpc_queue_call(call);
472 }
473
474 write_unlock_bh(&call->state_lock);
475}
476
477/*
478 * Create a new client call for sendmsg(). 444 * Create a new client call for sendmsg().
479 */ 445 */
480static struct rxrpc_call * 446static struct rxrpc_call *
@@ -549,7 +515,6 @@ int rxrpc_do_sendmsg(struct rxrpc_sock *rx, struct msghdr *msg, size_t len)
549 return PTR_ERR(call); 515 return PTR_ERR(call);
550 } 516 }
551 517
552 rxrpc_see_call(call);
553 _debug("CALL %d USR %lx ST %d on CONN %p", 518 _debug("CALL %d USR %lx ST %d on CONN %p",
554 call->debug_id, call->user_call_ID, call->state, call->conn); 519 call->debug_id, call->user_call_ID, call->state, call->conn);
555 520
@@ -557,8 +522,10 @@ int rxrpc_do_sendmsg(struct rxrpc_sock *rx, struct msghdr *msg, size_t len)
557 /* it's too late for this call */ 522 /* it's too late for this call */
558 ret = -ESHUTDOWN; 523 ret = -ESHUTDOWN;
559 } else if (cmd == RXRPC_CMD_SEND_ABORT) { 524 } else if (cmd == RXRPC_CMD_SEND_ABORT) {
560 rxrpc_send_abort(call, "CMD", abort_code, ECONNABORTED);
561 ret = 0; 525 ret = 0;
526 if (rxrpc_abort_call("CMD", call, 0, abort_code, ECONNABORTED))
527 ret = rxrpc_send_call_packet(call,
528 RXRPC_PACKET_TYPE_ABORT);
562 } else if (cmd != RXRPC_CMD_SEND_DATA) { 529 } else if (cmd != RXRPC_CMD_SEND_DATA) {
563 ret = -EINVAL; 530 ret = -EINVAL;
564 } else if (rxrpc_is_client_call(call) && 531 } else if (rxrpc_is_client_call(call) &&
@@ -639,7 +606,8 @@ void rxrpc_kernel_abort_call(struct socket *sock, struct rxrpc_call *call,
639 606
640 lock_sock(sock->sk); 607 lock_sock(sock->sk);
641 608
642 rxrpc_send_abort(call, why, abort_code, error); 609 if (rxrpc_abort_call(why, call, 0, abort_code, error))
610 rxrpc_send_call_packet(call, RXRPC_PACKET_TYPE_ABORT);
643 611
644 release_sock(sock->sk); 612 release_sock(sock->sk);
645 _leave(""); 613 _leave("");
diff --git a/net/rxrpc/skbuff.c b/net/rxrpc/skbuff.c
index 9b8f8456d3bf..620d9ccaf3c1 100644
--- a/net/rxrpc/skbuff.c
+++ b/net/rxrpc/skbuff.c
@@ -19,133 +19,6 @@
19#include "ar-internal.h" 19#include "ar-internal.h"
20 20
21/* 21/*
22 * set up for the ACK at the end of the receive phase when we discard the final
23 * receive phase data packet
24 * - called with softirqs disabled
25 */
26static void rxrpc_request_final_ACK(struct rxrpc_call *call)
27{
28 /* the call may be aborted before we have a chance to ACK it */
29 write_lock(&call->state_lock);
30
31 switch (call->state) {
32 case RXRPC_CALL_CLIENT_RECV_REPLY:
33 call->state = RXRPC_CALL_CLIENT_FINAL_ACK;
34 _debug("request final ACK");
35
36 set_bit(RXRPC_CALL_EV_ACK_FINAL, &call->events);
37 if (try_to_del_timer_sync(&call->ack_timer) >= 0)
38 rxrpc_queue_call(call);
39 break;
40
41 case RXRPC_CALL_SERVER_RECV_REQUEST:
42 call->state = RXRPC_CALL_SERVER_ACK_REQUEST;
43 default:
44 break;
45 }
46
47 write_unlock(&call->state_lock);
48}
49
50/*
51 * drop the bottom ACK off of the call ACK window and advance the window
52 */
53static void rxrpc_hard_ACK_data(struct rxrpc_call *call, struct sk_buff *skb)
54{
55 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
56 int loop;
57 u32 seq;
58
59 spin_lock_bh(&call->lock);
60
61 _debug("hard ACK #%u", sp->hdr.seq);
62
63 for (loop = 0; loop < RXRPC_ACKR_WINDOW_ASZ; loop++) {
64 call->ackr_window[loop] >>= 1;
65 call->ackr_window[loop] |=
66 call->ackr_window[loop + 1] << (BITS_PER_LONG - 1);
67 }
68
69 seq = sp->hdr.seq;
70 ASSERTCMP(seq, ==, call->rx_data_eaten + 1);
71 call->rx_data_eaten = seq;
72
73 if (call->ackr_win_top < UINT_MAX)
74 call->ackr_win_top++;
75
76 ASSERTIFCMP(call->state <= RXRPC_CALL_COMPLETE,
77 call->rx_data_post, >=, call->rx_data_recv);
78 ASSERTIFCMP(call->state <= RXRPC_CALL_COMPLETE,
79 call->rx_data_recv, >=, call->rx_data_eaten);
80
81 if (sp->hdr.flags & RXRPC_LAST_PACKET) {
82 rxrpc_request_final_ACK(call);
83 } else if (atomic_dec_and_test(&call->ackr_not_idle) &&
84 test_and_clear_bit(RXRPC_CALL_TX_SOFT_ACK, &call->flags)) {
85 /* We previously soft-ACK'd some received packets that have now
86 * been consumed, so send a hard-ACK if no more packets are
87 * immediately forthcoming to allow the transmitter to free up
88 * its Tx bufferage.
89 */
90 _debug("send Rx idle ACK");
91 __rxrpc_propose_ACK(call, RXRPC_ACK_IDLE,
92 skb->priority, sp->hdr.serial, false);
93 }
94
95 spin_unlock_bh(&call->lock);
96}
97
98/**
99 * rxrpc_kernel_data_consumed - Record consumption of data message
100 * @call: The call to which the message pertains.
101 * @skb: Message holding data
102 *
103 * Record the consumption of a data message and generate an ACK if appropriate.
104 * The call state is shifted if this was the final packet. The caller must be
105 * in process context with no spinlocks held.
106 *
107 * TODO: Actually generate the ACK here rather than punting this to the
108 * workqueue.
109 */
110void rxrpc_kernel_data_consumed(struct rxrpc_call *call, struct sk_buff *skb)
111{
112 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
113
114 _enter("%d,%p{%u}", call->debug_id, skb, sp->hdr.seq);
115
116 ASSERTCMP(sp->call, ==, call);
117 ASSERTCMP(sp->hdr.type, ==, RXRPC_PACKET_TYPE_DATA);
118
119 /* TODO: Fix the sequence number tracking */
120 ASSERTCMP(sp->hdr.seq, >=, call->rx_data_recv);
121 ASSERTCMP(sp->hdr.seq, <=, call->rx_data_recv + 1);
122 ASSERTCMP(sp->hdr.seq, >, call->rx_data_eaten);
123
124 call->rx_data_recv = sp->hdr.seq;
125 rxrpc_hard_ACK_data(call, skb);
126}
127
128/*
129 * Destroy a packet that has an RxRPC control buffer
130 */
131void rxrpc_packet_destructor(struct sk_buff *skb)
132{
133 struct rxrpc_skb_priv *sp = rxrpc_skb(skb);
134 struct rxrpc_call *call = sp->call;
135
136 _enter("%p{%p}", skb, call);
137
138 if (call) {
139 rxrpc_put_call_for_skb(call, skb);
140 sp->call = NULL;
141 }
142
143 if (skb->sk)
144 sock_rfree(skb);
145 _leave("");
146}
147
148/*
149 * Note the existence of a new-to-us socket buffer (allocated or dequeued). 22 * Note the existence of a new-to-us socket buffer (allocated or dequeued).
150 */ 23 */
151void rxrpc_new_skb(struct sk_buff *skb) 24void rxrpc_new_skb(struct sk_buff *skb)
generated by cgit v1.2.2 (git 2.25.0) at 2025-04-11 07:40:09 -0400