IB/srp: Add multichannel support

Improve performance by using multiple RDMA/RC channels per SCSI
host for communication with an SRP target. About the
implementation:
- Introduce a loop over all channels in the code that uses
  target->ch.
- Set the SRP_MULTICHAN_MULTI flag during login for the creation
  of the second and subsequent channels.
- RDMA completion vectors are chosen such that RDMA completion
  interrupts are handled by the CPU socket that submitted the I/O
  request. As one can see in this patch it has been assumed if a
  system contains n CPU sockets and m RDMA completion vectors
  have been assigned to an RDMA HCA that IRQ affinity has been
  configured such that completion vectors [i*m/n..(i+1)*m/n) are
  bound to CPU socket i with 0 <= i < n.
- Modify srp_free_ch_ib() and srp_free_req_data() such that it
  becomes safe to invoke these functions after the corresponding
  allocation function failed.
- Add a ch_count sysfs attribute per target port.

Signed-off-by: Bart Van Assche <bvanassche@acm.org>
Reviewed-by: Sagi Grimberg <sagig@mellanox.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>

1 files changed, 19 insertions, 6 deletions

diff --git a/Documentation/ABI/stable/sysfs-driver-ib_srp b/Documentation/ABI/stable/sysfs-driver-ib_srp
index b9688de8455b..7049a2b50359 100644
--- a/Documentation/ABI/stable/sysfs-driver-ib_srp
+++ b/Documentation/ABI/stable/sysfs-driver-ib_srp
@@ -55,12 +55,12 @@ Description:	Interface for making ib_srp connect to a new target.
                   only safe with partial memory descriptor list support enabled
                   (allow_ext_sg=1).
                 * comp_vector, a number in the range 0..n-1 specifying the
-                  MSI-X completion vector. Some HCA's allocate multiple (n)
-                  MSI-X vectors per HCA port. If the IRQ affinity masks of
-                  these interrupts have been configured such that each MSI-X
-                  interrupt is handled by a different CPU then the comp_vector
-                  parameter can be used to spread the SRP completion workload
-                  over multiple CPU's.
+                  MSI-X completion vector of the first RDMA channel. Some
+                  HCA's allocate multiple (n) MSI-X vectors per HCA port. If
+                  the IRQ affinity masks of these interrupts have been
+                  configured such that each MSI-X interrupt is handled by a
+                  different CPU then the comp_vector parameter can be used to
+                  spread the SRP completion workload over multiple CPU's.
                 * tl_retry_count, a number in the range 2..7 specifying the
                   IB RC retry count.
                 * queue_size, the maximum number of commands that the
@@ -88,6 +88,13 @@ Description:	Whether ib_srp is allowed to include a partial memory
                 descriptor list in an SRP_CMD when communicating with an SRP
                 target.
 
+What:           /sys/class/scsi_host/host<n>/ch_count
+Date:           April 1, 2015
+KernelVersion:  3.19
+Contact:        linux-rdma@vger.kernel.org
+Description:    Number of RDMA channels used for communication with the SRP
+                target.
+
 What:           /sys/class/scsi_host/host<n>/cmd_sg_entries
 Date:           May 19, 2011
 KernelVersion:  2.6.39
@@ -95,6 +102,12 @@ Contact:	linux-rdma@vger.kernel.org
 Description:    Maximum number of data buffer descriptors that may be sent to
                 the target in a single SRP_CMD request.
 
+What:           /sys/class/scsi_host/host<n>/comp_vector
+Date:           September 2, 2013
+KernelVersion:  3.11
+Contact:        linux-rdma@vger.kernel.org
+Description:    Completion vector used for the first RDMA channel.
+
 What:           /sys/class/scsi_host/host<n>/dgid
 Date:           June 17, 2006
 KernelVersion:  2.6.17