IB/core: Add "type 2" memory windows support

This patch enhances the IB core support for Memory Windows (MWs).

MWs allow an application to have better/flexible control over remote
access to memory.

Two types of MWs are supported, with the second type having two flavors:

    Type 1  - associated with PD only
    Type 2A - associated with QPN only
    Type 2B - associated with PD and QPN

Applications can allocate a MW once, and then repeatedly bind the MW
to different ranges in MRs that are associated to the same PD. Type 1
windows are bound through a verb, while type 2 windows are bound by
posting a work request.

The 32-bit memory key is composed of a 24-bit index and an 8-bit
key. The key is changed with each bind, thus allowing more control
over the peer's use of the memory key.

The changes introduced are the following:

* add memory window type enum and a corresponding parameter to ib_alloc_mw.
* type 2 memory window bind work request support.
* create a struct that contains the common part of the bind verb struct
  ibv_mw_bind and the bind work request into a single struct.
* add the ib_inc_rkey helper function to advance the tag part of an rkey.

Consumer interface details:

* new device capability flags IB_DEVICE_MEM_WINDOW_TYPE_2A and
  IB_DEVICE_MEM_WINDOW_TYPE_2B are added to indicate device support
  for these features.

  Devices can set either IB_DEVICE_MEM_WINDOW_TYPE_2A or
  IB_DEVICE_MEM_WINDOW_TYPE_2B if it supports type 2A or type 2B
  memory windows. It can set neither to indicate it doesn't support
  type 2 windows at all.

* modify existing provides and consumers code to the new param of
  ib_alloc_mw and the ib_mw_bind_info structure

Signed-off-by: Haggai Eran <haggaie@mellanox.com>
Signed-off-by: Shani Michaeli <shanim@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>

1 files changed, 10 insertions, 10 deletions

diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
index 745973b729af..93726560eaa8 100644
--- a/net/sunrpc/xprtrdma/verbs.c
+++ b/net/sunrpc/xprtrdma/verbs.c
@@ -1086,7 +1086,7 @@ rpcrdma_buffer_create(struct rpcrdma_buffer *buf, struct rpcrdma_ep *ep,
         case RPCRDMA_MEMWINDOWS:
                 /* Allocate one extra request's worth, for full cycling */
                 for (i = (buf->rb_max_requests+1) * RPCRDMA_MAX_SEGS; i; i--) {
-                        r->r.mw = ib_alloc_mw(ia->ri_pd);
+                        r->r.mw = ib_alloc_mw(ia->ri_pd, IB_MW_TYPE_1);
                         if (IS_ERR(r->r.mw)) {
                                 rc = PTR_ERR(r->r.mw);
                                 dprintk("RPC:       %s: ib_alloc_mw"
@@ -1673,12 +1673,12 @@ rpcrdma_register_memwin_external(struct rpcrdma_mr_seg *seg,
 
         *nsegs = 1;
         rpcrdma_map_one(ia, seg, writing);
-        param.mr = ia->ri_bind_mem;
+        param.bind_info.mr = ia->ri_bind_mem;
         param.wr_id = 0ULL;     /* no send cookie */
-        param.addr = seg->mr_dma;
-        param.length = seg->mr_len;
+        param.bind_info.addr = seg->mr_dma;
+        param.bind_info.length = seg->mr_len;
         param.send_flags = 0;
-        param.mw_access_flags = mem_priv;
+        param.bind_info.mw_access_flags = mem_priv;
 
         DECR_CQCOUNT(&r_xprt->rx_ep);
         rc = ib_bind_mw(ia->ri_id->qp, seg->mr_chunk.rl_mw->r.mw, &param);
@@ -1690,7 +1690,7 @@ rpcrdma_register_memwin_external(struct rpcrdma_mr_seg *seg,
                 rpcrdma_unmap_one(ia, seg);
         } else {
                 seg->mr_rkey = seg->mr_chunk.rl_mw->r.mw->rkey;
-                seg->mr_base = param.addr;
+                seg->mr_base = param.bind_info.addr;
                 seg->mr_nsegs = 1;
         }
         return rc;
@@ -1706,10 +1706,10 @@ rpcrdma_deregister_memwin_external(struct rpcrdma_mr_seg *seg,
         int rc;
 
         BUG_ON(seg->mr_nsegs != 1);
-        param.mr = ia->ri_bind_mem;
-        param.addr = 0ULL;      /* unbind */
-        param.length = 0;
-        param.mw_access_flags = 0;
+        param.bind_info.mr = ia->ri_bind_mem;
+        param.bind_info.addr = 0ULL;    /* unbind */
+        param.bind_info.length = 0;
+        param.bind_info.mw_access_flags = 0;
         if (*r) {
                 param.wr_id = (u64) (unsigned long) *r;
                 param.send_flags = IB_SEND_SIGNALED;