mlx4_core: Write MTTs from CPU instead with of WRITE_MTT FW command

Write MTT entries directly to ICM from the driver (eliminating use of
WRITE_MTT command).  This reduces the number of FW commands needed to
register an MR by at least a factor of 2 and speeds up memory
registration significantly.  This code will also be used to implement
FMRs.

Signed-off-by: Jack Morgenstein <jackm@dev.mellanox.co.il>
Signed-off-by: Michael S. Tsirkin <mst@dev.mellanox.co.il>
Signed-off-by: Roland Dreier <rolandd@cisco.com>

1 files changed, 11 insertions, 0 deletions

diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c
index ed7e8d76cde0..478b3ba74edc 100644
--- a/drivers/net/mlx4/main.c
+++ b/drivers/net/mlx4/main.c
@@ -300,6 +300,17 @@ static int __devinit mlx4_init_icm(struct mlx4_dev *dev,
                 goto err_unmap_cmpt;
         }
 
+        /*
+         * Reserved MTT entries must be aligned up to a cacheline
+         * boundary, since the FW will write to them, while the driver
+         * writes to all other MTT entries. (The variable
+         * dev->caps.mtt_entry_sz below is really the MTT segment
+         * size, not the raw entry size)
+         */
+        dev->caps.reserved_mtts =
+                ALIGN(dev->caps.reserved_mtts * dev->caps.mtt_entry_sz,
+                      dma_get_cache_alignment()) / dev->caps.mtt_entry_sz;
+
         err = mlx4_init_icm_table(dev, &priv->mr_table.mtt_table,
                                   init_hca->mtt_base,
                                   dev->caps.mtt_entry_sz,