diff options
| -rw-r--r-- | Documentation/filesystems/nfs-rdma.txt | 75 |
1 files changed, 44 insertions, 31 deletions
diff --git a/Documentation/filesystems/nfs-rdma.txt b/Documentation/filesystems/nfs-rdma.txt index d0ec45ae4e7d..9ad453d4891a 100644 --- a/Documentation/filesystems/nfs-rdma.txt +++ b/Documentation/filesystems/nfs-rdma.txt | |||
| @@ -5,7 +5,7 @@ | |||
| 5 | ################################################################################ | 5 | ################################################################################ |
| 6 | 6 | ||
| 7 | Author: NetApp and Open Grid Computing | 7 | Author: NetApp and Open Grid Computing |
| 8 | Date: April 15, 2008 | 8 | Date: May 29, 2008 |
| 9 | 9 | ||
| 10 | Table of Contents | 10 | Table of Contents |
| 11 | ~~~~~~~~~~~~~~~~~ | 11 | ~~~~~~~~~~~~~~~~~ |
| @@ -60,16 +60,18 @@ Installation | |||
| 60 | The procedures described in this document have been tested with | 60 | The procedures described in this document have been tested with |
| 61 | distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). | 61 | distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). |
| 62 | 62 | ||
| 63 | - Install nfs-utils-1.1.1 or greater on the client | 63 | - Install nfs-utils-1.1.2 or greater on the client |
| 64 | 64 | ||
| 65 | An NFS/RDMA mount point can only be obtained by using the mount.nfs | 65 | An NFS/RDMA mount point can be obtained by using the mount.nfs command in |
| 66 | command in nfs-utils-1.1.1 or greater. To see which version of mount.nfs | 66 | nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils version |
| 67 | you are using, type: | 67 | with support for NFS/RDMA mounts, but for various reasons we recommend using |
| 68 | nfs-utils-1.1.2 or greater). To see which version of mount.nfs you are | ||
| 69 | using, type: | ||
| 68 | 70 | ||
| 69 | > /sbin/mount.nfs -V | 71 | $ /sbin/mount.nfs -V |
| 70 | 72 | ||
| 71 | If the version is less than 1.1.1 or the command does not exist, | 73 | If the version is less than 1.1.2 or the command does not exist, |
| 72 | then you will need to install the latest version of nfs-utils. | 74 | you should install the latest version of nfs-utils. |
| 73 | 75 | ||
| 74 | Download the latest package from: | 76 | Download the latest package from: |
| 75 | 77 | ||
| @@ -77,22 +79,32 @@ Installation | |||
| 77 | 79 | ||
| 78 | Uncompress the package and follow the installation instructions. | 80 | Uncompress the package and follow the installation instructions. |
| 79 | 81 | ||
| 80 | If you will not be using GSS and NFSv4, the installation process | 82 | If you will not need the idmapper and gssd executables (you do not need |
| 81 | can be simplified by disabling these features when running configure: | 83 | these to create an NFS/RDMA enabled mount command), the installation |
| 84 | process can be simplified by disabling these features when running | ||
| 85 | configure: | ||
| 82 | 86 | ||
| 83 | > ./configure --disable-gss --disable-nfsv4 | 87 | $ ./configure --disable-gss --disable-nfsv4 |
| 84 | 88 | ||
| 85 | For more information on this see the package's README and INSTALL files. | 89 | To build nfs-utils you will need the tcp_wrappers package installed. For |
| 90 | more information on this see the package's README and INSTALL files. | ||
| 86 | 91 | ||
| 87 | After building the nfs-utils package, there will be a mount.nfs binary in | 92 | After building the nfs-utils package, there will be a mount.nfs binary in |
| 88 | the utils/mount directory. This binary can be used to initiate NFS v2, v3, | 93 | the utils/mount directory. This binary can be used to initiate NFS v2, v3, |
| 89 | or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4. | 94 | or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4. |
| 90 | The standard technique is to create a symlink called mount.nfs4 to mount.nfs. | 95 | The standard technique is to create a symlink called mount.nfs4 to mount.nfs. |
| 91 | 96 | ||
| 92 | NOTE: mount.nfs and therefore nfs-utils-1.1.1 or greater is only needed | 97 | This mount.nfs binary should be installed at /sbin/mount.nfs as follows: |
| 98 | |||
| 99 | $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs | ||
| 100 | |||
| 101 | In this location, mount.nfs will be invoked automatically for NFS mounts | ||
| 102 | by the system mount commmand. | ||
| 103 | |||
| 104 | NOTE: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed | ||
| 93 | on the NFS client machine. You do not need this specific version of | 105 | on the NFS client machine. You do not need this specific version of |
| 94 | nfs-utils on the server. Furthermore, only the mount.nfs command from | 106 | nfs-utils on the server. Furthermore, only the mount.nfs command from |
| 95 | nfs-utils-1.1.1 is needed on the client. | 107 | nfs-utils-1.1.2 is needed on the client. |
| 96 | 108 | ||
| 97 | - Install a Linux kernel with NFS/RDMA | 109 | - Install a Linux kernel with NFS/RDMA |
| 98 | 110 | ||
| @@ -156,8 +168,8 @@ Check RDMA and NFS Setup | |||
| 156 | this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel | 168 | this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel |
| 157 | card: | 169 | card: |
| 158 | 170 | ||
| 159 | > modprobe ib_mthca | 171 | $ modprobe ib_mthca |
| 160 | > modprobe ib_ipoib | 172 | $ modprobe ib_ipoib |
| 161 | 173 | ||
| 162 | If you are using InfiniBand, make sure there is a Subnet Manager (SM) | 174 | If you are using InfiniBand, make sure there is a Subnet Manager (SM) |
| 163 | running on the network. If your IB switch has an embedded SM, you can | 175 | running on the network. If your IB switch has an embedded SM, you can |
| @@ -166,7 +178,7 @@ Check RDMA and NFS Setup | |||
| 166 | 178 | ||
| 167 | If an SM is running on your network, you should see the following: | 179 | If an SM is running on your network, you should see the following: |
| 168 | 180 | ||
| 169 | > cat /sys/class/infiniband/driverX/ports/1/state | 181 | $ cat /sys/class/infiniband/driverX/ports/1/state |
| 170 | 4: ACTIVE | 182 | 4: ACTIVE |
| 171 | 183 | ||
| 172 | where driverX is mthca0, ipath5, ehca3, etc. | 184 | where driverX is mthca0, ipath5, ehca3, etc. |
| @@ -174,10 +186,10 @@ Check RDMA and NFS Setup | |||
| 174 | To further test the InfiniBand software stack, use IPoIB (this | 186 | To further test the InfiniBand software stack, use IPoIB (this |
| 175 | assumes you have two IB hosts named host1 and host2): | 187 | assumes you have two IB hosts named host1 and host2): |
| 176 | 188 | ||
| 177 | host1> ifconfig ib0 a.b.c.x | 189 | host1$ ifconfig ib0 a.b.c.x |
| 178 | host2> ifconfig ib0 a.b.c.y | 190 | host2$ ifconfig ib0 a.b.c.y |
| 179 | host1> ping a.b.c.y | 191 | host1$ ping a.b.c.y |
| 180 | host2> ping a.b.c.x | 192 | host2$ ping a.b.c.x |
| 181 | 193 | ||
| 182 | For other device types, follow the appropriate procedures. | 194 | For other device types, follow the appropriate procedures. |
| 183 | 195 | ||
| @@ -214,9 +226,9 @@ NFS/RDMA Setup | |||
| 214 | 226 | ||
| 215 | For InfiniBand using a Mellanox adapter: | 227 | For InfiniBand using a Mellanox adapter: |
| 216 | 228 | ||
| 217 | > modprobe ib_mthca | 229 | $ modprobe ib_mthca |
| 218 | > modprobe ib_ipoib | 230 | $ modprobe ib_ipoib |
| 219 | > ifconfig ib0 a.b.c.d | 231 | $ ifconfig ib0 a.b.c.d |
| 220 | 232 | ||
| 221 | NOTE: use unique addresses for the client and server | 233 | NOTE: use unique addresses for the client and server |
| 222 | 234 | ||
| @@ -225,30 +237,31 @@ NFS/RDMA Setup | |||
| 225 | If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), | 237 | If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), |
| 226 | load the RDMA transport module: | 238 | load the RDMA transport module: |
| 227 | 239 | ||
| 228 | > modprobe svcrdma | 240 | $ modprobe svcrdma |
| 229 | 241 | ||
| 230 | Regardless of how the server was built (module or built-in), start the server: | 242 | Regardless of how the server was built (module or built-in), start the server: |
| 231 | 243 | ||
| 232 | > /etc/init.d/nfs start | 244 | $ /etc/init.d/nfs start |
| 233 | 245 | ||
| 234 | or | 246 | or |
| 235 | 247 | ||
| 236 | > service nfs start | 248 | $ service nfs start |
| 237 | 249 | ||
| 238 | Instruct the server to listen on the RDMA transport: | 250 | Instruct the server to listen on the RDMA transport: |
| 239 | 251 | ||
| 240 | > echo rdma 2050 > /proc/fs/nfsd/portlist | 252 | $ echo rdma 2050 > /proc/fs/nfsd/portlist |
| 241 | 253 | ||
| 242 | - On the client system | 254 | - On the client system |
| 243 | 255 | ||
| 244 | If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), | 256 | If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), |
| 245 | load the RDMA client module: | 257 | load the RDMA client module: |
| 246 | 258 | ||
| 247 | > modprobe xprtrdma.ko | 259 | $ modprobe xprtrdma.ko |
| 248 | 260 | ||
| 249 | Regardless of how the client was built (module or built-in), issue the mount.nfs command: | 261 | Regardless of how the client was built (module or built-in), use this command to |
| 262 | mount the NFS/RDMA server: | ||
| 250 | 263 | ||
| 251 | > /path/to/your/mount.nfs <IPoIB-server-name-or-address>:/<export> /mnt -i -o rdma,port=2050 | 264 | $ mount -o rdma,port=2050 <IPoIB-server-name-or-address>:/<export> /mnt |
| 252 | 265 | ||
| 253 | To verify that the mount is using RDMA, run "cat /proc/mounts" and check the | 266 | To verify that the mount is using RDMA, run "cat /proc/mounts" and check the |
| 254 | "proto" field for the given mount. | 267 | "proto" field for the given mount. |
