diff options
-rw-r--r-- | Documentation/filesystems/nfs-rdma.txt | 75 |
1 files changed, 44 insertions, 31 deletions
diff --git a/Documentation/filesystems/nfs-rdma.txt b/Documentation/filesystems/nfs-rdma.txt index d0ec45ae4e7d..9ad453d4891a 100644 --- a/Documentation/filesystems/nfs-rdma.txt +++ b/Documentation/filesystems/nfs-rdma.txt | |||
@@ -5,7 +5,7 @@ | |||
5 | ################################################################################ | 5 | ################################################################################ |
6 | 6 | ||
7 | Author: NetApp and Open Grid Computing | 7 | Author: NetApp and Open Grid Computing |
8 | Date: April 15, 2008 | 8 | Date: May 29, 2008 |
9 | 9 | ||
10 | Table of Contents | 10 | Table of Contents |
11 | ~~~~~~~~~~~~~~~~~ | 11 | ~~~~~~~~~~~~~~~~~ |
@@ -60,16 +60,18 @@ Installation | |||
60 | The procedures described in this document have been tested with | 60 | The procedures described in this document have been tested with |
61 | distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). | 61 | distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). |
62 | 62 | ||
63 | - Install nfs-utils-1.1.1 or greater on the client | 63 | - Install nfs-utils-1.1.2 or greater on the client |
64 | 64 | ||
65 | An NFS/RDMA mount point can only be obtained by using the mount.nfs | 65 | An NFS/RDMA mount point can be obtained by using the mount.nfs command in |
66 | command in nfs-utils-1.1.1 or greater. To see which version of mount.nfs | 66 | nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils version |
67 | you are using, type: | 67 | with support for NFS/RDMA mounts, but for various reasons we recommend using |
68 | nfs-utils-1.1.2 or greater). To see which version of mount.nfs you are | ||
69 | using, type: | ||
68 | 70 | ||
69 | > /sbin/mount.nfs -V | 71 | $ /sbin/mount.nfs -V |
70 | 72 | ||
71 | If the version is less than 1.1.1 or the command does not exist, | 73 | If the version is less than 1.1.2 or the command does not exist, |
72 | then you will need to install the latest version of nfs-utils. | 74 | you should install the latest version of nfs-utils. |
73 | 75 | ||
74 | Download the latest package from: | 76 | Download the latest package from: |
75 | 77 | ||
@@ -77,22 +79,32 @@ Installation | |||
77 | 79 | ||
78 | Uncompress the package and follow the installation instructions. | 80 | Uncompress the package and follow the installation instructions. |
79 | 81 | ||
80 | If you will not be using GSS and NFSv4, the installation process | 82 | If you will not need the idmapper and gssd executables (you do not need |
81 | can be simplified by disabling these features when running configure: | 83 | these to create an NFS/RDMA enabled mount command), the installation |
84 | process can be simplified by disabling these features when running | ||
85 | configure: | ||
82 | 86 | ||
83 | > ./configure --disable-gss --disable-nfsv4 | 87 | $ ./configure --disable-gss --disable-nfsv4 |
84 | 88 | ||
85 | For more information on this see the package's README and INSTALL files. | 89 | To build nfs-utils you will need the tcp_wrappers package installed. For |
90 | more information on this see the package's README and INSTALL files. | ||
86 | 91 | ||
87 | After building the nfs-utils package, there will be a mount.nfs binary in | 92 | After building the nfs-utils package, there will be a mount.nfs binary in |
88 | the utils/mount directory. This binary can be used to initiate NFS v2, v3, | 93 | the utils/mount directory. This binary can be used to initiate NFS v2, v3, |
89 | or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4. | 94 | or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4. |
90 | The standard technique is to create a symlink called mount.nfs4 to mount.nfs. | 95 | The standard technique is to create a symlink called mount.nfs4 to mount.nfs. |
91 | 96 | ||
92 | NOTE: mount.nfs and therefore nfs-utils-1.1.1 or greater is only needed | 97 | This mount.nfs binary should be installed at /sbin/mount.nfs as follows: |
98 | |||
99 | $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs | ||
100 | |||
101 | In this location, mount.nfs will be invoked automatically for NFS mounts | ||
102 | by the system mount commmand. | ||
103 | |||
104 | NOTE: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed | ||
93 | on the NFS client machine. You do not need this specific version of | 105 | on the NFS client machine. You do not need this specific version of |
94 | nfs-utils on the server. Furthermore, only the mount.nfs command from | 106 | nfs-utils on the server. Furthermore, only the mount.nfs command from |
95 | nfs-utils-1.1.1 is needed on the client. | 107 | nfs-utils-1.1.2 is needed on the client. |
96 | 108 | ||
97 | - Install a Linux kernel with NFS/RDMA | 109 | - Install a Linux kernel with NFS/RDMA |
98 | 110 | ||
@@ -156,8 +168,8 @@ Check RDMA and NFS Setup | |||
156 | this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel | 168 | this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel |
157 | card: | 169 | card: |
158 | 170 | ||
159 | > modprobe ib_mthca | 171 | $ modprobe ib_mthca |
160 | > modprobe ib_ipoib | 172 | $ modprobe ib_ipoib |
161 | 173 | ||
162 | If you are using InfiniBand, make sure there is a Subnet Manager (SM) | 174 | If you are using InfiniBand, make sure there is a Subnet Manager (SM) |
163 | running on the network. If your IB switch has an embedded SM, you can | 175 | running on the network. If your IB switch has an embedded SM, you can |
@@ -166,7 +178,7 @@ Check RDMA and NFS Setup | |||
166 | 178 | ||
167 | If an SM is running on your network, you should see the following: | 179 | If an SM is running on your network, you should see the following: |
168 | 180 | ||
169 | > cat /sys/class/infiniband/driverX/ports/1/state | 181 | $ cat /sys/class/infiniband/driverX/ports/1/state |
170 | 4: ACTIVE | 182 | 4: ACTIVE |
171 | 183 | ||
172 | where driverX is mthca0, ipath5, ehca3, etc. | 184 | where driverX is mthca0, ipath5, ehca3, etc. |
@@ -174,10 +186,10 @@ Check RDMA and NFS Setup | |||
174 | To further test the InfiniBand software stack, use IPoIB (this | 186 | To further test the InfiniBand software stack, use IPoIB (this |
175 | assumes you have two IB hosts named host1 and host2): | 187 | assumes you have two IB hosts named host1 and host2): |
176 | 188 | ||
177 | host1> ifconfig ib0 a.b.c.x | 189 | host1$ ifconfig ib0 a.b.c.x |
178 | host2> ifconfig ib0 a.b.c.y | 190 | host2$ ifconfig ib0 a.b.c.y |
179 | host1> ping a.b.c.y | 191 | host1$ ping a.b.c.y |
180 | host2> ping a.b.c.x | 192 | host2$ ping a.b.c.x |
181 | 193 | ||
182 | For other device types, follow the appropriate procedures. | 194 | For other device types, follow the appropriate procedures. |
183 | 195 | ||
@@ -214,9 +226,9 @@ NFS/RDMA Setup | |||
214 | 226 | ||
215 | For InfiniBand using a Mellanox adapter: | 227 | For InfiniBand using a Mellanox adapter: |
216 | 228 | ||
217 | > modprobe ib_mthca | 229 | $ modprobe ib_mthca |
218 | > modprobe ib_ipoib | 230 | $ modprobe ib_ipoib |
219 | > ifconfig ib0 a.b.c.d | 231 | $ ifconfig ib0 a.b.c.d |
220 | 232 | ||
221 | NOTE: use unique addresses for the client and server | 233 | NOTE: use unique addresses for the client and server |
222 | 234 | ||
@@ -225,30 +237,31 @@ NFS/RDMA Setup | |||
225 | If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), | 237 | If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), |
226 | load the RDMA transport module: | 238 | load the RDMA transport module: |
227 | 239 | ||
228 | > modprobe svcrdma | 240 | $ modprobe svcrdma |
229 | 241 | ||
230 | Regardless of how the server was built (module or built-in), start the server: | 242 | Regardless of how the server was built (module or built-in), start the server: |
231 | 243 | ||
232 | > /etc/init.d/nfs start | 244 | $ /etc/init.d/nfs start |
233 | 245 | ||
234 | or | 246 | or |
235 | 247 | ||
236 | > service nfs start | 248 | $ service nfs start |
237 | 249 | ||
238 | Instruct the server to listen on the RDMA transport: | 250 | Instruct the server to listen on the RDMA transport: |
239 | 251 | ||
240 | > echo rdma 2050 > /proc/fs/nfsd/portlist | 252 | $ echo rdma 2050 > /proc/fs/nfsd/portlist |
241 | 253 | ||
242 | - On the client system | 254 | - On the client system |
243 | 255 | ||
244 | If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), | 256 | If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), |
245 | load the RDMA client module: | 257 | load the RDMA client module: |
246 | 258 | ||
247 | > modprobe xprtrdma.ko | 259 | $ modprobe xprtrdma.ko |
248 | 260 | ||
249 | Regardless of how the client was built (module or built-in), issue the mount.nfs command: | 261 | Regardless of how the client was built (module or built-in), use this command to |
262 | mount the NFS/RDMA server: | ||
250 | 263 | ||
251 | > /path/to/your/mount.nfs <IPoIB-server-name-or-address>:/<export> /mnt -i -o rdma,port=2050 | 264 | $ mount -o rdma,port=2050 <IPoIB-server-name-or-address>:/<export> /mnt |
252 | 265 | ||
253 | To verify that the mount is using RDMA, run "cat /proc/mounts" and check the | 266 | To verify that the mount is using RDMA, run "cat /proc/mounts" and check the |
254 | "proto" field for the given mount. | 267 | "proto" field for the given mount. |