diff options
Diffstat (limited to 'Documentation/filesystems/nfs-rdma.txt')
-rw-r--r-- | Documentation/filesystems/nfs-rdma.txt | 103 |
1 files changed, 59 insertions, 44 deletions
diff --git a/Documentation/filesystems/nfs-rdma.txt b/Documentation/filesystems/nfs-rdma.txt index d0ec45ae4e7d..44bd766f2e5d 100644 --- a/Documentation/filesystems/nfs-rdma.txt +++ b/Documentation/filesystems/nfs-rdma.txt | |||
@@ -5,7 +5,7 @@ | |||
5 | ################################################################################ | 5 | ################################################################################ |
6 | 6 | ||
7 | Author: NetApp and Open Grid Computing | 7 | Author: NetApp and Open Grid Computing |
8 | Date: April 15, 2008 | 8 | Date: May 29, 2008 |
9 | 9 | ||
10 | Table of Contents | 10 | Table of Contents |
11 | ~~~~~~~~~~~~~~~~~ | 11 | ~~~~~~~~~~~~~~~~~ |
@@ -60,16 +60,18 @@ Installation | |||
60 | The procedures described in this document have been tested with | 60 | The procedures described in this document have been tested with |
61 | distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). | 61 | distributions from Red Hat's Fedora Project (http://fedora.redhat.com/). |
62 | 62 | ||
63 | - Install nfs-utils-1.1.1 or greater on the client | 63 | - Install nfs-utils-1.1.2 or greater on the client |
64 | 64 | ||
65 | An NFS/RDMA mount point can only be obtained by using the mount.nfs | 65 | An NFS/RDMA mount point can be obtained by using the mount.nfs command in |
66 | command in nfs-utils-1.1.1 or greater. To see which version of mount.nfs | 66 | nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils |
67 | you are using, type: | 67 | version with support for NFS/RDMA mounts, but for various reasons we |
68 | recommend using nfs-utils-1.1.2 or greater). To see which version of | ||
69 | mount.nfs you are using, type: | ||
68 | 70 | ||
69 | > /sbin/mount.nfs -V | 71 | $ /sbin/mount.nfs -V |
70 | 72 | ||
71 | If the version is less than 1.1.1 or the command does not exist, | 73 | If the version is less than 1.1.2 or the command does not exist, |
72 | then you will need to install the latest version of nfs-utils. | 74 | you should install the latest version of nfs-utils. |
73 | 75 | ||
74 | Download the latest package from: | 76 | Download the latest package from: |
75 | 77 | ||
@@ -77,22 +79,33 @@ Installation | |||
77 | 79 | ||
78 | Uncompress the package and follow the installation instructions. | 80 | Uncompress the package and follow the installation instructions. |
79 | 81 | ||
80 | If you will not be using GSS and NFSv4, the installation process | 82 | If you will not need the idmapper and gssd executables (you do not need |
81 | can be simplified by disabling these features when running configure: | 83 | these to create an NFS/RDMA enabled mount command), the installation |
84 | process can be simplified by disabling these features when running | ||
85 | configure: | ||
82 | 86 | ||
83 | > ./configure --disable-gss --disable-nfsv4 | 87 | $ ./configure --disable-gss --disable-nfsv4 |
84 | 88 | ||
85 | For more information on this see the package's README and INSTALL files. | 89 | To build nfs-utils you will need the tcp_wrappers package installed. For |
90 | more information on this see the package's README and INSTALL files. | ||
86 | 91 | ||
87 | After building the nfs-utils package, there will be a mount.nfs binary in | 92 | After building the nfs-utils package, there will be a mount.nfs binary in |
88 | the utils/mount directory. This binary can be used to initiate NFS v2, v3, | 93 | the utils/mount directory. This binary can be used to initiate NFS v2, v3, |
89 | or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4. | 94 | or v4 mounts. To initiate a v4 mount, the binary must be called |
90 | The standard technique is to create a symlink called mount.nfs4 to mount.nfs. | 95 | mount.nfs4. The standard technique is to create a symlink called |
96 | mount.nfs4 to mount.nfs. | ||
91 | 97 | ||
92 | NOTE: mount.nfs and therefore nfs-utils-1.1.1 or greater is only needed | 98 | This mount.nfs binary should be installed at /sbin/mount.nfs as follows: |
99 | |||
100 | $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs | ||
101 | |||
102 | In this location, mount.nfs will be invoked automatically for NFS mounts | ||
103 | by the system mount commmand. | ||
104 | |||
105 | NOTE: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed | ||
93 | on the NFS client machine. You do not need this specific version of | 106 | on the NFS client machine. You do not need this specific version of |
94 | nfs-utils on the server. Furthermore, only the mount.nfs command from | 107 | nfs-utils on the server. Furthermore, only the mount.nfs command from |
95 | nfs-utils-1.1.1 is needed on the client. | 108 | nfs-utils-1.1.2 is needed on the client. |
96 | 109 | ||
97 | - Install a Linux kernel with NFS/RDMA | 110 | - Install a Linux kernel with NFS/RDMA |
98 | 111 | ||
@@ -156,8 +169,8 @@ Check RDMA and NFS Setup | |||
156 | this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel | 169 | this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel |
157 | card: | 170 | card: |
158 | 171 | ||
159 | > modprobe ib_mthca | 172 | $ modprobe ib_mthca |
160 | > modprobe ib_ipoib | 173 | $ modprobe ib_ipoib |
161 | 174 | ||
162 | If you are using InfiniBand, make sure there is a Subnet Manager (SM) | 175 | If you are using InfiniBand, make sure there is a Subnet Manager (SM) |
163 | running on the network. If your IB switch has an embedded SM, you can | 176 | running on the network. If your IB switch has an embedded SM, you can |
@@ -166,7 +179,7 @@ Check RDMA and NFS Setup | |||
166 | 179 | ||
167 | If an SM is running on your network, you should see the following: | 180 | If an SM is running on your network, you should see the following: |
168 | 181 | ||
169 | > cat /sys/class/infiniband/driverX/ports/1/state | 182 | $ cat /sys/class/infiniband/driverX/ports/1/state |
170 | 4: ACTIVE | 183 | 4: ACTIVE |
171 | 184 | ||
172 | where driverX is mthca0, ipath5, ehca3, etc. | 185 | where driverX is mthca0, ipath5, ehca3, etc. |
@@ -174,10 +187,10 @@ Check RDMA and NFS Setup | |||
174 | To further test the InfiniBand software stack, use IPoIB (this | 187 | To further test the InfiniBand software stack, use IPoIB (this |
175 | assumes you have two IB hosts named host1 and host2): | 188 | assumes you have two IB hosts named host1 and host2): |
176 | 189 | ||
177 | host1> ifconfig ib0 a.b.c.x | 190 | host1$ ifconfig ib0 a.b.c.x |
178 | host2> ifconfig ib0 a.b.c.y | 191 | host2$ ifconfig ib0 a.b.c.y |
179 | host1> ping a.b.c.y | 192 | host1$ ping a.b.c.y |
180 | host2> ping a.b.c.x | 193 | host2$ ping a.b.c.x |
181 | 194 | ||
182 | For other device types, follow the appropriate procedures. | 195 | For other device types, follow the appropriate procedures. |
183 | 196 | ||
@@ -202,11 +215,11 @@ NFS/RDMA Setup | |||
202 | /vol0 192.168.0.47(fsid=0,rw,async,insecure,no_root_squash) | 215 | /vol0 192.168.0.47(fsid=0,rw,async,insecure,no_root_squash) |
203 | /vol0 192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) | 216 | /vol0 192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash) |
204 | 217 | ||
205 | The IP address(es) is(are) the client's IPoIB address for an InfiniBand HCA or the | 218 | The IP address(es) is(are) the client's IPoIB address for an InfiniBand |
206 | cleint's iWARP address(es) for an RNIC. | 219 | HCA or the cleint's iWARP address(es) for an RNIC. |
207 | 220 | ||
208 | NOTE: The "insecure" option must be used because the NFS/RDMA client does not | 221 | NOTE: The "insecure" option must be used because the NFS/RDMA client does |
209 | use a reserved port. | 222 | not use a reserved port. |
210 | 223 | ||
211 | Each time a machine boots: | 224 | Each time a machine boots: |
212 | 225 | ||
@@ -214,43 +227,45 @@ NFS/RDMA Setup | |||
214 | 227 | ||
215 | For InfiniBand using a Mellanox adapter: | 228 | For InfiniBand using a Mellanox adapter: |
216 | 229 | ||
217 | > modprobe ib_mthca | 230 | $ modprobe ib_mthca |
218 | > modprobe ib_ipoib | 231 | $ modprobe ib_ipoib |
219 | > ifconfig ib0 a.b.c.d | 232 | $ ifconfig ib0 a.b.c.d |
220 | 233 | ||
221 | NOTE: use unique addresses for the client and server | 234 | NOTE: use unique addresses for the client and server |
222 | 235 | ||
223 | - Start the NFS server | 236 | - Start the NFS server |
224 | 237 | ||
225 | If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), | 238 | If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in |
226 | load the RDMA transport module: | 239 | kernel config), load the RDMA transport module: |
227 | 240 | ||
228 | > modprobe svcrdma | 241 | $ modprobe svcrdma |
229 | 242 | ||
230 | Regardless of how the server was built (module or built-in), start the server: | 243 | Regardless of how the server was built (module or built-in), start the |
244 | server: | ||
231 | 245 | ||
232 | > /etc/init.d/nfs start | 246 | $ /etc/init.d/nfs start |
233 | 247 | ||
234 | or | 248 | or |
235 | 249 | ||
236 | > service nfs start | 250 | $ service nfs start |
237 | 251 | ||
238 | Instruct the server to listen on the RDMA transport: | 252 | Instruct the server to listen on the RDMA transport: |
239 | 253 | ||
240 | > echo rdma 2050 > /proc/fs/nfsd/portlist | 254 | $ echo rdma 2050 > /proc/fs/nfsd/portlist |
241 | 255 | ||
242 | - On the client system | 256 | - On the client system |
243 | 257 | ||
244 | If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config), | 258 | If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in |
245 | load the RDMA client module: | 259 | kernel config), load the RDMA client module: |
246 | 260 | ||
247 | > modprobe xprtrdma.ko | 261 | $ modprobe xprtrdma.ko |
248 | 262 | ||
249 | Regardless of how the client was built (module or built-in), issue the mount.nfs command: | 263 | Regardless of how the client was built (module or built-in), use this |
264 | command to mount the NFS/RDMA server: | ||
250 | 265 | ||
251 | > /path/to/your/mount.nfs <IPoIB-server-name-or-address>:/<export> /mnt -i -o rdma,port=2050 | 266 | $ mount -o rdma,port=2050 <IPoIB-server-name-or-address>:/<export> /mnt |
252 | 267 | ||
253 | To verify that the mount is using RDMA, run "cat /proc/mounts" and check the | 268 | To verify that the mount is using RDMA, run "cat /proc/mounts" and check |
254 | "proto" field for the given mount. | 269 | the "proto" field for the given mount. |
255 | 270 | ||
256 | Congratulations! You're using NFS/RDMA! | 271 | Congratulations! You're using NFS/RDMA! |