aboutsummaryrefslogtreecommitdiffstats
path: root/Documentation/networking/bonding.txt
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@ppc970.osdl.org>2005-04-16 18:20:36 -0400
committerLinus Torvalds <torvalds@ppc970.osdl.org>2005-04-16 18:20:36 -0400
commit1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 (patch)
tree0bba044c4ce775e45a88a51686b5d9f90697ea9d /Documentation/networking/bonding.txt
Linux-2.6.12-rc2v2.6.12-rc2
Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
Diffstat (limited to 'Documentation/networking/bonding.txt')
-rw-r--r--Documentation/networking/bonding.txt1618
1 files changed, 1618 insertions, 0 deletions
diff --git a/Documentation/networking/bonding.txt b/Documentation/networking/bonding.txt
new file mode 100644
index 000000000000..0bc2ed136a38
--- /dev/null
+++ b/Documentation/networking/bonding.txt
@@ -0,0 +1,1618 @@
1
2 Linux Ethernet Bonding Driver HOWTO
3
4Initial release : Thomas Davis <tadavis at lbl.gov>
5Corrections, HA extensions : 2000/10/03-15 :
6 - Willy Tarreau <willy at meta-x.org>
7 - Constantine Gavrilov <const-g at xpert.com>
8 - Chad N. Tindel <ctindel at ieee dot org>
9 - Janice Girouard <girouard at us dot ibm dot com>
10 - Jay Vosburgh <fubar at us dot ibm dot com>
11
12Reorganized and updated Feb 2005 by Jay Vosburgh
13
14Note :
15------
16
17The bonding driver originally came from Donald Becker's beowulf patches for
18kernel 2.0. It has changed quite a bit since, and the original tools from
19extreme-linux and beowulf sites will not work with this version of the driver.
20
21For new versions of the driver, patches for older kernels and the updated
22userspace tools, please follow the links at the end of this file.
23
24Table of Contents
25=================
26
271. Bonding Driver Installation
28
292. Bonding Driver Options
30
313. Configuring Bonding Devices
323.1 Configuration with sysconfig support
333.2 Configuration with initscripts support
343.3 Configuring Bonding Manually
353.4 Configuring Multiple Bonds
36
375. Querying Bonding Configuration
385.1 Bonding Configuration
395.2 Network Configuration
40
416. Switch Configuration
42
437. 802.1q VLAN Support
44
458. Link Monitoring
468.1 ARP Monitor Operation
478.2 Configuring Multiple ARP Targets
488.3 MII Monitor Operation
49
509. Potential Trouble Sources
519.1 Adventures in Routing
529.2 Ethernet Device Renaming
539.3 Painfully Slow Or No Failed Link Detection By Miimon
54
5510. SNMP agents
56
5711. Promiscuous mode
58
5912. High Availability Information
6012.1 High Availability in a Single Switch Topology
6112.1.1 Bonding Mode Selection for Single Switch Topology
6212.1.2 Link Monitoring for Single Switch Topology
6312.2 High Availability in a Multiple Switch Topology
6412.2.1 Bonding Mode Selection for Multiple Switch Topology
6512.2.2 Link Monitoring for Multiple Switch Topology
6612.3 Switch Behavior Issues for High Availability
67
6813. Hardware Specific Considerations
6913.1 IBM BladeCenter
70
7114. Frequently Asked Questions
72
7315. Resources and Links
74
75
761. Bonding Driver Installation
77==============================
78
79 Most popular distro kernels ship with the bonding driver
80already available as a module and the ifenslave user level control
81program installed and ready for use. If your distro does not, or you
82have need to compile bonding from source (e.g., configuring and
83installing a mainline kernel from kernel.org), you'll need to perform
84the following steps:
85
861.1 Configure and build the kernel with bonding
87-----------------------------------------------
88
89 The latest version of the bonding driver is available in the
90drivers/net/bonding subdirectory of the most recent kernel source
91(which is available on http://kernel.org).
92
93 Prior to the 2.4.11 kernel, the bonding driver was maintained
94largely outside the kernel tree; patches for some earlier kernels are
95available on the bonding sourceforge site, although those patches are
96still several years out of date. Most users will want to use either
97the most recent kernel from kernel.org or whatever kernel came with
98their distro.
99
100 Configure kernel with "make menuconfig" (or "make xconfig" or
101"make config"), then select "Bonding driver support" in the "Network
102device support" section. It is recommended that you configure the
103driver as module since it is currently the only way to pass parameters
104to the driver or configure more than one bonding device.
105
106 Build and install the new kernel and modules, then proceed to
107step 2.
108
1091.2 Install ifenslave Control Utility
110-------------------------------------
111
112 The ifenslave user level control program is included in the
113kernel source tree, in the file Documentation/networking/ifenslave.c.
114It is generally recommended that you use the ifenslave that
115corresponds to the kernel that you are using (either from the same
116source tree or supplied with the distro), however, ifenslave
117executables from older kernels should function (but features newer
118than the ifenslave release are not supported). Running an ifenslave
119that is newer than the kernel is not supported, and may or may not
120work.
121
122 To install ifenslave, do the following:
123
124# gcc -Wall -O -I/usr/src/linux/include ifenslave.c -o ifenslave
125# cp ifenslave /sbin/ifenslave
126
127 If your kernel source is not in "/usr/src/linux," then replace
128"/usr/src/linux/include" in the above with the location of your kernel
129source include directory.
130
131 You may wish to back up any existing /sbin/ifenslave, or, for
132testing or informal use, tag the ifenslave to the kernel version
133(e.g., name the ifenslave executable /sbin/ifenslave-2.6.10).
134
135IMPORTANT NOTE:
136
137 If you omit the "-I" or specify an incorrect directory, you
138may end up with an ifenslave that is incompatible with the kernel
139you're trying to build it for. Some distros (e.g., Red Hat from 7.1
140onwards) do not have /usr/include/linux symbolically linked to the
141default kernel source include directory.
142
143
1442. Bonding Driver Options
145=========================
146
147 Options for the bonding driver are supplied as parameters to
148the bonding module at load time. They may be given as command line
149arguments to the insmod or modprobe command, but are usually specified
150in either the /etc/modprobe.conf configuration file, or in a
151distro-specific configuration file (some of which are detailed in the
152next section).
153
154 The available bonding driver parameters are listed below. If a
155parameter is not specified the default value is used. When initially
156configuring a bond, it is recommended "tail -f /var/log/messages" be
157run in a separate window to watch for bonding driver error messages.
158
159 It is critical that either the miimon or arp_interval and
160arp_ip_target parameters be specified, otherwise serious network
161degradation will occur during link failures. Very few devices do not
162support at least miimon, so there is really no reason not to use it.
163
164 Options with textual values will accept either the text name
165 or, for backwards compatibility, the option value. E.g.,
166 "mode=802.3ad" and "mode=4" set the same mode.
167
168 The parameters are as follows:
169
170arp_interval
171
172 Specifies the ARP monitoring frequency in milli-seconds. If
173 ARP monitoring is used in a load-balancing mode (mode 0 or 2),
174 the switch should be configured in a mode that evenly
175 distributes packets across all links - such as round-robin. If
176 the switch is configured to distribute the packets in an XOR
177 fashion, all replies from the ARP targets will be received on
178 the same link which could cause the other team members to
179 fail. ARP monitoring should not be used in conjunction with
180 miimon. A value of 0 disables ARP monitoring. The default
181 value is 0.
182
183arp_ip_target
184
185 Specifies the ip addresses to use when arp_interval is > 0.
186 These are the targets of the ARP request sent to determine the
187 health of the link to the targets. Specify these values in
188 ddd.ddd.ddd.ddd format. Multiple ip adresses must be
189 seperated by a comma. At least one IP address must be given
190 for ARP monitoring to function. The maximum number of targets
191 that can be specified is 16. The default value is no IP
192 addresses.
193
194downdelay
195
196 Specifies the time, in milliseconds, to wait before disabling
197 a slave after a link failure has been detected. This option
198 is only valid for the miimon link monitor. The downdelay
199 value should be a multiple of the miimon value; if not, it
200 will be rounded down to the nearest multiple. The default
201 value is 0.
202
203lacp_rate
204
205 Option specifying the rate in which we'll ask our link partner
206 to transmit LACPDU packets in 802.3ad mode. Possible values
207 are:
208
209 slow or 0
210 Request partner to transmit LACPDUs every 30 seconds (default)
211
212 fast or 1
213 Request partner to transmit LACPDUs every 1 second
214
215max_bonds
216
217 Specifies the number of bonding devices to create for this
218 instance of the bonding driver. E.g., if max_bonds is 3, and
219 the bonding driver is not already loaded, then bond0, bond1
220 and bond2 will be created. The default value is 1.
221
222miimon
223
224 Specifies the frequency in milli-seconds that MII link
225 monitoring will occur. A value of zero disables MII link
226 monitoring. A value of 100 is a good starting point. The
227 use_carrier option, below, affects how the link state is
228 determined. See the High Availability section for additional
229 information. The default value is 0.
230
231mode
232
233 Specifies one of the bonding policies. The default is
234 balance-rr (round robin). Possible values are:
235
236 balance-rr or 0
237
238 Round-robin policy: Transmit packets in sequential
239 order from the first available slave through the
240 last. This mode provides load balancing and fault
241 tolerance.
242
243 active-backup or 1
244
245 Active-backup policy: Only one slave in the bond is
246 active. A different slave becomes active if, and only
247 if, the active slave fails. The bond's MAC address is
248 externally visible on only one port (network adapter)
249 to avoid confusing the switch. This mode provides
250 fault tolerance. The primary option affects the
251 behavior of this mode.
252
253 balance-xor or 2
254
255 XOR policy: Transmit based on [(source MAC address
256 XOR'd with destination MAC address) modulo slave
257 count]. This selects the same slave for each
258 destination MAC address. This mode provides load
259 balancing and fault tolerance.
260
261 broadcast or 3
262
263 Broadcast policy: transmits everything on all slave
264 interfaces. This mode provides fault tolerance.
265
266 802.3ad or 4
267
268 IEEE 802.3ad Dynamic link aggregation. Creates
269 aggregation groups that share the same speed and
270 duplex settings. Utilizes all slaves in the active
271 aggregator according to the 802.3ad specification.
272
273 Pre-requisites:
274
275 1. Ethtool support in the base drivers for retrieving
276 the speed and duplex of each slave.
277
278 2. A switch that supports IEEE 802.3ad Dynamic link
279 aggregation.
280
281 Most switches will require some type of configuration
282 to enable 802.3ad mode.
283
284 balance-tlb or 5
285
286 Adaptive transmit load balancing: channel bonding that
287 does not require any special switch support. The
288 outgoing traffic is distributed according to the
289 current load (computed relative to the speed) on each
290 slave. Incoming traffic is received by the current
291 slave. If the receiving slave fails, another slave
292 takes over the MAC address of the failed receiving
293 slave.
294
295 Prerequisite:
296
297 Ethtool support in the base drivers for retrieving the
298 speed of each slave.
299
300 balance-alb or 6
301
302 Adaptive load balancing: includes balance-tlb plus
303 receive load balancing (rlb) for IPV4 traffic, and
304 does not require any special switch support. The
305 receive load balancing is achieved by ARP negotiation.
306 The bonding driver intercepts the ARP Replies sent by
307 the local system on their way out and overwrites the
308 source hardware address with the unique hardware
309 address of one of the slaves in the bond such that
310 different peers use different hardware addresses for
311 the server.
312
313 Receive traffic from connections created by the server
314 is also balanced. When the local system sends an ARP
315 Request the bonding driver copies and saves the peer's
316 IP information from the ARP packet. When the ARP
317 Reply arrives from the peer, its hardware address is
318 retrieved and the bonding driver initiates an ARP
319 reply to this peer assigning it to one of the slaves
320 in the bond. A problematic outcome of using ARP
321 negotiation for balancing is that each time that an
322 ARP request is broadcast it uses the hardware address
323 of the bond. Hence, peers learn the hardware address
324 of the bond and the balancing of receive traffic
325 collapses to the current slave. This is handled by
326 sending updates (ARP Replies) to all the peers with
327 their individually assigned hardware address such that
328 the traffic is redistributed. Receive traffic is also
329 redistributed when a new slave is added to the bond
330 and when an inactive slave is re-activated. The
331 receive load is distributed sequentially (round robin)
332 among the group of highest speed slaves in the bond.
333
334 When a link is reconnected or a new slave joins the
335 bond the receive traffic is redistributed among all
336 active slaves in the bond by intiating ARP Replies
337 with the selected mac address to each of the
338 clients. The updelay parameter (detailed below) must
339 be set to a value equal or greater than the switch's
340 forwarding delay so that the ARP Replies sent to the
341 peers will not be blocked by the switch.
342
343 Prerequisites:
344
345 1. Ethtool support in the base drivers for retrieving
346 the speed of each slave.
347
348 2. Base driver support for setting the hardware
349 address of a device while it is open. This is
350 required so that there will always be one slave in the
351 team using the bond hardware address (the
352 curr_active_slave) while having a unique hardware
353 address for each slave in the bond. If the
354 curr_active_slave fails its hardware address is
355 swapped with the new curr_active_slave that was
356 chosen.
357
358primary
359
360 A string (eth0, eth2, etc) specifying which slave is the
361 primary device. The specified device will always be the
362 active slave while it is available. Only when the primary is
363 off-line will alternate devices be used. This is useful when
364 one slave is preferred over another, e.g., when one slave has
365 higher throughput than another.
366
367 The primary option is only valid for active-backup mode.
368
369updelay
370
371 Specifies the time, in milliseconds, to wait before enabling a
372 slave after a link recovery has been detected. This option is
373 only valid for the miimon link monitor. The updelay value
374 should be a multiple of the miimon value; if not, it will be
375 rounded down to the nearest multiple. The default value is 0.
376
377use_carrier
378
379 Specifies whether or not miimon should use MII or ETHTOOL
380 ioctls vs. netif_carrier_ok() to determine the link
381 status. The MII or ETHTOOL ioctls are less efficient and
382 utilize a deprecated calling sequence within the kernel. The
383 netif_carrier_ok() relies on the device driver to maintain its
384 state with netif_carrier_on/off; at this writing, most, but
385 not all, device drivers support this facility.
386
387 If bonding insists that the link is up when it should not be,
388 it may be that your network device driver does not support
389 netif_carrier_on/off. The default state for netif_carrier is
390 "carrier on," so if a driver does not support netif_carrier,
391 it will appear as if the link is always up. In this case,
392 setting use_carrier to 0 will cause bonding to revert to the
393 MII / ETHTOOL ioctl method to determine the link state.
394
395 A value of 1 enables the use of netif_carrier_ok(), a value of
396 0 will use the deprecated MII / ETHTOOL ioctls. The default
397 value is 1.
398
399
400
4013. Configuring Bonding Devices
402==============================
403
404 There are, essentially, two methods for configuring bonding:
405with support from the distro's network initialization scripts, and
406without. Distros generally use one of two packages for the network
407initialization scripts: initscripts or sysconfig. Recent versions of
408these packages have support for bonding, while older versions do not.
409
410 We will first describe the options for configuring bonding for
411distros using versions of initscripts and sysconfig with full or
412partial support for bonding, then provide information on enabling
413bonding without support from the network initialization scripts (i.e.,
414older versions of initscripts or sysconfig).
415
416 If you're unsure whether your distro uses sysconfig or
417initscripts, or don't know if it's new enough, have no fear.
418Determining this is fairly straightforward.
419
420 First, issue the command:
421
422$ rpm -qf /sbin/ifup
423
424 It will respond with a line of text starting with either
425"initscripts" or "sysconfig," followed by some numbers. This is the
426package that provides your network initialization scripts.
427
428 Next, to determine if your installation supports bonding,
429issue the command:
430
431$ grep ifenslave /sbin/ifup
432
433 If this returns any matches, then your initscripts or
434sysconfig has support for bonding.
435
4363.1 Configuration with sysconfig support
437----------------------------------------
438
439 This section applies to distros using a version of sysconfig
440with bonding support, for example, SuSE Linux Enterprise Server 9.
441
442 SuSE SLES 9's networking configuration system does support
443bonding, however, at this writing, the YaST system configuration
444frontend does not provide any means to work with bonding devices.
445Bonding devices can be managed by hand, however, as follows.
446
447 First, if they have not already been configured, configure the
448slave devices. On SLES 9, this is most easily done by running the
449yast2 sysconfig configuration utility. The goal is for to create an
450ifcfg-id file for each slave device. The simplest way to accomplish
451this is to configure the devices for DHCP. The name of the
452configuration file for each device will be of the form:
453
454ifcfg-id-xx:xx:xx:xx:xx:xx
455
456 Where the "xx" portion will be replaced with the digits from
457the device's permanent MAC address.
458
459 Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been
460created, it is necessary to edit the configuration files for the slave
461devices (the MAC addresses correspond to those of the slave devices).
462Before editing, the file will contain muliple lines, and will look
463something like this:
464
465BOOTPROTO='dhcp'
466STARTMODE='on'
467USERCTL='no'
468UNIQUE='XNzu.WeZGOGF+4wE'
469_nm_name='bus-pci-0001:61:01.0'
470
471 Change the BOOTPROTO and STARTMODE lines to the following:
472
473BOOTPROTO='none'
474STARTMODE='off'
475
476 Do not alter the UNIQUE or _nm_name lines. Remove any other
477lines (USERCTL, etc).
478
479 Once the ifcfg-id-xx:xx:xx:xx:xx:xx files have been modified,
480it's time to create the configuration file for the bonding device
481itself. This file is named ifcfg-bondX, where X is the number of the
482bonding device to create, starting at 0. The first such file is
483ifcfg-bond0, the second is ifcfg-bond1, and so on. The sysconfig
484network configuration system will correctly start multiple instances
485of bonding.
486
487 The contents of the ifcfg-bondX file is as follows:
488
489BOOTPROTO="static"
490BROADCAST="10.0.2.255"
491IPADDR="10.0.2.10"
492NETMASK="255.255.0.0"
493NETWORK="10.0.2.0"
494REMOTE_IPADDR=""
495STARTMODE="onboot"
496BONDING_MASTER="yes"
497BONDING_MODULE_OPTS="mode=active-backup miimon=100"
498BONDING_SLAVE0="eth0"
499BONDING_SLAVE1="eth1"
500
501 Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK
502values with the appropriate values for your network.
503
504 Note that configuring the bonding device with BOOTPROTO='dhcp'
505does not work; the scripts attempt to obtain the device address from
506DHCP prior to adding any of the slave devices. Without active slaves,
507the DHCP requests are not sent to the network.
508
509 The STARTMODE specifies when the device is brought online.
510The possible values are:
511
512 onboot: The device is started at boot time. If you're not
513 sure, this is probably what you want.
514
515 manual: The device is started only when ifup is called
516 manually. Bonding devices may be configured this
517 way if you do not wish them to start automatically
518 at boot for some reason.
519
520 hotplug: The device is started by a hotplug event. This is not
521 a valid choice for a bonding device.
522
523 off or ignore: The device configuration is ignored.
524
525 The line BONDING_MASTER='yes' indicates that the device is a
526bonding master device. The only useful value is "yes."
527
528 The contents of BONDING_MODULE_OPTS are supplied to the
529instance of the bonding module for this device. Specify the options
530for the bonding mode, link monitoring, and so on here. Do not include
531the max_bonds bonding parameter; this will confuse the configuration
532system if you have multiple bonding devices.
533
534 Finally, supply one BONDING_SLAVEn="ethX" for each slave,
535where "n" is an increasing value, one for each slave, and "ethX" is
536the name of the slave device (eth0, eth1, etc).
537
538 When all configuration files have been modified or created,
539networking must be restarted for the configuration changes to take
540effect. This can be accomplished via the following:
541
542# /etc/init.d/network restart
543
544 Note that the network control script (/sbin/ifdown) will
545remove the bonding module as part of the network shutdown processing,
546so it is not necessary to remove the module by hand if, e.g., the
547module paramters have changed.
548
549 Also, at this writing, YaST/YaST2 will not manage bonding
550devices (they do not show bonding interfaces on its list of network
551devices). It is necessary to edit the configuration file by hand to
552change the bonding configuration.
553
554 Additional general options and details of the ifcfg file
555format can be found in an example ifcfg template file:
556
557/etc/sysconfig/network/ifcfg.template
558
559 Note that the template does not document the various BONDING_
560settings described above, but does describe many of the other options.
561
5623.2 Configuration with initscripts support
563------------------------------------------
564
565 This section applies to distros using a version of initscripts
566with bonding support, for example, Red Hat Linux 9 or Red Hat
567Enterprise Linux version 3. On these systems, the network
568initialization scripts have some knowledge of bonding, and can be
569configured to control bonding devices.
570
571 These distros will not automatically load the network adapter
572driver unless the ethX device is configured with an IP address.
573Because of this constraint, users must manually configure a
574network-script file for all physical adapters that will be members of
575a bondX link. Network script files are located in the directory:
576
577/etc/sysconfig/network-scripts
578
579 The file name must be prefixed with "ifcfg-eth" and suffixed
580with the adapter's physical adapter number. For example, the script
581for eth0 would be named /etc/sysconfig/network-scripts/ifcfg-eth0.
582Place the following text in the file:
583
584DEVICE=eth0
585USERCTL=no
586ONBOOT=yes
587MASTER=bond0
588SLAVE=yes
589BOOTPROTO=none
590
591 The DEVICE= line will be different for every ethX device and
592must correspond with the name of the file, i.e., ifcfg-eth1 must have
593a device line of DEVICE=eth1. The setting of the MASTER= line will
594also depend on the final bonding interface name chosen for your bond.
595As with other network devices, these typically start at 0, and go up
596one for each device, i.e., the first bonding instance is bond0, the
597second is bond1, and so on.
598
599 Next, create a bond network script. The file name for this
600script will be /etc/sysconfig/network-scripts/ifcfg-bondX where X is
601the number of the bond. For bond0 the file is named "ifcfg-bond0",
602for bond1 it is named "ifcfg-bond1", and so on. Within that file,
603place the following text:
604
605DEVICE=bond0
606IPADDR=192.168.1.1
607NETMASK=255.255.255.0
608NETWORK=192.168.1.0
609BROADCAST=192.168.1.255
610ONBOOT=yes
611BOOTPROTO=none
612USERCTL=no
613
614 Be sure to change the networking specific lines (IPADDR,
615NETMASK, NETWORK and BROADCAST) to match your network configuration.
616
617 Finally, it is necessary to edit /etc/modules.conf to load the
618bonding module when the bond0 interface is brought up. The following
619sample lines in /etc/modules.conf will load the bonding module, and
620select its options:
621
622alias bond0 bonding
623options bond0 mode=balance-alb miimon=100
624
625 Replace the sample parameters with the appropriate set of
626options for your configuration.
627
628 Finally run "/etc/rc.d/init.d/network restart" as root. This
629will restart the networking subsystem and your bond link should be now
630up and running.
631
632
6333.3 Configuring Bonding Manually
634--------------------------------
635
636 This section applies to distros whose network initialization
637scripts (the sysconfig or initscripts package) do not have specific
638knowledge of bonding. One such distro is SuSE Linux Enterprise Server
639version 8.
640
641 The general methodology for these systems is to place the
642bonding module parameters into /etc/modprobe.conf, then add modprobe
643and/or ifenslave commands to the system's global init script. The
644name of the global init script differs; for sysconfig, it is
645/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local.
646
647 For example, if you wanted to make a simple bond of two e100
648devices (presumed to be eth0 and eth1), and have it persist across
649reboots, edit the appropriate file (/etc/init.d/boot.local or
650/etc/rc.d/rc.local), and add the following:
651
652modprobe bonding -obond0 mode=balance-alb miimon=100
653modprobe e100
654ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
655ifenslave bond0 eth0
656ifenslave bond0 eth1
657
658 Replace the example bonding module parameters and bond0
659network configuration (IP address, netmask, etc) with the appropriate
660values for your configuration. The above example loads the bonding
661module with the name "bond0," this simplifies the naming if multiple
662bonding modules are loaded (each successive instance of the module is
663given a different name, and the module instance names match the
664bonding interface names).
665
666 Unfortunately, this method will not provide support for the
667ifup and ifdown scripts on the bond devices. To reload the bonding
668configuration, it is necessary to run the initialization script, e.g.,
669
670# /etc/init.d/boot.local
671
672 or
673
674# /etc/rc.d/rc.local
675
676 It may be desirable in such a case to create a separate script
677which only initializes the bonding configuration, then call that
678separate script from within boot.local. This allows for bonding to be
679enabled without re-running the entire global init script.
680
681 To shut down the bonding devices, it is necessary to first
682mark the bonding device itself as being down, then remove the
683appropriate device driver modules. For our example above, you can do
684the following:
685
686# ifconfig bond0 down
687# rmmod bond0
688# rmmod e100
689
690 Again, for convenience, it may be desirable to create a script
691with these commands.
692
693
6943.4 Configuring Multiple Bonds
695------------------------------
696
697 This section contains information on configuring multiple
698bonding devices with differing options. If you require multiple
699bonding devices, but all with the same options, see the "max_bonds"
700module paramter, documented above.
701
702 To create multiple bonding devices with differing options, it
703is necessary to load the bonding driver multiple times. Note that
704current versions of the sysconfig network initialization scripts
705handle this automatically; if your distro uses these scripts, no
706special action is needed. See the section Configuring Bonding
707Devices, above, if you're not sure about your network initialization
708scripts.
709
710 To load multiple instances of the module, it is necessary to
711specify a different name for each instance (the module loading system
712requires that every loaded module, even multiple instances of the same
713module, have a unique name). This is accomplished by supplying
714multiple sets of bonding options in /etc/modprobe.conf, for example:
715
716alias bond0 bonding
717options bond0 -o bond0 mode=balance-rr miimon=100
718
719alias bond1 bonding
720options bond1 -o bond1 mode=balance-alb miimon=50
721
722 will load the bonding module two times. The first instance is
723named "bond0" and creates the bond0 device in balance-rr mode with an
724miimon of 100. The second instance is named "bond1" and creates the
725bond1 device in balance-alb mode with an miimon of 50.
726
727 This may be repeated any number of times, specifying a new and
728unique name in place of bond0 or bond1 for each instance.
729
730 When the appropriate module paramters are in place, then
731configure bonding according to the instructions for your distro.
732
7335. Querying Bonding Configuration
734=================================
735
7365.1 Bonding Configuration
737-------------------------
738
739 Each bonding device has a read-only file residing in the
740/proc/net/bonding directory. The file contents include information
741about the bonding configuration, options and state of each slave.
742
743 For example, the contents of /proc/net/bonding/bond0 after the
744driver is loaded with parameters of mode=0 and miimon=1000 is
745generally as follows:
746
747 Ethernet Channel Bonding Driver: 2.6.1 (October 29, 2004)
748 Bonding Mode: load balancing (round-robin)
749 Currently Active Slave: eth0
750 MII Status: up
751 MII Polling Interval (ms): 1000
752 Up Delay (ms): 0
753 Down Delay (ms): 0
754
755 Slave Interface: eth1
756 MII Status: up
757 Link Failure Count: 1
758
759 Slave Interface: eth0
760 MII Status: up
761 Link Failure Count: 1
762
763 The precise format and contents will change depending upon the
764bonding configuration, state, and version of the bonding driver.
765
7665.2 Network configuration
767-------------------------
768
769 The network configuration can be inspected using the ifconfig
770command. Bonding devices will have the MASTER flag set; Bonding slave
771devices will have the SLAVE flag set. The ifconfig output does not
772contain information on which slaves are associated with which masters.
773
774 In the example below, the bond0 interface is the master
775(MASTER) while eth0 and eth1 are slaves (SLAVE). Notice all slaves of
776bond0 have the same MAC address (HWaddr) as bond0 for all modes except
777TLB and ALB that require a unique MAC address for each slave.
778
779# /sbin/ifconfig
780bond0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
781 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
782 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
783 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
784 TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
785 collisions:0 txqueuelen:0
786
787eth0 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
788 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
789 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
790 RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
791 TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
792 collisions:0 txqueuelen:100
793 Interrupt:10 Base address:0x1080
794
795eth1 Link encap:Ethernet HWaddr 00:C0:F0:1F:37:B4
796 inet addr:XXX.XXX.XXX.YYY Bcast:XXX.XXX.XXX.255 Mask:255.255.252.0
797 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
798 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
799 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
800 collisions:0 txqueuelen:100
801 Interrupt:9 Base address:0x1400
802
8036. Switch Configuration
804=======================
805
806 For this section, "switch" refers to whatever system the
807bonded devices are directly connected to (i.e., where the other end of
808the cable plugs into). This may be an actual dedicated switch device,
809or it may be another regular system (e.g., another computer running
810Linux),
811
812 The active-backup, balance-tlb and balance-alb modes do not
813require any specific configuration of the switch.
814
815 The 802.3ad mode requires that the switch have the appropriate
816ports configured as an 802.3ad aggregation. The precise method used
817to configure this varies from switch to switch, but, for example, a
818Cisco 3550 series switch requires that the appropriate ports first be
819grouped together in a single etherchannel instance, then that
820etherchannel is set to mode "lacp" to enable 802.3ad (instead of
821standard EtherChannel).
822
823 The balance-rr, balance-xor and broadcast modes generally
824require that the switch have the appropriate ports grouped together.
825The nomenclature for such a group differs between switches, it may be
826called an "etherchannel" (as in the Cisco example, above), a "trunk
827group" or some other similar variation. For these modes, each switch
828will also have its own configuration options for the switch's transmit
829policy to the bond. Typical choices include XOR of either the MAC or
830IP addresses. The transmit policy of the two peers does not need to
831match. For these three modes, the bonding mode really selects a
832transmit policy for an EtherChannel group; all three will interoperate
833with another EtherChannel group.
834
835
8367. 802.1q VLAN Support
837======================
838
839 It is possible to configure VLAN devices over a bond interface
840using the 8021q driver. However, only packets coming from the 8021q
841driver and passing through bonding will be tagged by default. Self
842generated packets, for example, bonding's learning packets or ARP
843packets generated by either ALB mode or the ARP monitor mechanism, are
844tagged internally by bonding itself. As a result, bonding must
845"learn" the VLAN IDs configured above it, and use those IDs to tag
846self generated packets.
847
848 For reasons of simplicity, and to support the use of adapters
849that can do VLAN hardware acceleration offloding, the bonding
850interface declares itself as fully hardware offloaing capable, it gets
851the add_vid/kill_vid notifications to gather the necessary
852information, and it propagates those actions to the slaves. In case
853of mixed adapter types, hardware accelerated tagged packets that
854should go through an adapter that is not offloading capable are
855"un-accelerated" by the bonding driver so the VLAN tag sits in the
856regular location.
857
858 VLAN interfaces *must* be added on top of a bonding interface
859only after enslaving at least one slave. The bonding interface has a
860hardware address of 00:00:00:00:00:00 until the first slave is added.
861If the VLAN interface is created prior to the first enslavement, it
862would pick up the all-zeroes hardware address. Once the first slave
863is attached to the bond, the bond device itself will pick up the
864slave's hardware address, which is then available for the VLAN device.
865
866 Also, be aware that a similar problem can occur if all slaves
867are released from a bond that still has one or more VLAN interfaces on
868top of it. When a new slave is added, the bonding interface will
869obtain its hardware address from the first slave, which might not
870match the hardware address of the VLAN interfaces (which was
871ultimately copied from an earlier slave).
872
873 There are two methods to insure that the VLAN device operates
874with the correct hardware address if all slaves are removed from a
875bond interface:
876
877 1. Remove all VLAN interfaces then recreate them
878
879 2. Set the bonding interface's hardware address so that it
880matches the hardware address of the VLAN interfaces.
881
882 Note that changing a VLAN interface's HW address would set the
883underlying device -- i.e. the bonding interface -- to promiscouos
884mode, which might not be what you want.
885
886
8878. Link Monitoring
888==================
889
890 The bonding driver at present supports two schemes for
891monitoring a slave device's link state: the ARP monitor and the MII
892monitor.
893
894 At the present time, due to implementation restrictions in the
895bonding driver itself, it is not possible to enable both ARP and MII
896monitoring simultaneously.
897
8988.1 ARP Monitor Operation
899-------------------------
900
901 The ARP monitor operates as its name suggests: it sends ARP
902queries to one or more designated peer systems on the network, and
903uses the response as an indication that the link is operating. This
904gives some assurance that traffic is actually flowing to and from one
905or more peers on the local network.
906
907 The ARP monitor relies on the device driver itself to verify
908that traffic is flowing. In particular, the driver must keep up to
909date the last receive time, dev->last_rx, and transmit start time,
910dev->trans_start. If these are not updated by the driver, then the
911ARP monitor will immediately fail any slaves using that driver, and
912those slaves will stay down. If networking monitoring (tcpdump, etc)
913shows the ARP requests and replies on the network, then it may be that
914your device driver is not updating last_rx and trans_start.
915
9168.2 Configuring Multiple ARP Targets
917------------------------------------
918
919 While ARP monitoring can be done with just one target, it can
920be useful in a High Availability setup to have several targets to
921monitor. In the case of just one target, the target itself may go
922down or have a problem making it unresponsive to ARP requests. Having
923an additional target (or several) increases the reliability of the ARP
924monitoring.
925
926 Multiple ARP targets must be seperated by commas as follows:
927
928# example options for ARP monitoring with three targets
929alias bond0 bonding
930options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9
931
932 For just a single target the options would resemble:
933
934# example options for ARP monitoring with one target
935alias bond0 bonding
936options bond0 arp_interval=60 arp_ip_target=192.168.0.100
937
938
9398.3 MII Monitor Operation
940-------------------------
941
942 The MII monitor monitors only the carrier state of the local
943network interface. It accomplishes this in one of three ways: by
944depending upon the device driver to maintain its carrier state, by
945querying the device's MII registers, or by making an ethtool query to
946the device.
947
948 If the use_carrier module parameter is 1 (the default value),
949then the MII monitor will rely on the driver for carrier state
950information (via the netif_carrier subsystem). As explained in the
951use_carrier parameter information, above, if the MII monitor fails to
952detect carrier loss on the device (e.g., when the cable is physically
953disconnected), it may be that the driver does not support
954netif_carrier.
955
956 If use_carrier is 0, then the MII monitor will first query the
957device's (via ioctl) MII registers and check the link state. If that
958request fails (not just that it returns carrier down), then the MII
959monitor will make an ethtool ETHOOL_GLINK request to attempt to obtain
960the same information. If both methods fail (i.e., the driver either
961does not support or had some error in processing both the MII register
962and ethtool requests), then the MII monitor will assume the link is
963up.
964
9659. Potential Sources of Trouble
966===============================
967
9689.1 Adventures in Routing
969-------------------------
970
971 When bonding is configured, it is important that the slave
972devices not have routes that supercede routes of the master (or,
973generally, not have routes at all). For example, suppose the bonding
974device bond0 has two slaves, eth0 and eth1, and the routing table is
975as follows:
976
977Kernel IP routing table
978Destination Gateway Genmask Flags MSS Window irtt Iface
97910.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth0
98010.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 eth1
98110.0.0.0 0.0.0.0 255.255.0.0 U 40 0 0 bond0
982127.0.0.0 0.0.0.0 255.0.0.0 U 40 0 0 lo
983
984 This routing configuration will likely still update the
985receive/transmit times in the driver (needed by the ARP monitor), but
986may bypass the bonding driver (because outgoing traffic to, in this
987case, another host on network 10 would use eth0 or eth1 before bond0).
988
989 The ARP monitor (and ARP itself) may become confused by this
990configuration, because ARP requests (generated by the ARP monitor)
991will be sent on one interface (bond0), but the corresponding reply
992will arrive on a different interface (eth0). This reply looks to ARP
993as an unsolicited ARP reply (because ARP matches replies on an
994interface basis), and is discarded. The MII monitor is not affected
995by the state of the routing table.
996
997 The solution here is simply to insure that slaves do not have
998routes of their own, and if for some reason they must, those routes do
999not supercede routes of their master. This should generally be the
1000case, but unusual configurations or errant manual or automatic static
1001route additions may cause trouble.
1002
10039.2 Ethernet Device Renaming
1004----------------------------
1005
1006 On systems with network configuration scripts that do not
1007associate physical devices directly with network interface names (so
1008that the same physical device always has the same "ethX" name), it may
1009be necessary to add some special logic to either /etc/modules.conf or
1010/etc/modprobe.conf (depending upon which is installed on the system).
1011
1012 For example, given a modules.conf containing the following:
1013
1014alias bond0 bonding
1015options bond0 mode=some-mode miimon=50
1016alias eth0 tg3
1017alias eth1 tg3
1018alias eth2 e1000
1019alias eth3 e1000
1020
1021 If neither eth0 and eth1 are slaves to bond0, then when the
1022bond0 interface comes up, the devices may end up reordered. This
1023happens because bonding is loaded first, then its slave device's
1024drivers are loaded next. Since no other drivers have been loaded,
1025when the e1000 driver loads, it will receive eth0 and eth1 for its
1026devices, but the bonding configuration tries to enslave eth2 and eth3
1027(which may later be assigned to the tg3 devices).
1028
1029 Adding the following:
1030
1031add above bonding e1000 tg3
1032
1033 causes modprobe to load e1000 then tg3, in that order, when
1034bonding is loaded. This command is fully documented in the
1035modules.conf manual page.
1036
1037 On systems utilizing modprobe.conf (or modprobe.conf.local),
1038an equivalent problem can occur. In this case, the following can be
1039added to modprobe.conf (or modprobe.conf.local, as appropriate), as
1040follows (all on one line; it has been split here for clarity):
1041
1042install bonding /sbin/modprobe tg3; /sbin/modprobe e1000;
1043 /sbin/modprobe --ignore-install bonding
1044
1045 This will, when loading the bonding module, rather than
1046performing the normal action, instead execute the provided command.
1047This command loads the device drivers in the order needed, then calls
1048modprobe with --ingore-install to cause the normal action to then take
1049place. Full documentation on this can be found in the modprobe.conf
1050and modprobe manual pages.
1051
10529.3. Painfully Slow Or No Failed Link Detection By Miimon
1053---------------------------------------------------------
1054
1055 By default, bonding enables the use_carrier option, which
1056instructs bonding to trust the driver to maintain carrier state.
1057
1058 As discussed in the options section, above, some drivers do
1059not support the netif_carrier_on/_off link state tracking system.
1060With use_carrier enabled, bonding will always see these links as up,
1061regardless of their actual state.
1062
1063 Additionally, other drivers do support netif_carrier, but do
1064not maintain it in real time, e.g., only polling the link state at
1065some fixed interval. In this case, miimon will detect failures, but
1066only after some long period of time has expired. If it appears that
1067miimon is very slow in detecting link failures, try specifying
1068use_carrier=0 to see if that improves the failure detection time. If
1069it does, then it may be that the driver checks the carrier state at a
1070fixed interval, but does not cache the MII register values (so the
1071use_carrier=0 method of querying the registers directly works). If
1072use_carrier=0 does not improve the failover, then the driver may cache
1073the registers, or the problem may be elsewhere.
1074
1075 Also, remember that miimon only checks for the device's
1076carrier state. It has no way to determine the state of devices on or
1077beyond other ports of a switch, or if a switch is refusing to pass
1078traffic while still maintaining carrier on.
1079
108010. SNMP agents
1081===============
1082
1083 If running SNMP agents, the bonding driver should be loaded
1084before any network drivers participating in a bond. This requirement
1085is due to the the interface index (ipAdEntIfIndex) being associated to
1086the first interface found with a given IP address. That is, there is
1087only one ipAdEntIfIndex for each IP address. For example, if eth0 and
1088eth1 are slaves of bond0 and the driver for eth0 is loaded before the
1089bonding driver, the interface for the IP address will be associated
1090with the eth0 interface. This configuration is shown below, the IP
1091address 192.168.1.1 has an interface index of 2 which indexes to eth0
1092in the ifDescr table (ifDescr.2).
1093
1094 interfaces.ifTable.ifEntry.ifDescr.1 = lo
1095 interfaces.ifTable.ifEntry.ifDescr.2 = eth0
1096 interfaces.ifTable.ifEntry.ifDescr.3 = eth1
1097 interfaces.ifTable.ifEntry.ifDescr.4 = eth2
1098 interfaces.ifTable.ifEntry.ifDescr.5 = eth3
1099 interfaces.ifTable.ifEntry.ifDescr.6 = bond0
1100 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5
1101 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
1102 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4
1103 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
1104
1105 This problem is avoided by loading the bonding driver before
1106any network drivers participating in a bond. Below is an example of
1107loading the bonding driver first, the IP address 192.168.1.1 is
1108correctly associated with ifDescr.2.
1109
1110 interfaces.ifTable.ifEntry.ifDescr.1 = lo
1111 interfaces.ifTable.ifEntry.ifDescr.2 = bond0
1112 interfaces.ifTable.ifEntry.ifDescr.3 = eth0
1113 interfaces.ifTable.ifEntry.ifDescr.4 = eth1
1114 interfaces.ifTable.ifEntry.ifDescr.5 = eth2
1115 interfaces.ifTable.ifEntry.ifDescr.6 = eth3
1116 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6
1117 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
1118 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5
1119 ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
1120
1121 While some distributions may not report the interface name in
1122ifDescr, the association between the IP address and IfIndex remains
1123and SNMP functions such as Interface_Scan_Next will report that
1124association.
1125
112611. Promiscuous mode
1127====================
1128
1129 When running network monitoring tools, e.g., tcpdump, it is
1130common to enable promiscuous mode on the device, so that all traffic
1131is seen (instead of seeing only traffic destined for the local host).
1132The bonding driver handles promiscuous mode changes to the bonding
1133master device (e.g., bond0), and propogates the setting to the slave
1134devices.
1135
1136 For the balance-rr, balance-xor, broadcast, and 802.3ad modes,
1137the promiscuous mode setting is propogated to all slaves.
1138
1139 For the active-backup, balance-tlb and balance-alb modes, the
1140promiscuous mode setting is propogated only to the active slave.
1141
1142 For balance-tlb mode, the active slave is the slave currently
1143receiving inbound traffic.
1144
1145 For balance-alb mode, the active slave is the slave used as a
1146"primary." This slave is used for mode-specific control traffic, for
1147sending to peers that are unassigned or if the load is unbalanced.
1148
1149 For the active-backup, balance-tlb and balance-alb modes, when
1150the active slave changes (e.g., due to a link failure), the
1151promiscuous setting will be propogated to the new active slave.
1152
115312. High Availability Information
1154=================================
1155
1156 High Availability refers to configurations that provide
1157maximum network availability by having redundant or backup devices,
1158links and switches between the host and the rest of the world.
1159
1160 There are currently two basic methods for configuring to
1161maximize availability. They are dependent on the network topology and
1162the primary goal of the configuration, but in general, a configuration
1163can be optimized for maximum available bandwidth, or for maximum
1164network availability.
1165
116612.1 High Availability in a Single Switch Topology
1167--------------------------------------------------
1168
1169 If two hosts (or a host and a switch) are directly connected
1170via multiple physical links, then there is no network availability
1171penalty for optimizing for maximum bandwidth: there is only one switch
1172(or peer), so if it fails, you have no alternative access to fail over
1173to.
1174
1175Example 1 : host to switch (or other host)
1176
1177 +----------+ +----------+
1178 | |eth0 eth0| switch |
1179 | Host A +--------------------------+ or |
1180 | +--------------------------+ other |
1181 | |eth1 eth1| host |
1182 +----------+ +----------+
1183
1184
118512.1.1 Bonding Mode Selection for single switch topology
1186--------------------------------------------------------
1187
1188 This configuration is the easiest to set up and to understand,
1189although you will have to decide which bonding mode best suits your
1190needs. The tradeoffs for each mode are detailed below:
1191
1192balance-rr: This mode is the only mode that will permit a single
1193 TCP/IP connection to stripe traffic across multiple
1194 interfaces. It is therefore the only mode that will allow a
1195 single TCP/IP stream to utilize more than one interface's
1196 worth of throughput. This comes at a cost, however: the
1197 striping often results in peer systems receiving packets out
1198 of order, causing TCP/IP's congestion control system to kick
1199 in, often by retransmitting segments.
1200
1201 It is possible to adjust TCP/IP's congestion limits by
1202 altering the net.ipv4.tcp_reordering sysctl parameter. The
1203 usual default value is 3, and the maximum useful value is 127.
1204 For a four interface balance-rr bond, expect that a single
1205 TCP/IP stream will utilize no more than approximately 2.3
1206 interface's worth of throughput, even after adjusting
1207 tcp_reordering.
1208
1209 If you are utilizing protocols other than TCP/IP, UDP for
1210 example, and your application can tolerate out of order
1211 delivery, then this mode can allow for single stream datagram
1212 performance that scales near linearly as interfaces are added
1213 to the bond.
1214
1215 This mode requires the switch to have the appropriate ports
1216 configured for "etherchannel" or "trunking."
1217
1218active-backup: There is not much advantage in this network topology to
1219 the active-backup mode, as the inactive backup devices are all
1220 connected to the same peer as the primary. In this case, a
1221 load balancing mode (with link monitoring) will provide the
1222 same level of network availability, but with increased
1223 available bandwidth. On the plus side, it does not require
1224 any configuration of the switch.
1225
1226balance-xor: This mode will limit traffic such that packets destined
1227 for specific peers will always be sent over the same
1228 interface. Since the destination is determined by the MAC
1229 addresses involved, this may be desirable if you have a large
1230 network with many hosts. It is likely to be suboptimal if all
1231 your traffic is passed through a single router, however. As
1232 with balance-rr, the switch ports need to be configured for
1233 "etherchannel" or "trunking."
1234
1235broadcast: Like active-backup, there is not much advantage to this
1236 mode in this type of network topology.
1237
1238802.3ad: This mode can be a good choice for this type of network
1239 topology. The 802.3ad mode is an IEEE standard, so all peers
1240 that implement 802.3ad should interoperate well. The 802.3ad
1241 protocol includes automatic configuration of the aggregates,
1242 so minimal manual configuration of the switch is needed
1243 (typically only to designate that some set of devices is
1244 usable for 802.3ad). The 802.3ad standard also mandates that
1245 frames be delivered in order (within certain limits), so in
1246 general single connections will not see misordering of
1247 packets. The 802.3ad mode does have some drawbacks: the
1248 standard mandates that all devices in the aggregate operate at
1249 the same speed and duplex. Also, as with all bonding load
1250 balance modes other than balance-rr, no single connection will
1251 be able to utilize more than a single interface's worth of
1252 bandwidth. Additionally, the linux bonding 802.3ad
1253 implementation distributes traffic by peer (using an XOR of
1254 MAC addresses), so in general all traffic to a particular
1255 destination will use the same interface. Finally, the 802.3ad
1256 mode mandates the use of the MII monitor, therefore, the ARP
1257 monitor is not available in this mode.
1258
1259balance-tlb: This mode is also a good choice for this type of
1260 topology. It has no special switch configuration
1261 requirements, and balances outgoing traffic by peer, in a
1262 vaguely intelligent manner (not a simple XOR as in balance-xor
1263 or 802.3ad mode), so that unlucky MAC addresses will not all
1264 "bunch up" on a single interface. Interfaces may be of
1265 differing speeds. On the down side, in this mode all incoming
1266 traffic arrives over a single interface, this mode requires
1267 certain ethtool support in the network device driver of the
1268 slave interfaces, and the ARP monitor is not available.
1269
1270balance-alb: This mode is everything that balance-tlb is, and more. It
1271 has all of the features (and restrictions) of balance-tlb, and
1272 will also balance incoming traffic from peers (as described in
1273 the Bonding Module Options section, above). The only extra
1274 down side to this mode is that the network device driver must
1275 support changing the hardware address while the device is
1276 open.
1277
127812.1.2 Link Monitoring for Single Switch Topology
1279-------------------------------------------------
1280
1281 The choice of link monitoring may largely depend upon which
1282mode you choose to use. The more advanced load balancing modes do not
1283support the use of the ARP monitor, and are thus restricted to using
1284the MII monitor (which does not provide as high a level of assurance
1285as the ARP monitor).
1286
1287
128812.2 High Availability in a Multiple Switch Topology
1289----------------------------------------------------
1290
1291 With multiple switches, the configuration of bonding and the
1292network changes dramatically. In multiple switch topologies, there is
1293a tradeoff between network availability and usable bandwidth.
1294
1295 Below is a sample network, configured to maximize the
1296availability of the network:
1297
1298 | |
1299 |port3 port3|
1300 +-----+----+ +-----+----+
1301 | |port2 ISL port2| |
1302 | switch A +--------------------------+ switch B |
1303 | | | |
1304 +-----+----+ +-----++---+
1305 |port1 port1|
1306 | +-------+ |
1307 +-------------+ host1 +---------------+
1308 eth0 +-------+ eth1
1309
1310 In this configuration, there is a link between the two
1311switches (ISL, or inter switch link), and multiple ports connecting to
1312the outside world ("port3" on each switch). There is no technical
1313reason that this could not be extended to a third switch.
1314
131512.2.1 Bonding Mode Selection for Multiple Switch Topology
1316----------------------------------------------------------
1317
1318 In a topology such as this, the active-backup and broadcast
1319modes are the only useful bonding modes; the other modes require all
1320links to terminate on the same peer for them to behave rationally.
1321
1322active-backup: This is generally the preferred mode, particularly if
1323 the switches have an ISL and play together well. If the
1324 network configuration is such that one switch is specifically
1325 a backup switch (e.g., has lower capacity, higher cost, etc),
1326 then the primary option can be used to insure that the
1327 preferred link is always used when it is available.
1328
1329broadcast: This mode is really a special purpose mode, and is suitable
1330 only for very specific needs. For example, if the two
1331 switches are not connected (no ISL), and the networks beyond
1332 them are totally independant. In this case, if it is
1333 necessary for some specific one-way traffic to reach both
1334 independent networks, then the broadcast mode may be suitable.
1335
133612.2.2 Link Monitoring Selection for Multiple Switch Topology
1337-------------------------------------------------------------
1338
1339 The choice of link monitoring ultimately depends upon your
1340switch. If the switch can reliably fail ports in response to other
1341failures, then either the MII or ARP monitors should work. For
1342example, in the above example, if the "port3" link fails at the remote
1343end, the MII monitor has no direct means to detect this. The ARP
1344monitor could be configured with a target at the remote end of port3,
1345thus detecting that failure without switch support.
1346
1347 In general, however, in a multiple switch topology, the ARP
1348monitor can provide a higher level of reliability in detecting link
1349failures. Additionally, it should be configured with multiple targets
1350(at least one for each switch in the network). This will insure that,
1351regardless of which switch is active, the ARP monitor has a suitable
1352target to query.
1353
1354
135512.3 Switch Behavior Issues for High Availability
1356-------------------------------------------------
1357
1358 You may encounter issues with the timing of link up and down
1359reporting by the switch.
1360
1361 First, when a link comes up, some switches may indicate that
1362the link is up (carrier available), but not pass traffic over the
1363interface for some period of time. This delay is typically due to
1364some type of autonegotiation or routing protocol, but may also occur
1365during switch initialization (e.g., during recovery after a switch
1366failure). If you find this to be a problem, specify an appropriate
1367value to the updelay bonding module option to delay the use of the
1368relevant interface(s).
1369
1370 Second, some switches may "bounce" the link state one or more
1371times while a link is changing state. This occurs most commonly while
1372the switch is initializing. Again, an appropriate updelay value may
1373help, but note that if all links are down, then updelay is ignored
1374when any link becomes active (the slave closest to completing its
1375updelay is chosen).
1376
1377 Note that when a bonding interface has no active links, the
1378driver will immediately reuse the first link that goes up, even if
1379updelay parameter was specified. If there are slave interfaces
1380waiting for the updelay timeout to expire, the interface that first
1381went into that state will be immediately reused. This reduces down
1382time of the network if the value of updelay has been overestimated.
1383
1384 In addition to the concerns about switch timings, if your
1385switches take a long time to go into backup mode, it may be desirable
1386to not activate a backup interface immediately after a link goes down.
1387Failover may be delayed via the downdelay bonding module option.
1388
138913. Hardware Specific Considerations
1390====================================
1391
1392 This section contains additional information for configuring
1393bonding on specific hardware platforms, or for interfacing bonding
1394with particular switches or other devices.
1395
139613.1 IBM BladeCenter
1397--------------------
1398
1399 This applies to the JS20 and similar systems.
1400
1401 On the JS20 blades, the bonding driver supports only
1402balance-rr, active-backup, balance-tlb and balance-alb modes. This is
1403largely due to the network topology inside the BladeCenter, detailed
1404below.
1405
1406JS20 network adapter information
1407--------------------------------
1408
1409 All JS20s come with two Broadcom Gigabit Ethernet ports
1410integrated on the planar. In the BladeCenter chassis, the eth0 port
1411of all JS20 blades is hard wired to I/O Module #1; similarly, all eth1
1412ports are wired to I/O Module #2. An add-on Broadcom daughter card
1413can be installed on a JS20 to provide two more Gigabit Ethernet ports.
1414These ports, eth2 and eth3, are wired to I/O Modules 3 and 4,
1415respectively.
1416
1417 Each I/O Module may contain either a switch or a passthrough
1418module (which allows ports to be directly connected to an external
1419switch). Some bonding modes require a specific BladeCenter internal
1420network topology in order to function; these are detailed below.
1421
1422 Additional BladeCenter-specific networking information can be
1423found in two IBM Redbooks (www.ibm.com/redbooks):
1424
1425"IBM eServer BladeCenter Networking Options"
1426"IBM eServer BladeCenter Layer 2-7 Network Switching"
1427
1428BladeCenter networking configuration
1429------------------------------------
1430
1431 Because a BladeCenter can be configured in a very large number
1432of ways, this discussion will be confined to describing basic
1433configurations.
1434
1435 Normally, Ethernet Switch Modules (ESM) are used in I/O
1436modules 1 and 2. In this configuration, the eth0 and eth1 ports of a
1437JS20 will be connected to different internal switches (in the
1438respective I/O modules).
1439
1440 An optical passthru module (OPM) connects the I/O module
1441directly to an external switch. By using OPMs in I/O module #1 and
1442#2, the eth0 and eth1 interfaces of a JS20 can be redirected to the
1443outside world and connected to a common external switch.
1444
1445 Depending upon the mix of ESM and OPM modules, the network
1446will appear to bonding as either a single switch topology (all OPM
1447modules) or as a multiple switch topology (one or more ESM modules,
1448zero or more OPM modules). It is also possible to connect ESM modules
1449together, resulting in a configuration much like the example in "High
1450Availability in a multiple switch topology."
1451
1452Requirements for specifc modes
1453------------------------------
1454
1455 The balance-rr mode requires the use of OPM modules for
1456devices in the bond, all connected to an common external switch. That
1457switch must be configured for "etherchannel" or "trunking" on the
1458appropriate ports, as is usual for balance-rr.
1459
1460 The balance-alb and balance-tlb modes will function with
1461either switch modules or passthrough modules (or a mix). The only
1462specific requirement for these modes is that all network interfaces
1463must be able to reach all destinations for traffic sent over the
1464bonding device (i.e., the network must converge at some point outside
1465the BladeCenter).
1466
1467 The active-backup mode has no additional requirements.
1468
1469Link monitoring issues
1470----------------------
1471
1472 When an Ethernet Switch Module is in place, only the ARP
1473monitor will reliably detect link loss to an external switch. This is
1474nothing unusual, but examination of the BladeCenter cabinet would
1475suggest that the "external" network ports are the ethernet ports for
1476the system, when it fact there is a switch between these "external"
1477ports and the devices on the JS20 system itself. The MII monitor is
1478only able to detect link failures between the ESM and the JS20 system.
1479
1480 When a passthrough module is in place, the MII monitor does
1481detect failures to the "external" port, which is then directly
1482connected to the JS20 system.
1483
1484Other concerns
1485--------------
1486
1487 The Serial Over LAN link is established over the primary
1488ethernet (eth0) only, therefore, any loss of link to eth0 will result
1489in losing your SoL connection. It will not fail over with other
1490network traffic.
1491
1492 It may be desirable to disable spanning tree on the switch
1493(either the internal Ethernet Switch Module, or an external switch) to
1494avoid fail-over delays issues when using bonding.
1495
1496
149714. Frequently Asked Questions
1498==============================
1499
15001. Is it SMP safe?
1501
1502 Yes. The old 2.0.xx channel bonding patch was not SMP safe.
1503The new driver was designed to be SMP safe from the start.
1504
15052. What type of cards will work with it?
1506
1507 Any Ethernet type cards (you can even mix cards - a Intel
1508EtherExpress PRO/100 and a 3com 3c905b, for example). They need not
1509be of the same speed.
1510
15113. How many bonding devices can I have?
1512
1513 There is no limit.
1514
15154. How many slaves can a bonding device have?
1516
1517 This is limited only by the number of network interfaces Linux
1518supports and/or the number of network cards you can place in your
1519system.
1520
15215. What happens when a slave link dies?
1522
1523 If link monitoring is enabled, then the failing device will be
1524disabled. The active-backup mode will fail over to a backup link, and
1525other modes will ignore the failed link. The link will continue to be
1526monitored, and should it recover, it will rejoin the bond (in whatever
1527manner is appropriate for the mode). See the section on High
1528Availability for additional information.
1529
1530 Link monitoring can be enabled via either the miimon or
1531arp_interval paramters (described in the module paramters section,
1532above). In general, miimon monitors the carrier state as sensed by
1533the underlying network device, and the arp monitor (arp_interval)
1534monitors connectivity to another host on the local network.
1535
1536 If no link monitoring is configured, the bonding driver will
1537be unable to detect link failures, and will assume that all links are
1538always available. This will likely result in lost packets, and a
1539resulting degredation of performance. The precise performance loss
1540depends upon the bonding mode and network configuration.
1541
15426. Can bonding be used for High Availability?
1543
1544 Yes. See the section on High Availability for details.
1545
15467. Which switches/systems does it work with?
1547
1548 The full answer to this depends upon the desired mode.
1549
1550 In the basic balance modes (balance-rr and balance-xor), it
1551works with any system that supports etherchannel (also called
1552trunking). Most managed switches currently available have such
1553support, and many unmananged switches as well.
1554
1555 The advanced balance modes (balance-tlb and balance-alb) do
1556not have special switch requirements, but do need device drivers that
1557support specific features (described in the appropriate section under
1558module paramters, above).
1559
1560 In 802.3ad mode, it works with with systems that support IEEE
1561802.3ad Dynamic Link Aggregation. Most managed and many unmanaged
1562switches currently available support 802.3ad.
1563
1564 The active-backup mode should work with any Layer-II switch.
1565
15668. Where does a bonding device get its MAC address from?
1567
1568 If not explicitly configured with ifconfig, the MAC address of
1569the bonding device is taken from its first slave device. This MAC
1570address is then passed to all following slaves and remains persistent
1571(even if the the first slave is removed) until the bonding device is
1572brought down or reconfigured.
1573
1574 If you wish to change the MAC address, you can set it with
1575ifconfig:
1576
1577# ifconfig bond0 hw ether 00:11:22:33:44:55
1578
1579 The MAC address can be also changed by bringing down/up the
1580device and then changing its slaves (or their order):
1581
1582# ifconfig bond0 down ; modprobe -r bonding
1583# ifconfig bond0 .... up
1584# ifenslave bond0 eth...
1585
1586 This method will automatically take the address from the next
1587slave that is added.
1588
1589 To restore your slaves' MAC addresses, you need to detach them
1590from the bond (`ifenslave -d bond0 eth0'). The bonding driver will
1591then restore the MAC addresses that the slaves had before they were
1592enslaved.
1593
159415. Resources and Links
1595=======================
1596
1597The latest version of the bonding driver can be found in the latest
1598version of the linux kernel, found on http://kernel.org
1599
1600Discussions regarding the bonding driver take place primarily on the
1601bonding-devel mailing list, hosted at sourceforge.net. If you have
1602questions or problems, post them to the list.
1603
1604bonding-devel@lists.sourceforge.net
1605
1606https://lists.sourceforge.net/lists/listinfo/bonding-devel
1607
1608There is also a project site on sourceforge.
1609
1610http://www.sourceforge.net/projects/bonding
1611
1612Donald Becker's Ethernet Drivers and diag programs may be found at :
1613 - http://www.scyld.com/network/
1614
1615You will also find a lot of information regarding Ethernet, NWay, MII,
1616etc. at www.scyld.com.
1617
1618-- END --