1 files changed, 1618 insertions, 0 deletions

diff --git a/Documentation/networking/bonding.txt b/Documentation/networking/bonding.txt
new file mode 100644
index 000000000000..0bc2ed136a38
--- /dev/null
+++ b/Documentation/networking/bonding.txt
@@ -0,0 +1,1618 @@
+
+                   Linux Ethernet Bonding Driver HOWTO
+
+Initial release : Thomas Davis <tadavis at lbl.gov>
+Corrections, HA extensions : 2000/10/03-15 :
+  - Willy Tarreau <willy at meta-x.org>
+  - Constantine Gavrilov <const-g at xpert.com>
+  - Chad N. Tindel <ctindel at ieee dot org>
+  - Janice Girouard <girouard at us dot ibm dot com>
+  - Jay Vosburgh <fubar at us dot ibm dot com>
+
+Reorganized and updated Feb 2005 by Jay Vosburgh
+
+Note :
+------
+        
+The bonding driver originally came from Donald Becker's beowulf patches for
+kernel 2.0. It has changed quite a bit since, and the original tools from
+extreme-linux and beowulf sites will not work with this version of the driver.
+
+For new versions of the driver, patches for older kernels and the updated
+userspace tools, please follow the links at the end of this file.
+
+Table of Contents
+=================
+
+1. Bonding Driver Installation
+
+2. Bonding Driver Options
+
+3. Configuring Bonding Devices
+3.1     Configuration with sysconfig support
+3.2     Configuration with initscripts support
+3.3     Configuring Bonding Manually
+3.4     Configuring Multiple Bonds
+
+5. Querying Bonding Configuration
+5.1     Bonding Configuration
+5.2     Network Configuration
+
+6. Switch Configuration
+
+7. 802.1q VLAN Support
+
+8. Link Monitoring
+8.1     ARP Monitor Operation
+8.2     Configuring Multiple ARP Targets
+8.3     MII Monitor Operation
+
+9. Potential Trouble Sources
+9.1     Adventures in Routing
+9.2     Ethernet Device Renaming
+9.3     Painfully Slow Or No Failed Link Detection By Miimon
+
+10. SNMP agents
+
+11. Promiscuous mode
+
+12. High Availability Information
+12.1    High Availability in a Single Switch Topology
+12.1.1          Bonding Mode Selection for Single Switch Topology
+12.1.2          Link Monitoring for Single Switch Topology
+12.2    High Availability in a Multiple Switch Topology
+12.2.1          Bonding Mode Selection for Multiple Switch Topology
+12.2.2          Link Monitoring for Multiple Switch Topology
+12.3    Switch Behavior Issues for High Availability
+
+13. Hardware Specific Considerations
+13.1    IBM BladeCenter
+
+14. Frequently Asked Questions
+
+15. Resources and Links
+
+
+1. Bonding Driver Installation
+==============================
+
+        Most popular distro kernels ship with the bonding driver
+already available as a module and the ifenslave user level control
+program installed and ready for use. If your distro does not, or you
+have need to compile bonding from source (e.g., configuring and
+installing a mainline kernel from kernel.org), you'll need to perform
+the following steps:
+
+1.1 Configure and build the kernel with bonding
+-----------------------------------------------
+
+        The latest version of the bonding driver is available in the
+drivers/net/bonding subdirectory of the most recent kernel source
+(which is available on http://kernel.org).
+
+        Prior to the 2.4.11 kernel, the bonding driver was maintained
+largely outside the kernel tree; patches for some earlier kernels are
+available on the bonding sourceforge site, although those patches are
+still several years out of date.  Most users will want to use either
+the most recent kernel from kernel.org or whatever kernel came with
+their distro.
+
+        Configure kernel with "make menuconfig" (or "make xconfig" or
+"make config"), then select "Bonding driver support" in the "Network
+device support" section.  It is recommended that you configure the
+driver as module since it is currently the only way to pass parameters
+to the driver or configure more than one bonding device.
+
+        Build and install the new kernel and modules, then proceed to
+step 2.
+
+1.2 Install ifenslave Control Utility
+-------------------------------------
+
+        The ifenslave user level control program is included in the
+kernel source tree, in the file Documentation/networking/ifenslave.c.
+It is generally recommended that you use the ifenslave that
+corresponds to the kernel that you are using (either from the same
+source tree or supplied with the distro), however, ifenslave
+executables from older kernels should function (but features newer
+than the ifenslave release are not supported).  Running an ifenslave
+that is newer than the kernel is not supported, and may or may not
+work.
+
+        To install ifenslave, do the following:
+
+# gcc -Wall -O -I/usr/src/linux/include ifenslave.c -o ifenslave
+# cp ifenslave /sbin/ifenslave
+
+        If your kernel source is not in "/usr/src/linux," then replace
+"/usr/src/linux/include" in the above with the location of your kernel
+source include directory.
+
+        You may wish to back up any existing /sbin/ifenslave, or, for
+testing or informal use, tag the ifenslave to the kernel version
+(e.g., name the ifenslave executable /sbin/ifenslave-2.6.10).
+
+IMPORTANT NOTE:
+
+        If you omit the "-I" or specify an incorrect directory, you
+may end up with an ifenslave that is incompatible with the kernel
+you're trying to build it for.  Some distros (e.g., Red Hat from 7.1
+onwards) do not have /usr/include/linux symbolically linked to the
+default kernel source include directory.
+
+
+2. Bonding Driver Options
+=========================
+
+        Options for the bonding driver are supplied as parameters to
+the bonding module at load time.  They may be given as command line
+arguments to the insmod or modprobe command, but are usually specified
+in either the /etc/modprobe.conf configuration file, or in a
+distro-specific configuration file (some of which are detailed in the
+next section).
+
+        The available bonding driver parameters are listed below. If a
+parameter is not specified the default value is used.  When initially
+configuring a bond, it is recommended "tail -f /var/log/messages" be
+run in a separate window to watch for bonding driver error messages.
+
+        It is critical that either the miimon or arp_interval and
+arp_ip_target parameters be specified, otherwise serious network
+degradation will occur during link failures.  Very few devices do not
+support at least miimon, so there is really no reason not to use it.
+
+        Options with textual values will accept either the text name
+        or, for backwards compatibility, the option value.  E.g.,
+        "mode=802.3ad" and "mode=4" set the same mode.
+
+        The parameters are as follows:
+
+arp_interval
+
+        Specifies the ARP monitoring frequency in milli-seconds. If
+        ARP monitoring is used in a load-balancing mode (mode 0 or 2),
+        the switch should be configured in a mode that evenly
+        distributes packets across all links - such as round-robin. If
+        the switch is configured to distribute the packets in an XOR
+        fashion, all replies from the ARP targets will be received on
+        the same link which could cause the other team members to
+        fail. ARP monitoring should not be used in conjunction with
+        miimon. A value of 0 disables ARP monitoring. The default
+        value is 0.
+
+arp_ip_target
+
+        Specifies the ip addresses to use when arp_interval is > 0.
+        These are the targets of the ARP request sent to determine the
+        health of the link to the targets.  Specify these values in
+        ddd.ddd.ddd.ddd format.  Multiple ip adresses must be
+        seperated by a comma.  At least one IP address must be given
+        for ARP monitoring to function.  The maximum number of targets
+        that can be specified is 16.  The default value is no IP
+        addresses.
+
+downdelay
+
+        Specifies the time, in milliseconds, to wait before disabling
+        a slave after a link failure has been detected.  This option
+        is only valid for the miimon link monitor.  The downdelay
+        value should be a multiple of the miimon value; if not, it
+        will be rounded down to the nearest multiple.  The default
+        value is 0.
+
+lacp_rate
+
+        Option specifying the rate in which we'll ask our link partner
+        to transmit LACPDU packets in 802.3ad mode.  Possible values
+        are:
+
+        slow or 0
+                Request partner to transmit LACPDUs every 30 seconds (default)
+
+        fast or 1
+                Request partner to transmit LACPDUs every 1 second
+
+max_bonds
+
+        Specifies the number of bonding devices to create for this
+        instance of the bonding driver.  E.g., if max_bonds is 3, and
+        the bonding driver is not already loaded, then bond0, bond1
+        and bond2 will be created.  The default value is 1.
+
+miimon
+
+        Specifies the frequency in milli-seconds that MII link
+        monitoring will occur.  A value of zero disables MII link
+        monitoring.  A value of 100 is a good starting point.  The
+        use_carrier option, below, affects how the link state is
+        determined.  See the High Availability section for additional
+        information.  The default value is 0.
+
+mode
+
+        Specifies one of the bonding policies. The default is
+        balance-rr (round robin).  Possible values are:
+
+        balance-rr or 0
+
+                Round-robin policy: Transmit packets in sequential
+                order from the first available slave through the
+                last.  This mode provides load balancing and fault
+                tolerance.
+
+        active-backup or 1
+
+                Active-backup policy: Only one slave in the bond is
+                active.  A different slave becomes active if, and only
+                if, the active slave fails.  The bond's MAC address is
+                externally visible on only one port (network adapter)
+                to avoid confusing the switch.  This mode provides
+                fault tolerance.  The primary option affects the
+                behavior of this mode.
+
+        balance-xor or 2
+
+                XOR policy: Transmit based on [(source MAC address
+                XOR'd with destination MAC address) modulo slave
+                count].  This selects the same slave for each
+                destination MAC address.  This mode provides load
+                balancing and fault tolerance.
+
+        broadcast or 3
+
+                Broadcast policy: transmits everything on all slave
+                interfaces.  This mode provides fault tolerance.
+
+        802.3ad or 4
+
+                IEEE 802.3ad Dynamic link aggregation.  Creates
+                aggregation groups that share the same speed and
+                duplex settings.  Utilizes all slaves in the active
+                aggregator according to the 802.3ad specification.
+
+                Pre-requisites:
+
+                1. Ethtool support in the base drivers for retrieving
+                the speed and duplex of each slave.
+
+                2. A switch that supports IEEE 802.3ad Dynamic link
+                aggregation.
+
+                Most switches will require some type of configuration
+                to enable 802.3ad mode.
+
+        balance-tlb or 5
+
+                Adaptive transmit load balancing: channel bonding that
+                does not require any special switch support.  The
+                outgoing traffic is distributed according to the
+                current load (computed relative to the speed) on each
+                slave.  Incoming traffic is received by the current
+                slave.  If the receiving slave fails, another slave
+                takes over the MAC address of the failed receiving
+                slave.
+
+                Prerequisite:
+
+                Ethtool support in the base drivers for retrieving the
+                speed of each slave.
+
+        balance-alb or 6
+
+                Adaptive load balancing: includes balance-tlb plus
+                receive load balancing (rlb) for IPV4 traffic, and
+                does not require any special switch support.  The
+                receive load balancing is achieved by ARP negotiation.
+                The bonding driver intercepts the ARP Replies sent by
+                the local system on their way out and overwrites the
+                source hardware address with the unique hardware
+                address of one of the slaves in the bond such that
+                different peers use different hardware addresses for
+                the server.
+
+                Receive traffic from connections created by the server
+                is also balanced.  When the local system sends an ARP
+                Request the bonding driver copies and saves the peer's
+                IP information from the ARP packet.  When the ARP
+                Reply arrives from the peer, its hardware address is
+                retrieved and the bonding driver initiates an ARP
+                reply to this peer assigning it to one of the slaves
+                in the bond.  A problematic outcome of using ARP
+                negotiation for balancing is that each time that an
+                ARP request is broadcast it uses the hardware address
+                of the bond.  Hence, peers learn the hardware address
+                of the bond and the balancing of receive traffic
+                collapses to the current slave.  This is handled by
+                sending updates (ARP Replies) to all the peers with
+                their individually assigned hardware address such that
+                the traffic is redistributed.  Receive traffic is also
+                redistributed when a new slave is added to the bond
+                and when an inactive slave is re-activated.  The
+                receive load is distributed sequentially (round robin)
+                among the group of highest speed slaves in the bond.
+
+                When a link is reconnected or a new slave joins the
+                bond the receive traffic is redistributed among all
+                active slaves in the bond by intiating ARP Replies
+                with the selected mac address to each of the
+                clients. The updelay parameter (detailed below) must
+                be set to a value equal or greater than the switch's
+                forwarding delay so that the ARP Replies sent to the
+                peers will not be blocked by the switch.
+
+                Prerequisites:
+
+                1. Ethtool support in the base drivers for retrieving
+                the speed of each slave.
+
+                2. Base driver support for setting the hardware
+                address of a device while it is open.  This is
+                required so that there will always be one slave in the
+                team using the bond hardware address (the
+                curr_active_slave) while having a unique hardware
+                address for each slave in the bond.  If the
+                curr_active_slave fails its hardware address is
+                swapped with the new curr_active_slave that was
+                chosen.
+
+primary
+
+        A string (eth0, eth2, etc) specifying which slave is the
+        primary device.  The specified device will always be the
+        active slave while it is available.  Only when the primary is
+        off-line will alternate devices be used.  This is useful when
+        one slave is preferred over another, e.g., when one slave has
+        higher throughput than another.
+
+        The primary option is only valid for active-backup mode.
+
+updelay
+
+        Specifies the time, in milliseconds, to wait before enabling a
+        slave after a link recovery has been detected.  This option is
+        only valid for the miimon link monitor.  The updelay value
+        should be a multiple of the miimon value; if not, it will be
+        rounded down to the nearest multiple.  The default value is 0.
+
+use_carrier
+
+        Specifies whether or not miimon should use MII or ETHTOOL
+        ioctls vs. netif_carrier_ok() to determine the link
+        status. The MII or ETHTOOL ioctls are less efficient and
+        utilize a deprecated calling sequence within the kernel.  The
+        netif_carrier_ok() relies on the device driver to maintain its
+        state with netif_carrier_on/off; at this writing, most, but
+        not all, device drivers support this facility.
+
+        If bonding insists that the link is up when it should not be,
+        it may be that your network device driver does not support
+        netif_carrier_on/off.  The default state for netif_carrier is
+        "carrier on," so if a driver does not support netif_carrier,
+        it will appear as if the link is always up.  In this case,
+        setting use_carrier to 0 will cause bonding to revert to the
+        MII / ETHTOOL ioctl method to determine the link state.
+
+        A value of 1 enables the use of netif_carrier_ok(), a value of
+        0 will use the deprecated MII / ETHTOOL ioctls.  The default
+        value is 1.
+
+
+
+3. Configuring Bonding Devices
+==============================
+
+        There are, essentially, two methods for configuring bonding:
+with support from the distro's network initialization scripts, and
+without.  Distros generally use one of two packages for the network
+initialization scripts: initscripts or sysconfig.  Recent versions of
+these packages have support for bonding, while older versions do not.
+
+        We will first describe the options for configuring bonding for
+distros using versions of initscripts and sysconfig with full or
+partial support for bonding, then provide information on enabling
+bonding without support from the network initialization scripts (i.e.,
+older versions of initscripts or sysconfig).
+
+        If you're unsure whether your distro uses sysconfig or
+initscripts, or don't know if it's new enough, have no fear.
+Determining this is fairly straightforward.
+
+        First, issue the command:
+
+$ rpm -qf /sbin/ifup
+
+        It will respond with a line of text starting with either
+"initscripts" or "sysconfig," followed by some numbers.  This is the
+package that provides your network initialization scripts.
+
+        Next, to determine if your installation supports bonding,
+issue the command:
+
+$ grep ifenslave /sbin/ifup
+
+        If this returns any matches, then your initscripts or
+sysconfig has support for bonding.
+
+3.1 Configuration with sysconfig support
+----------------------------------------
+
+        This section applies to distros using a version of sysconfig
+with bonding support, for example, SuSE Linux Enterprise Server 9.
+
+        SuSE SLES 9's networking configuration system does support
+bonding, however, at this writing, the YaST system configuration
+frontend does not provide any means to work with bonding devices.
+Bonding devices can be managed by hand, however, as follows.
+
+        First, if they have not already been configured, configure the
+slave devices.  On SLES 9, this is most easily done by running the
+yast2 sysconfig configuration utility.  The goal is for to create an
+ifcfg-id file for each slave device.  The simplest way to accomplish
+this is to configure the devices for DHCP.  The name of the
+configuration file for each device will be of the form:
+
+ifcfg-id-xx:xx:xx:xx:xx:xx
+
+        Where the "xx" portion will be replaced with the digits from
+the device's permanent MAC address.
+
+        Once the set of ifcfg-id-xx:xx:xx:xx:xx:xx files has been
+created, it is necessary to edit the configuration files for the slave
+devices (the MAC addresses correspond to those of the slave devices).
+Before editing, the file will contain muliple lines, and will look
+something like this:
+
+BOOTPROTO='dhcp'
+STARTMODE='on'
+USERCTL='no'
+UNIQUE='XNzu.WeZGOGF+4wE'
+_nm_name='bus-pci-0001:61:01.0'
+
+        Change the BOOTPROTO and STARTMODE lines to the following:
+
+BOOTPROTO='none'
+STARTMODE='off'
+
+        Do not alter the UNIQUE or _nm_name lines.  Remove any other
+lines (USERCTL, etc).
+
+        Once the ifcfg-id-xx:xx:xx:xx:xx:xx files have been modified,
+it's time to create the configuration file for the bonding device
+itself.  This file is named ifcfg-bondX, where X is the number of the
+bonding device to create, starting at 0.  The first such file is
+ifcfg-bond0, the second is ifcfg-bond1, and so on.  The sysconfig
+network configuration system will correctly start multiple instances
+of bonding.
+
+        The contents of the ifcfg-bondX file is as follows:
+
+BOOTPROTO="static"
+BROADCAST="10.0.2.255"
+IPADDR="10.0.2.10"
+NETMASK="255.255.0.0"
+NETWORK="10.0.2.0"
+REMOTE_IPADDR=""
+STARTMODE="onboot"
+BONDING_MASTER="yes"
+BONDING_MODULE_OPTS="mode=active-backup miimon=100"
+BONDING_SLAVE0="eth0"
+BONDING_SLAVE1="eth1"
+
+        Replace the sample BROADCAST, IPADDR, NETMASK and NETWORK
+values with the appropriate values for your network.
+
+        Note that configuring the bonding device with BOOTPROTO='dhcp'
+does not work; the scripts attempt to obtain the device address from
+DHCP prior to adding any of the slave devices.  Without active slaves,
+the DHCP requests are not sent to the network.
+
+        The STARTMODE specifies when the device is brought online.
+The possible values are:
+
+        onboot:  The device is started at boot time.  If you're not
+                 sure, this is probably what you want.
+
+        manual:  The device is started only when ifup is called
+                 manually.  Bonding devices may be configured this
+                 way if you do not wish them to start automatically
+                 at boot for some reason.
+
+        hotplug: The device is started by a hotplug event.  This is not
+                 a valid choice for a bonding device.
+
+        off or ignore: The device configuration is ignored.
+
+        The line BONDING_MASTER='yes' indicates that the device is a
+bonding master device.  The only useful value is "yes."
+
+        The contents of BONDING_MODULE_OPTS are supplied to the
+instance of the bonding module for this device.  Specify the options
+for the bonding mode, link monitoring, and so on here.  Do not include
+the max_bonds bonding parameter; this will confuse the configuration
+system if you have multiple bonding devices.
+
+        Finally, supply one BONDING_SLAVEn="ethX" for each slave,
+where "n" is an increasing value, one for each slave, and "ethX" is
+the name of the slave device (eth0, eth1, etc).
+
+        When all configuration files have been modified or created,
+networking must be restarted for the configuration changes to take
+effect.  This can be accomplished via the following:
+
+# /etc/init.d/network restart
+
+        Note that the network control script (/sbin/ifdown) will
+remove the bonding module as part of the network shutdown processing,
+so it is not necessary to remove the module by hand if, e.g., the
+module paramters have changed.
+
+        Also, at this writing, YaST/YaST2 will not manage bonding
+devices (they do not show bonding interfaces on its list of network
+devices).  It is necessary to edit the configuration file by hand to
+change the bonding configuration.
+
+        Additional general options and details of the ifcfg file
+format can be found in an example ifcfg template file:
+
+/etc/sysconfig/network/ifcfg.template
+
+        Note that the template does not document the various BONDING_
+settings described above, but does describe many of the other options.
+
+3.2 Configuration with initscripts support
+------------------------------------------
+
+        This section applies to distros using a version of initscripts
+with bonding support, for example, Red Hat Linux 9 or Red Hat
+Enterprise Linux version 3.  On these systems, the network
+initialization scripts have some knowledge of bonding, and can be
+configured to control bonding devices.
+
+        These distros will not automatically load the network adapter
+driver unless the ethX device is configured with an IP address.
+Because of this constraint, users must manually configure a
+network-script file for all physical adapters that will be members of
+a bondX link.  Network script files are located in the directory:
+
+/etc/sysconfig/network-scripts
+
+        The file name must be prefixed with "ifcfg-eth" and suffixed
+with the adapter's physical adapter number.  For example, the script
+for eth0 would be named /etc/sysconfig/network-scripts/ifcfg-eth0.
+Place the following text in the file:
+
+DEVICE=eth0
+USERCTL=no
+ONBOOT=yes
+MASTER=bond0
+SLAVE=yes
+BOOTPROTO=none
+
+        The DEVICE= line will be different for every ethX device and
+must correspond with the name of the file, i.e., ifcfg-eth1 must have
+a device line of DEVICE=eth1.  The setting of the MASTER= line will
+also depend on the final bonding interface name chosen for your bond.
+As with other network devices, these typically start at 0, and go up
+one for each device, i.e., the first bonding instance is bond0, the
+second is bond1, and so on.
+
+        Next, create a bond network script.  The file name for this
+script will be /etc/sysconfig/network-scripts/ifcfg-bondX where X is
+the number of the bond.  For bond0 the file is named "ifcfg-bond0",
+for bond1 it is named "ifcfg-bond1", and so on.  Within that file,
+place the following text:
+
+DEVICE=bond0
+IPADDR=192.168.1.1
+NETMASK=255.255.255.0
+NETWORK=192.168.1.0
+BROADCAST=192.168.1.255
+ONBOOT=yes
+BOOTPROTO=none
+USERCTL=no
+
+        Be sure to change the networking specific lines (IPADDR,
+NETMASK, NETWORK and BROADCAST) to match your network configuration.
+
+        Finally, it is necessary to edit /etc/modules.conf to load the
+bonding module when the bond0 interface is brought up.  The following
+sample lines in /etc/modules.conf will load the bonding module, and
+select its options:
+
+alias bond0 bonding
+options bond0 mode=balance-alb miimon=100
+
+        Replace the sample parameters with the appropriate set of
+options for your configuration.
+
+        Finally run "/etc/rc.d/init.d/network restart" as root.  This
+will restart the networking subsystem and your bond link should be now
+up and running.
+
+
+3.3 Configuring Bonding Manually
+--------------------------------
+
+        This section applies to distros whose network initialization
+scripts (the sysconfig or initscripts package) do not have specific
+knowledge of bonding.  One such distro is SuSE Linux Enterprise Server
+version 8.
+
+        The general methodology for these systems is to place the
+bonding module parameters into /etc/modprobe.conf, then add modprobe
+and/or ifenslave commands to the system's global init script.  The
+name of the global init script differs; for sysconfig, it is
+/etc/init.d/boot.local and for initscripts it is /etc/rc.d/rc.local.
+
+        For example, if you wanted to make a simple bond of two e100
+devices (presumed to be eth0 and eth1), and have it persist across
+reboots, edit the appropriate file (/etc/init.d/boot.local or
+/etc/rc.d/rc.local), and add the following:
+
+modprobe bonding -obond0 mode=balance-alb miimon=100
+modprobe e100
+ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up
+ifenslave bond0 eth0
+ifenslave bond0 eth1
+
+        Replace the example bonding module parameters and bond0
+network configuration (IP address, netmask, etc) with the appropriate
+values for your configuration.  The above example loads the bonding
+module with the name "bond0," this simplifies the naming if multiple
+bonding modules are loaded (each successive instance of the module is
+given a different name, and the module instance names match the
+bonding interface names).
+
+        Unfortunately, this method will not provide support for the
+ifup and ifdown scripts on the bond devices.  To reload the bonding
+configuration, it is necessary to run the initialization script, e.g.,
+
+# /etc/init.d/boot.local
+
+        or
+
+# /etc/rc.d/rc.local
+
+        It may be desirable in such a case to create a separate script
+which only initializes the bonding configuration, then call that
+separate script from within boot.local.  This allows for bonding to be
+enabled without re-running the entire global init script.
+
+        To shut down the bonding devices, it is necessary to first
+mark the bonding device itself as being down, then remove the
+appropriate device driver modules.  For our example above, you can do
+the following:
+
+# ifconfig bond0 down
+# rmmod bond0
+# rmmod e100
+
+        Again, for convenience, it may be desirable to create a script
+with these commands.
+
+
+3.4 Configuring Multiple Bonds
+------------------------------
+
+        This section contains information on configuring multiple
+bonding devices with differing options.  If you require multiple
+bonding devices, but all with the same options, see the "max_bonds"
+module paramter, documented above.
+
+        To create multiple bonding devices with differing options, it
+is necessary to load the bonding driver multiple times.  Note that
+current versions of the sysconfig network initialization scripts
+handle this automatically; if your distro uses these scripts, no
+special action is needed.  See the section Configuring Bonding
+Devices, above, if you're not sure about your network initialization
+scripts.
+
+        To load multiple instances of the module, it is necessary to
+specify a different name for each instance (the module loading system
+requires that every loaded module, even multiple instances of the same
+module, have a unique name).  This is accomplished by supplying
+multiple sets of bonding options in /etc/modprobe.conf, for example:
+        
+alias bond0 bonding
+options bond0 -o bond0 mode=balance-rr miimon=100
+
+alias bond1 bonding
+options bond1 -o bond1 mode=balance-alb miimon=50
+
+        will load the bonding module two times.  The first instance is
+named "bond0" and creates the bond0 device in balance-rr mode with an
+miimon of 100.  The second instance is named "bond1" and creates the
+bond1 device in balance-alb mode with an miimon of 50.
+
+        This may be repeated any number of times, specifying a new and
+unique name in place of bond0 or bond1 for each instance.
+
+        When the appropriate module paramters are in place, then
+configure bonding according to the instructions for your distro.
+
+5. Querying Bonding Configuration 
+=================================
+
+5.1 Bonding Configuration
+-------------------------
+
+        Each bonding device has a read-only file residing in the
+/proc/net/bonding directory.  The file contents include information
+about the bonding configuration, options and state of each slave.
+
+        For example, the contents of /proc/net/bonding/bond0 after the
+driver is loaded with parameters of mode=0 and miimon=1000 is
+generally as follows:
+
+        Ethernet Channel Bonding Driver: 2.6.1 (October 29, 2004)
+        Bonding Mode: load balancing (round-robin)
+        Currently Active Slave: eth0
+        MII Status: up
+        MII Polling Interval (ms): 1000
+        Up Delay (ms): 0
+        Down Delay (ms): 0
+
+        Slave Interface: eth1
+        MII Status: up
+        Link Failure Count: 1
+
+        Slave Interface: eth0
+        MII Status: up
+        Link Failure Count: 1
+
+        The precise format and contents will change depending upon the
+bonding configuration, state, and version of the bonding driver.
+
+5.2 Network configuration
+-------------------------
+
+        The network configuration can be inspected using the ifconfig
+command.  Bonding devices will have the MASTER flag set; Bonding slave
+devices will have the SLAVE flag set.  The ifconfig output does not
+contain information on which slaves are associated with which masters.
+
+        In the example below, the bond0 interface is the master
+(MASTER) while eth0 and eth1 are slaves (SLAVE). Notice all slaves of
+bond0 have the same MAC address (HWaddr) as bond0 for all modes except
+TLB and ALB that require a unique MAC address for each slave.
+
+# /sbin/ifconfig
+bond0     Link encap:Ethernet  HWaddr 00:C0:F0:1F:37:B4
+          inet addr:XXX.XXX.XXX.YYY  Bcast:XXX.XXX.XXX.255  Mask:255.255.252.0
+          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
+          RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0
+          TX packets:3286647 errors:1 dropped:0 overruns:1 carrier:0
+          collisions:0 txqueuelen:0
+
+eth0      Link encap:Ethernet  HWaddr 00:C0:F0:1F:37:B4
+          inet addr:XXX.XXX.XXX.YYY  Bcast:XXX.XXX.XXX.255  Mask:255.255.252.0
+          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
+          RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0
+          TX packets:1643167 errors:1 dropped:0 overruns:1 carrier:0
+          collisions:0 txqueuelen:100
+          Interrupt:10 Base address:0x1080
+
+eth1      Link encap:Ethernet  HWaddr 00:C0:F0:1F:37:B4
+          inet addr:XXX.XXX.XXX.YYY  Bcast:XXX.XXX.XXX.255  Mask:255.255.252.0
+          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
+          RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0
+          TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0
+          collisions:0 txqueuelen:100
+          Interrupt:9 Base address:0x1400
+
+6. Switch Configuration
+=======================
+
+        For this section, "switch" refers to whatever system the
+bonded devices are directly connected to (i.e., where the other end of
+the cable plugs into).  This may be an actual dedicated switch device,
+or it may be another regular system (e.g., another computer running
+Linux),
+
+        The active-backup, balance-tlb and balance-alb modes do not
+require any specific configuration of the switch.
+
+        The 802.3ad mode requires that the switch have the appropriate
+ports configured as an 802.3ad aggregation.  The precise method used
+to configure this varies from switch to switch, but, for example, a
+Cisco 3550 series switch requires that the appropriate ports first be
+grouped together in a single etherchannel instance, then that
+etherchannel is set to mode "lacp" to enable 802.3ad (instead of
+standard EtherChannel).
+
+        The balance-rr, balance-xor and broadcast modes generally
+require that the switch have the appropriate ports grouped together.
+The nomenclature for such a group differs between switches, it may be
+called an "etherchannel" (as in the Cisco example, above), a "trunk
+group" or some other similar variation.  For these modes, each switch
+will also have its own configuration options for the switch's transmit
+policy to the bond.  Typical choices include XOR of either the MAC or
+IP addresses.  The transmit policy of the two peers does not need to
+match.  For these three modes, the bonding mode really selects a
+transmit policy for an EtherChannel group; all three will interoperate
+with another EtherChannel group.
+
+
+7. 802.1q VLAN Support
+======================
+
+        It is possible to configure VLAN devices over a bond interface
+using the 8021q driver.  However, only packets coming from the 8021q
+driver and passing through bonding will be tagged by default.  Self
+generated packets, for example, bonding's learning packets or ARP
+packets generated by either ALB mode or the ARP monitor mechanism, are
+tagged internally by bonding itself.  As a result, bonding must
+"learn" the VLAN IDs configured above it, and use those IDs to tag
+self generated packets.
+
+        For reasons of simplicity, and to support the use of adapters
+that can do VLAN hardware acceleration offloding, the bonding
+interface declares itself as fully hardware offloaing capable, it gets
+the add_vid/kill_vid notifications to gather the necessary
+information, and it propagates those actions to the slaves.  In case
+of mixed adapter types, hardware accelerated tagged packets that
+should go through an adapter that is not offloading capable are
+"un-accelerated" by the bonding driver so the VLAN tag sits in the
+regular location.
+
+        VLAN interfaces *must* be added on top of a bonding interface
+only after enslaving at least one slave.  The bonding interface has a
+hardware address of 00:00:00:00:00:00 until the first slave is added.
+If the VLAN interface is created prior to the first enslavement, it
+would pick up the all-zeroes hardware address.  Once the first slave
+is attached to the bond, the bond device itself will pick up the
+slave's hardware address, which is then available for the VLAN device.
+
+        Also, be aware that a similar problem can occur if all slaves
+are released from a bond that still has one or more VLAN interfaces on
+top of it.  When a new slave is added, the bonding interface will
+obtain its hardware address from the first slave, which might not
+match the hardware address of the VLAN interfaces (which was
+ultimately copied from an earlier slave).
+
+        There are two methods to insure that the VLAN device operates
+with the correct hardware address if all slaves are removed from a
+bond interface:
+
+        1. Remove all VLAN interfaces then recreate them
+
+        2. Set the bonding interface's hardware address so that it
+matches the hardware address of the VLAN interfaces.
+
+        Note that changing a VLAN interface's HW address would set the
+underlying device -- i.e. the bonding interface -- to promiscouos
+mode, which might not be what you want.
+
+
+8. Link Monitoring
+==================
+
+        The bonding driver at present supports two schemes for
+monitoring a slave device's link state: the ARP monitor and the MII
+monitor.
+
+        At the present time, due to implementation restrictions in the
+bonding driver itself, it is not possible to enable both ARP and MII
+monitoring simultaneously.
+
+8.1 ARP Monitor Operation
+-------------------------
+
+        The ARP monitor operates as its name suggests: it sends ARP
+queries to one or more designated peer systems on the network, and
+uses the response as an indication that the link is operating.  This
+gives some assurance that traffic is actually flowing to and from one
+or more peers on the local network.
+
+        The ARP monitor relies on the device driver itself to verify
+that traffic is flowing.  In particular, the driver must keep up to
+date the last receive time, dev->last_rx, and transmit start time,
+dev->trans_start.  If these are not updated by the driver, then the
+ARP monitor will immediately fail any slaves using that driver, and
+those slaves will stay down.  If networking monitoring (tcpdump, etc)
+shows the ARP requests and replies on the network, then it may be that
+your device driver is not updating last_rx and trans_start.
+
+8.2 Configuring Multiple ARP Targets
+------------------------------------
+
+        While ARP monitoring can be done with just one target, it can
+be useful in a High Availability setup to have several targets to
+monitor.  In the case of just one target, the target itself may go
+down or have a problem making it unresponsive to ARP requests.  Having
+an additional target (or several) increases the reliability of the ARP
+monitoring.
+
+        Multiple ARP targets must be seperated by commas as follows:
+
+# example options for ARP monitoring with three targets
+alias bond0 bonding
+options bond0 arp_interval=60 arp_ip_target=192.168.0.1,192.168.0.3,192.168.0.9
+
+        For just a single target the options would resemble:
+
+# example options for ARP monitoring with one target
+alias bond0 bonding
+options bond0 arp_interval=60 arp_ip_target=192.168.0.100
+
+
+8.3 MII Monitor Operation
+-------------------------
+
+        The MII monitor monitors only the carrier state of the local
+network interface.  It accomplishes this in one of three ways: by
+depending upon the device driver to maintain its carrier state, by
+querying the device's MII registers, or by making an ethtool query to
+the device.
+
+        If the use_carrier module parameter is 1 (the default value),
+then the MII monitor will rely on the driver for carrier state
+information (via the netif_carrier subsystem).  As explained in the
+use_carrier parameter information, above, if the MII monitor fails to
+detect carrier loss on the device (e.g., when the cable is physically
+disconnected), it may be that the driver does not support
+netif_carrier.
+
+        If use_carrier is 0, then the MII monitor will first query the
+device's (via ioctl) MII registers and check the link state.  If that
+request fails (not just that it returns carrier down), then the MII
+monitor will make an ethtool ETHOOL_GLINK request to attempt to obtain
+the same information.  If both methods fail (i.e., the driver either
+does not support or had some error in processing both the MII register
+and ethtool requests), then the MII monitor will assume the link is
+up.
+
+9. Potential Sources of Trouble
+===============================
+
+9.1 Adventures in Routing
+-------------------------
+
+        When bonding is configured, it is important that the slave
+devices not have routes that supercede routes of the master (or,
+generally, not have routes at all).  For example, suppose the bonding
+device bond0 has two slaves, eth0 and eth1, and the routing table is
+as follows:
+
+Kernel IP routing table
+Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
+10.0.0.0        0.0.0.0         255.255.0.0     U        40 0          0 eth0
+10.0.0.0        0.0.0.0         255.255.0.0     U        40 0          0 eth1
+10.0.0.0        0.0.0.0         255.255.0.0     U        40 0          0 bond0
+127.0.0.0       0.0.0.0         255.0.0.0       U        40 0          0 lo
+
+        This routing configuration will likely still update the
+receive/transmit times in the driver (needed by the ARP monitor), but
+may bypass the bonding driver (because outgoing traffic to, in this
+case, another host on network 10 would use eth0 or eth1 before bond0).
+
+        The ARP monitor (and ARP itself) may become confused by this
+configuration, because ARP requests (generated by the ARP monitor)
+will be sent on one interface (bond0), but the corresponding reply
+will arrive on a different interface (eth0).  This reply looks to ARP
+as an unsolicited ARP reply (because ARP matches replies on an
+interface basis), and is discarded.  The MII monitor is not affected
+by the state of the routing table.
+
+        The solution here is simply to insure that slaves do not have
+routes of their own, and if for some reason they must, those routes do
+not supercede routes of their master.  This should generally be the
+case, but unusual configurations or errant manual or automatic static
+route additions may cause trouble.
+
+9.2 Ethernet Device Renaming
+----------------------------
+
+        On systems with network configuration scripts that do not
+associate physical devices directly with network interface names (so
+that the same physical device always has the same "ethX" name), it may
+be necessary to add some special logic to either /etc/modules.conf or
+/etc/modprobe.conf (depending upon which is installed on the system).
+
+        For example, given a modules.conf containing the following:
+
+alias bond0 bonding
+options bond0 mode=some-mode miimon=50
+alias eth0 tg3
+alias eth1 tg3
+alias eth2 e1000
+alias eth3 e1000
+
+        If neither eth0 and eth1 are slaves to bond0, then when the
+bond0 interface comes up, the devices may end up reordered.  This
+happens because bonding is loaded first, then its slave device's
+drivers are loaded next.  Since no other drivers have been loaded,
+when the e1000 driver loads, it will receive eth0 and eth1 for its
+devices, but the bonding configuration tries to enslave eth2 and eth3
+(which may later be assigned to the tg3 devices).
+
+        Adding the following:
+
+add above bonding e1000 tg3
+
+        causes modprobe to load e1000 then tg3, in that order, when
+bonding is loaded.  This command is fully documented in the
+modules.conf manual page.
+
+        On systems utilizing modprobe.conf (or modprobe.conf.local),
+an equivalent problem can occur.  In this case, the following can be
+added to modprobe.conf (or modprobe.conf.local, as appropriate), as
+follows (all on one line; it has been split here for clarity):
+
+install bonding /sbin/modprobe tg3; /sbin/modprobe e1000;
+        /sbin/modprobe --ignore-install bonding
+
+        This will, when loading the bonding module, rather than
+performing the normal action, instead execute the provided command.
+This command loads the device drivers in the order needed, then calls
+modprobe with --ingore-install to cause the normal action to then take
+place.  Full documentation on this can be found in the modprobe.conf
+and modprobe manual pages.
+
+9.3. Painfully Slow Or No Failed Link Detection By Miimon
+---------------------------------------------------------
+
+        By default, bonding enables the use_carrier option, which
+instructs bonding to trust the driver to maintain carrier state.
+
+        As discussed in the options section, above, some drivers do
+not support the netif_carrier_on/_off link state tracking system.
+With use_carrier enabled, bonding will always see these links as up,
+regardless of their actual state.
+
+        Additionally, other drivers do support netif_carrier, but do
+not maintain it in real time, e.g., only polling the link state at
+some fixed interval.  In this case, miimon will detect failures, but
+only after some long period of time has expired.  If it appears that
+miimon is very slow in detecting link failures, try specifying
+use_carrier=0 to see if that improves the failure detection time.  If
+it does, then it may be that the driver checks the carrier state at a
+fixed interval, but does not cache the MII register values (so the
+use_carrier=0 method of querying the registers directly works).  If
+use_carrier=0 does not improve the failover, then the driver may cache
+the registers, or the problem may be elsewhere.
+
+        Also, remember that miimon only checks for the device's
+carrier state.  It has no way to determine the state of devices on or
+beyond other ports of a switch, or if a switch is refusing to pass
+traffic while still maintaining carrier on.
+
+10. SNMP agents
+===============
+
+        If running SNMP agents, the bonding driver should be loaded
+before any network drivers participating in a bond.  This requirement
+is due to the the interface index (ipAdEntIfIndex) being associated to
+the first interface found with a given IP address.  That is, there is
+only one ipAdEntIfIndex for each IP address.  For example, if eth0 and
+eth1 are slaves of bond0 and the driver for eth0 is loaded before the
+bonding driver, the interface for the IP address will be associated
+with the eth0 interface.  This configuration is shown below, the IP
+address 192.168.1.1 has an interface index of 2 which indexes to eth0
+in the ifDescr table (ifDescr.2).
+
+     interfaces.ifTable.ifEntry.ifDescr.1 = lo
+     interfaces.ifTable.ifEntry.ifDescr.2 = eth0
+     interfaces.ifTable.ifEntry.ifDescr.3 = eth1
+     interfaces.ifTable.ifEntry.ifDescr.4 = eth2
+     interfaces.ifTable.ifEntry.ifDescr.5 = eth3
+     interfaces.ifTable.ifEntry.ifDescr.6 = bond0
+     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 5
+     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
+     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 4
+     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
+
+        This problem is avoided by loading the bonding driver before
+any network drivers participating in a bond.  Below is an example of
+loading the bonding driver first, the IP address 192.168.1.1 is
+correctly associated with ifDescr.2.
+
+     interfaces.ifTable.ifEntry.ifDescr.1 = lo
+     interfaces.ifTable.ifEntry.ifDescr.2 = bond0
+     interfaces.ifTable.ifEntry.ifDescr.3 = eth0
+     interfaces.ifTable.ifEntry.ifDescr.4 = eth1
+     interfaces.ifTable.ifEntry.ifDescr.5 = eth2
+     interfaces.ifTable.ifEntry.ifDescr.6 = eth3
+     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.10.10.10 = 6
+     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.192.168.1.1 = 2
+     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.10.74.20.94 = 5
+     ip.ipAddrTable.ipAddrEntry.ipAdEntIfIndex.127.0.0.1 = 1
+
+        While some distributions may not report the interface name in
+ifDescr, the association between the IP address and IfIndex remains
+and SNMP functions such as Interface_Scan_Next will report that
+association.
+
+11. Promiscuous mode
+====================
+
+        When running network monitoring tools, e.g., tcpdump, it is
+common to enable promiscuous mode on the device, so that all traffic
+is seen (instead of seeing only traffic destined for the local host).
+The bonding driver handles promiscuous mode changes to the bonding
+master device (e.g., bond0), and propogates the setting to the slave
+devices.
+
+        For the balance-rr, balance-xor, broadcast, and 802.3ad modes,
+the promiscuous mode setting is propogated to all slaves.
+
+        For the active-backup, balance-tlb and balance-alb modes, the
+promiscuous mode setting is propogated only to the active slave.
+
+        For balance-tlb mode, the active slave is the slave currently
+receiving inbound traffic.
+
+        For balance-alb mode, the active slave is the slave used as a
+"primary."  This slave is used for mode-specific control traffic, for
+sending to peers that are unassigned or if the load is unbalanced.
+
+        For the active-backup, balance-tlb and balance-alb modes, when
+the active slave changes (e.g., due to a link failure), the
+promiscuous setting will be propogated to the new active slave.
+
+12. High Availability Information
+=================================
+
+        High Availability refers to configurations that provide
+maximum network availability by having redundant or backup devices,
+links and switches between the host and the rest of the world.
+
+        There are currently two basic methods for configuring to
+maximize availability. They are dependent on the network topology and
+the primary goal of the configuration, but in general, a configuration
+can be optimized for maximum available bandwidth, or for maximum
+network availability.
+
+12.1 High Availability in a Single Switch Topology
+--------------------------------------------------
+
+        If two hosts (or a host and a switch) are directly connected
+via multiple physical links, then there is no network availability
+penalty for optimizing for maximum bandwidth: there is only one switch
+(or peer), so if it fails, you have no alternative access to fail over
+to.
+
+Example 1 : host to switch (or other host)
+
+          +----------+                          +----------+
+          |          |eth0                  eth0|  switch  |
+          | Host A   +--------------------------+    or    |
+          |          +--------------------------+  other   |
+          |          |eth1                  eth1|  host    |
+          +----------+                          +----------+
+
+
+12.1.1 Bonding Mode Selection for single switch topology
+--------------------------------------------------------
+
+        This configuration is the easiest to set up and to understand,
+although you will have to decide which bonding mode best suits your
+needs.  The tradeoffs for each mode are detailed below:
+
+balance-rr: This mode is the only mode that will permit a single
+        TCP/IP connection to stripe traffic across multiple
+        interfaces. It is therefore the only mode that will allow a
+        single TCP/IP stream to utilize more than one interface's
+        worth of throughput.  This comes at a cost, however: the
+        striping often results in peer systems receiving packets out
+        of order, causing TCP/IP's congestion control system to kick
+        in, often by retransmitting segments.
+
+        It is possible to adjust TCP/IP's congestion limits by
+        altering the net.ipv4.tcp_reordering sysctl parameter.  The
+        usual default value is 3, and the maximum useful value is 127.
+        For a four interface balance-rr bond, expect that a single
+        TCP/IP stream will utilize no more than approximately 2.3
+        interface's worth of throughput, even after adjusting
+        tcp_reordering.
+
+        If you are utilizing protocols other than TCP/IP, UDP for
+        example, and your application can tolerate out of order
+        delivery, then this mode can allow for single stream datagram
+        performance that scales near linearly as interfaces are added
+        to the bond.
+
+        This mode requires the switch to have the appropriate ports
+        configured for "etherchannel" or "trunking."
+
+active-backup: There is not much advantage in this network topology to
+        the active-backup mode, as the inactive backup devices are all
+        connected to the same peer as the primary.  In this case, a
+        load balancing mode (with link monitoring) will provide the
+        same level of network availability, but with increased
+        available bandwidth.  On the plus side, it does not require
+        any configuration of the switch.
+
+balance-xor: This mode will limit traffic such that packets destined
+        for specific peers will always be sent over the same
+        interface.  Since the destination is determined by the MAC
+        addresses involved, this may be desirable if you have a large
+        network with many hosts.  It is likely to be suboptimal if all
+        your traffic is passed through a single router, however.  As
+        with balance-rr, the switch ports need to be configured for
+        "etherchannel" or "trunking."
+
+broadcast: Like active-backup, there is not much advantage to this
+        mode in this type of network topology.
+
+802.3ad: This mode can be a good choice for this type of network
+        topology.  The 802.3ad mode is an IEEE standard, so all peers
+        that implement 802.3ad should interoperate well.  The 802.3ad
+        protocol includes automatic configuration of the aggregates,
+        so minimal manual configuration of the switch is needed
+        (typically only to designate that some set of devices is
+        usable for 802.3ad).  The 802.3ad standard also mandates that
+        frames be delivered in order (within certain limits), so in
+        general single connections will not see misordering of
+        packets.  The 802.3ad mode does have some drawbacks: the
+        standard mandates that all devices in the aggregate operate at
+        the same speed and duplex.  Also, as with all bonding load
+        balance modes other than balance-rr, no single connection will
+        be able to utilize more than a single interface's worth of
+        bandwidth.  Additionally, the linux bonding 802.3ad
+        implementation distributes traffic by peer (using an XOR of
+        MAC addresses), so in general all traffic to a particular
+        destination will use the same interface.  Finally, the 802.3ad
+        mode mandates the use of the MII monitor, therefore, the ARP
+        monitor is not available in this mode.
+
+balance-tlb: This mode is also a good choice for this type of
+        topology.  It has no special switch configuration
+        requirements, and balances outgoing traffic by peer, in a
+        vaguely intelligent manner (not a simple XOR as in balance-xor
+        or 802.3ad mode), so that unlucky MAC addresses will not all
+        "bunch up" on a single interface.  Interfaces may be of
+        differing speeds.  On the down side, in this mode all incoming
+        traffic arrives over a single interface, this mode requires
+        certain ethtool support in the network device driver of the
+        slave interfaces, and the ARP monitor is not available.
+
+balance-alb: This mode is everything that balance-tlb is, and more. It
+        has all of the features (and restrictions) of balance-tlb, and
+        will also balance incoming traffic from peers (as described in
+        the Bonding Module Options section, above).  The only extra
+        down side to this mode is that the network device driver must
+        support changing the hardware address while the device is
+        open.
+
+12.1.2 Link Monitoring for Single Switch Topology
+-------------------------------------------------
+
+        The choice of link monitoring may largely depend upon which
+mode you choose to use.  The more advanced load balancing modes do not
+support the use of the ARP monitor, and are thus restricted to using
+the MII monitor (which does not provide as high a level of assurance
+as the ARP monitor).
+
+
+12.2 High Availability in a Multiple Switch Topology
+----------------------------------------------------
+
+        With multiple switches, the configuration of bonding and the
+network changes dramatically.  In multiple switch topologies, there is
+a tradeoff between network availability and usable bandwidth.
+
+        Below is a sample network, configured to maximize the
+availability of the network:
+
+                |                                     |
+                |port3                           port3|
+          +-----+----+                          +-----+----+
+          |          |port2       ISL      port2|          |
+          | switch A +--------------------------+ switch B |
+          |          |                          |          |
+          +-----+----+                          +-----++---+
+                |port1                           port1|
+                |             +-------+               |
+                +-------------+ host1 +---------------+
+                         eth0 +-------+ eth1
+
+        In this configuration, there is a link between the two
+switches (ISL, or inter switch link), and multiple ports connecting to
+the outside world ("port3" on each switch).  There is no technical
+reason that this could not be extended to a third switch.
+
+12.2.1 Bonding Mode Selection for Multiple Switch Topology
+----------------------------------------------------------
+
+        In a topology such as this, the active-backup and broadcast
+modes are the only useful bonding modes; the other modes require all
+links to terminate on the same peer for them to behave rationally.
+
+active-backup: This is generally the preferred mode, particularly if
+        the switches have an ISL and play together well.  If the
+        network configuration is such that one switch is specifically
+        a backup switch (e.g., has lower capacity, higher cost, etc),
+        then the primary option can be used to insure that the
+        preferred link is always used when it is available.
+
+broadcast: This mode is really a special purpose mode, and is suitable
+        only for very specific needs.  For example, if the two
+        switches are not connected (no ISL), and the networks beyond
+        them are totally independant.  In this case, if it is
+        necessary for some specific one-way traffic to reach both
+        independent networks, then the broadcast mode may be suitable.
+
+12.2.2 Link Monitoring Selection for Multiple Switch Topology
+-------------------------------------------------------------
+
+        The choice of link monitoring ultimately depends upon your
+switch.  If the switch can reliably fail ports in response to other
+failures, then either the MII or ARP monitors should work.  For
+example, in the above example, if the "port3" link fails at the remote
+end, the MII monitor has no direct means to detect this.  The ARP
+monitor could be configured with a target at the remote end of port3,
+thus detecting that failure without switch support.
+
+        In general, however, in a multiple switch topology, the ARP
+monitor can provide a higher level of reliability in detecting link
+failures.  Additionally, it should be configured with multiple targets
+(at least one for each switch in the network).  This will insure that,
+regardless of which switch is active, the ARP monitor has a suitable
+target to query.
+
+
+12.3 Switch Behavior Issues for High Availability
+-------------------------------------------------
+
+        You may encounter issues with the timing of link up and down
+reporting by the switch.
+
+        First, when a link comes up, some switches may indicate that
+the link is up (carrier available), but not pass traffic over the
+interface for some period of time.  This delay is typically due to
+some type of autonegotiation or routing protocol, but may also occur
+during switch initialization (e.g., during recovery after a switch
+failure).  If you find this to be a problem, specify an appropriate
+value to the updelay bonding module option to delay the use of the
+relevant interface(s).
+
+        Second, some switches may "bounce" the link state one or more
+times while a link is changing state.  This occurs most commonly while
+the switch is initializing.  Again, an appropriate updelay value may
+help, but note that if all links are down, then updelay is ignored
+when any link becomes active (the slave closest to completing its
+updelay is chosen).
+
+        Note that when a bonding interface has no active links, the
+driver will immediately reuse the first link that goes up, even if
+updelay parameter was specified.  If there are slave interfaces
+waiting for the updelay timeout to expire, the interface that first
+went into that state will be immediately reused.  This reduces down
+time of the network if the value of updelay has been overestimated.
+
+        In addition to the concerns about switch timings, if your
+switches take a long time to go into backup mode, it may be desirable
+to not activate a backup interface immediately after a link goes down.
+Failover may be delayed via the downdelay bonding module option.
+
+13. Hardware Specific Considerations
+====================================
+
+        This section contains additional information for configuring
+bonding on specific hardware platforms, or for interfacing bonding
+with particular switches or other devices.
+
+13.1 IBM BladeCenter
+--------------------
+
+        This applies to the JS20 and similar systems.
+
+        On the JS20 blades, the bonding driver supports only
+balance-rr, active-backup, balance-tlb and balance-alb modes.  This is
+largely due to the network topology inside the BladeCenter, detailed
+below.
+
+JS20 network adapter information
+--------------------------------
+
+        All JS20s come with two Broadcom Gigabit Ethernet ports
+integrated on the planar.  In the BladeCenter chassis, the eth0 port
+of all JS20 blades is hard wired to I/O Module #1; similarly, all eth1
+ports are wired to I/O Module #2.  An add-on Broadcom daughter card
+can be installed on a JS20 to provide two more Gigabit Ethernet ports.
+These ports, eth2 and eth3, are wired to I/O Modules 3 and 4,
+respectively.
+
+        Each I/O Module may contain either a switch or a passthrough
+module (which allows ports to be directly connected to an external
+switch).  Some bonding modes require a specific BladeCenter internal
+network topology in order to function; these are detailed below.
+
+        Additional BladeCenter-specific networking information can be
+found in two IBM Redbooks (www.ibm.com/redbooks):
+
+"IBM eServer BladeCenter Networking Options"
+"IBM eServer BladeCenter Layer 2-7 Network Switching"
+
+BladeCenter networking configuration
+------------------------------------
+
+        Because a BladeCenter can be configured in a very large number
+of ways, this discussion will be confined to describing basic
+configurations.
+
+        Normally, Ethernet Switch Modules (ESM) are used in I/O
+modules 1 and 2.  In this configuration, the eth0 and eth1 ports of a
+JS20 will be connected to different internal switches (in the
+respective I/O modules).
+
+        An optical passthru module (OPM) connects the I/O module
+directly to an external switch.  By using OPMs in I/O module #1 and
+#2, the eth0 and eth1 interfaces of a JS20 can be redirected to the
+outside world and connected to a common external switch.
+
+        Depending upon the mix of ESM and OPM modules, the network
+will appear to bonding as either a single switch topology (all OPM
+modules) or as a multiple switch topology (one or more ESM modules,
+zero or more OPM modules).  It is also possible to connect ESM modules
+together, resulting in a configuration much like the example in "High
+Availability in a multiple switch topology."
+
+Requirements for specifc modes
+------------------------------
+
+        The balance-rr mode requires the use of OPM modules for
+devices in the bond, all connected to an common external switch.  That
+switch must be configured for "etherchannel" or "trunking" on the
+appropriate ports, as is usual for balance-rr.
+
+        The balance-alb and balance-tlb modes will function with
+either switch modules or passthrough modules (or a mix).  The only
+specific requirement for these modes is that all network interfaces
+must be able to reach all destinations for traffic sent over the
+bonding device (i.e., the network must converge at some point outside
+the BladeCenter).
+
+        The active-backup mode has no additional requirements.
+
+Link monitoring issues
+----------------------
+
+        When an Ethernet Switch Module is in place, only the ARP
+monitor will reliably detect link loss to an external switch.  This is
+nothing unusual, but examination of the BladeCenter cabinet would
+suggest that the "external" network ports are the ethernet ports for
+the system, when it fact there is a switch between these "external"
+ports and the devices on the JS20 system itself.  The MII monitor is
+only able to detect link failures between the ESM and the JS20 system.
+
+        When a passthrough module is in place, the MII monitor does
+detect failures to the "external" port, which is then directly
+connected to the JS20 system.
+
+Other concerns
+--------------
+
+        The Serial Over LAN link is established over the primary
+ethernet (eth0) only, therefore, any loss of link to eth0 will result
+in losing your SoL connection.  It will not fail over with other
+network traffic.
+
+        It may be desirable to disable spanning tree on the switch
+(either the internal Ethernet Switch Module, or an external switch) to
+avoid fail-over delays issues when using bonding.
+
+        
+14. Frequently Asked Questions
+==============================
+
+1.  Is it SMP safe?
+
+        Yes. The old 2.0.xx channel bonding patch was not SMP safe.
+The new driver was designed to be SMP safe from the start.
+
+2.  What type of cards will work with it?
+
+        Any Ethernet type cards (you can even mix cards - a Intel
+EtherExpress PRO/100 and a 3com 3c905b, for example).  They need not
+be of the same speed.
+
+3.  How many bonding devices can I have?
+
+        There is no limit.
+
+4.  How many slaves can a bonding device have?
+
+        This is limited only by the number of network interfaces Linux
+supports and/or the number of network cards you can place in your
+system.
+
+5.  What happens when a slave link dies?
+
+        If link monitoring is enabled, then the failing device will be
+disabled.  The active-backup mode will fail over to a backup link, and
+other modes will ignore the failed link.  The link will continue to be
+monitored, and should it recover, it will rejoin the bond (in whatever
+manner is appropriate for the mode). See the section on High
+Availability for additional information.
+        
+        Link monitoring can be enabled via either the miimon or
+arp_interval paramters (described in the module paramters section,
+above).  In general, miimon monitors the carrier state as sensed by
+the underlying network device, and the arp monitor (arp_interval)
+monitors connectivity to another host on the local network.
+
+        If no link monitoring is configured, the bonding driver will
+be unable to detect link failures, and will assume that all links are
+always available.  This will likely result in lost packets, and a
+resulting degredation of performance.  The precise performance loss
+depends upon the bonding mode and network configuration.
+
+6.  Can bonding be used for High Availability?
+
+        Yes.  See the section on High Availability for details.
+
+7.  Which switches/systems does it work with?
+
+        The full answer to this depends upon the desired mode.
+
+        In the basic balance modes (balance-rr and balance-xor), it
+works with any system that supports etherchannel (also called
+trunking).  Most managed switches currently available have such
+support, and many unmananged switches as well.
+
+        The advanced balance modes (balance-tlb and balance-alb) do
+not have special switch requirements, but do need device drivers that
+support specific features (described in the appropriate section under
+module paramters, above).
+
+        In 802.3ad mode, it works with with systems that support IEEE
+802.3ad Dynamic Link Aggregation.  Most managed and many unmanaged
+switches currently available support 802.3ad.
+
+        The active-backup mode should work with any Layer-II switch.
+
+8.  Where does a bonding device get its MAC address from?
+
+        If not explicitly configured with ifconfig, the MAC address of
+the bonding device is taken from its first slave device. This MAC
+address is then passed to all following slaves and remains persistent
+(even if the the first slave is removed) until the bonding device is
+brought down or reconfigured.
+
+        If you wish to change the MAC address, you can set it with
+ifconfig:
+
+# ifconfig bond0 hw ether 00:11:22:33:44:55
+
+        The MAC address can be also changed by bringing down/up the
+device and then changing its slaves (or their order):
+
+# ifconfig bond0 down ; modprobe -r bonding
+# ifconfig bond0 .... up
+# ifenslave bond0 eth...
+
+        This method will automatically take the address from the next
+slave that is added.
+
+        To restore your slaves' MAC addresses, you need to detach them
+from the bond (`ifenslave -d bond0 eth0'). The bonding driver will
+then restore the MAC addresses that the slaves had before they were
+enslaved.
+
+15. Resources and Links
+=======================
+
+The latest version of the bonding driver can be found in the latest
+version of the linux kernel, found on http://kernel.org
+
+Discussions regarding the bonding driver take place primarily on the
+bonding-devel mailing list, hosted at sourceforge.net.  If you have
+questions or problems, post them to the list.
+
+bonding-devel@lists.sourceforge.net
+
+https://lists.sourceforge.net/lists/listinfo/bonding-devel
+
+There is also a project site on sourceforge.
+
+http://www.sourceforge.net/projects/bonding
+
+Donald Becker's Ethernet Drivers and diag programs may be found at :
+ - http://www.scyld.com/network/
+
+You will also find a lot of information regarding Ethernet, NWay, MII,
+etc. at www.scyld.com.
+
+-- END --