32.6 Link Aggregation and Failover

Written by Andrew Thompson.

32.6.1 Introduction

The lagg(4) interface allows aggregation of multiple network interfaces as one virtual interface for the purpose of providing fault-tolerance and high-speed links.

32.6.2 Operating Modes

Failover

Sends and receives traffic only through the master port. If the master port becomes unavailable, the next active port is used. The first interface added is the master port; any interfaces added after that are used as failover devices. If failover to a non-master port occurs, the original port will become master when it becomes available again.

Cisco Fast EtherChannel

Cisco Fast EtherChannel (FEC), is a static setup and does not negotiate aggregation with the peer or exchange frames to monitor the link. If the switch supports LACP then that should be used instead.

FEC balances outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. The hash includes the Ethernet source and destination address, and, if available, the VLAN tag, and the IPv4/IPv6 source and destination address.

LACP

The IEEE 802.3ad Link Aggregation Control Protocol (LACP) and the Marker Protocol. LACP will negotiate a set of aggregable links with the peer in to one or more Link Aggregated Groups (LAG). Each LAG is composed of ports of the same speed, set to full-duplex operation. The traffic will be balanced across the ports in the LAG with the greatest total speed, in most cases there will only be one LAG which contains all ports. In the event of changes in physical connectivity, Link Aggregation will quickly converge to a new configuration.

LACP balances outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. The hash includes the Ethernet source and destination address, and, if available, the VLAN tag, and the IPv4/IPv6 source and destination address.

Loadbalance

This is an alias of FEC mode.

Round-robin

Distributes outgoing traffic using a round-robin scheduler through all active ports and accepts incoming traffic from any active port. This mode violates Ethernet Frame ordering and should be used with caution.

32.6.3 Examples

Example 32-1. LACP aggregation with a Cisco Switch

This example connects two interfaces on a FreeBSD machine to the switch as a single load balanced and fault tolerant link. More interfaces can be added to increase throughput and fault tolerance. Since frame ordering is mandatory on Ethernet links then any traffic between two stations always flows over the same physical link limiting the maximum speed to that of one interface. The transmit algorithm attempts to use as much information as it can to distinguish different traffic flows and balance across the available interfaces.

On the Cisco switch add the FastEthernet0/1 and FastEthernet0/2 interfaces to the channel-group 1:

interface FastEthernet0/1
 channel-group 1 mode active
 channel-protocol lacp
!
interface FastEthernet0/2
 channel-group 1 mode active
 channel-protocol lacp

Create the lagg(4) interface using fxp0 and fxp1, and bring the interfaces up with the IP Address of 10.0.0.3/24:

# ifconfig fxp0 up
# ifconfig fxp1 up
# ifconfig lagg0 create 
# ifconfig lagg0 up laggproto lacp laggport fxp0 laggport fxp1 10.0.0.3/24

View the interface status by running:

# ifconfig lagg0

Ports marked as ACTIVE are part of the active aggregation group that has been negotiated with the remote switch and traffic will be transmitted and received. Use the verbose output of ifconfig(8) to view the LAG identifiers.

lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:05:5d:71:8d:b8
        media: Ethernet autoselect
        status: active
        laggproto lacp
        laggport: fxp1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: fxp0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

To see the port status on the switch, use show lacp neighbor:

switch# show lacp neighbor 
Flags:  S - Device is requesting Slow LACPDUs 
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode     

Channel group 1 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID         Age     Key     Number   State
Fa0/1     SA      32768     0005.5d71.8db8  29s    0x146   0x3      0x3D  
Fa0/2     SA      32768     0005.5d71.8db8  29s    0x146   0x4      0x3D

For more detail use the show lacp neighbor detail command.

To retain this configuration across reboots, the following entries can be added to /etc/rc.conf:

ifconfig_fxp0="up"
ifconfig_fxp1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto lacp laggport fxp0 laggport fxp1 10.0.0.3/24"

Example 32-2. Failover mode

Failover mode can be used to switch over to a secondary interface if the link is lost on the master interface. Bring the underlying physical interfaces up. Create the lagg(4) interface, using fxp0 as the master interface and fxp1 as the secondary interface and assign an IP Address of 10.0.0.15/24:

# ifconfig fxp0 up
# ifconfig fxp1 up
# ifconfig lagg0 create
# ifconfig lagg0 up laggproto failover laggport fxp0 laggport fxp1 10.0.0.15/24

The interface will look something like this, the major differences will be the MAC address and the device names:

# ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:05:5d:71:8d:b8
        inet 10.0.0.15 netmask 0xffffff00 broadcast 10.0.0.255
        media: Ethernet autoselect
        status: active
        laggproto failover
        laggport: fxp1 flags=0<>
        laggport: fxp0 flags=5<MASTER,ACTIVE>

Traffic will be transmitted and received on fxp0. If the link is lost on fxp0 then fxp1 will become the active link. If the link is restored on the master interface then it will once again become the active link.

To retain this configuration across reboots, the following entries can be added to /etc/rc.conf:

ifconfig_fxp0="up"
ifconfig_fxp1="up"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto failover laggport fxp0 laggport fxp1 10.0.0.15/24"

Example 32-3. Failover mode between wired and wireless interfaces

For laptop users, it is usually desirable to make wireless as a secondary interface, which is to be used when the wired connection is not available. With lagg(4), it is possible to use one IP address, prefer the wired connection for both performance and security reasons, while maintaining the ability to transfer data over the wireless connection.

In this setup, we will need to override the underlying wireless interface's MAC address to match the lagg(4)'s, which is inherited from the master interface being used, the wired interface.

In this setup, we will treat the wired interface, bge0, as the master, and the wireless interface, wlan0, as the failover interface. The wlan0 was created from iwn0 which we will set up with the wired connection's MAC address. The first step would be to obtain the MAC address from the wired interface:

# ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:21:70:da:ae:37
        inet6 fe80::221:70ff:feda:ae37%bge0 prefixlen 64 scopeid 0x2 
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

You can replace the bge0 to match your reality, and will get a different ether line which is the MAC address of your wired interface. Now, we change the underlying wireless interface, iwn0:

# ifconfig iwn0 ether 00:21:70:da:ae:37

Bring up the wireless interface but don't set up any IP address on it:

# ifconfig wlan0 create wlandev iwn0 ssid my_router up

Bring the bge0 interface up. Create the lagg(4) interface with bge0 as master, and failover to wlan0 if necessary:

# ifconfig bge0 up
# ifconfig lagg0 create
# ifconfig lagg0 up laggproto failover laggport bge0 laggport wlan0

The interface will look something like this, the major differences will be the MAC address and the device names:

# ifconfig lagg0
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:21:70:da:ae:37
        media: Ethernet autoselect
        status: active
        laggproto failover
        laggport: wlan0 flags=0<>
        laggport: bge0 flags=5<MASTER,ACTIVE>

Then start the DHCP client to obtain an IP address:

# dhclient lagg0

To retain this configuration across reboots, the following entries can be added to /etc/rc.conf:

ifconfig_bge0="up"
ifconfig_iwn0="ether 00:21:70:da:ae:37"
wlans_iwn0="wlan0"
ifconfig_wlan0="WPA"
cloned_interfaces="lagg0"
ifconfig_lagg0="laggproto failover laggport bge0 laggport wlan0 DHCP"