Recently I moved some servers around and face a task to setup LACP between server and switches (Dell’s S4048-ON stack).

hardware

Stack (2 switches) : DELL (Force10) S4048-ON, Software Version: 9.8

Server : Supermicro, Intel Corporation 82599ES 10-Gigabit SFI/SFP+ network card, CentOS 7 (release 7.7.1908)

topology

Server connected with fibreoptic cables (OM3+) to stack, ports ( TenGigabitEthernet 1/10, TenGigabitEthernet 2/10 ).

configuration

server side

Server’s interfaces (enp5s0f0, enp5s0f1) configuration files and bond0 configuration looks like bellow. Notice I set MTU to 9000 ( default is 1500 ).

[root@server /etc/sysconfig/network-scripts]$ cat ifcfg-enp5s0f0
DEVICE=enp5s0f0
NAME=enp5s0f0
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
ONBOOT=yes
NM_CONTROLLED=no
MTU=9000
MASTER=bond0
SLAVE=yes

[root@server /etc/sysconfig/network-scripts]$ cat ifcfg-enp5s0f1
DEVICE=enp5s0f1
NAME=enp5s0f1
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
ONBOOT=yes
NM_CONTROLLED=no
MTU=9000
MASTER=bond0
SLAVE=yes

[root@server /etc/sysconfig/network-scripts]$ cat ifcfg-bond0
DEVICE=bond0
TYPE=Bond
NAME=bond0
BONDING_MASTER=yes
BOOTPROTO=none
ONBOOT=on
IPADDR=10.1.1.22
NETMASK=255.255.255.0
BONDING_OPTS="mode=4 miimon=100 lacp_rate=1"
NM_CONTROLLED=no
USERCTL=no
IPV6INIT=no
MTU=9000
switch side

Bellow commands I used to configure stack.

interface TenGigabitEthernet 1/10
 description server-enp5s0f0
 no ip address
 mtu 10200

 port-channel-protocol LACP
  port-channel 10 mode active
 no shutdown

interface TenGigabitEthernet 2/10
 description server-enp5s0f1
 no ip address
 mtu 10200

 port-channel-protocol LACP
  port-channel 10 mode active
 no shutdown

interface Port-channel 10
 description server-lacp
 no ip address
 mtu 10200
 switchport
 lacp long-timeout
 no shutdown

I use non default vlan for this connection.

interface Vlan 45
 untagged Port-channel 10

verification

LACP is dynamic, so unless everything works ( server is configured, cables are connected, … ) port-channel status will be ‘‘down’’ and there will be no ports affiliated.

switch# show interfaces port-channel 10 brief
Codes: L - LACP Port-channel
       O - OpenFlow Controller Port-channel
       A - Auto Port-channel
       I - Internally Lagged

    LAG  Mode  Status       Uptime      Ports
L   10   L2    down         00:00:00

And this is how connection looks when it is up and running.

switch# show interfaces port-channel 10 brief
Codes: L - LACP Port-channel
       O - OpenFlow Controller Port-channel
       A - Auto Port-channel
       I - Internally Lagged

    LAG  Mode  Status       Uptime      Ports
L   10   L2    up           00:00:45    Te 1/10    (Up)
                                        Te 2/10    (Up)

Bond interface status on server side:

[root@server ~]$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable

Slave Interface: enp5s0f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: XX:YY:ZZ:AA:BB:00
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0

Slave Interface: enp5s0f1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: XX:YY:ZZ:AA:BB:01
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0

Output of ‘‘ip address’’ command

...
4: enp5s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether XX:YY:ZZ:AA:BB:00 brd ff:ff:ff:ff:ff:ff
5: enp5s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether XX:YY:ZZ:AA:BB:00 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    link/ether XX:YY:ZZ:AA:BB:00 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.22/24 brd 10.1.1.255 scope global bond0
       valid_lft forever preferred_lft forever
...