Debian 11 (Bullseye) and the problem with bond (bonding) LACP interfaces

Written by - 10 comments

Published on September 1st 2021 - last updated on October 17th 2021 - Listed in Linux Hardware Network


Back in June I did my first server tests with Debian 11 (Bullseye). Back then with Release Candidate 1, I realized something's off when using bonded interfaces, using 802.3ad or LACP bonding.

Trying bond-master config (from Debian Buster)

Basically the same bond interface config was taken from a Debian 10 (Buster) machine:

cka@buster:~$ cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# First interface enp3s0f0
auto enp3s0f0
iface enp3s0f0 inet manual
  bond-master bond0

# Second interface enp3s0f1
auto enp3s0f1
iface enp3s0f1 inet manual
  bond-master bond0

# Frontend bond interface
auto bond0
iface bond0 inet static
  address 192.168.100.4/24
  gateway 192.168.100.1
  bond-mode 802.3ad
  bond-miimon 100
  bond-lacp-rate 1

The bond interface is in this case defined in the physical interface stanzas: Both enp3s0f0 and enp3s0f1 interfaces point to a bond-master interface, bond0 in this case.

But using this config does not work in Debian 11!

Trying slaves (interfaces) config (from official documentation)

The official Debian bonding documentation mentions to define the physical interfaces on the bonding interface itself using the slaves keyword:

auto enp3s0f0
iface enp3s0f0 inet manual

auto enp3s0f1
iface enp3s0f1 inet manual

auto bond0
iface bond0 inet static
  address 192.168.100.4/24
  gateway 192.168.100.1
  slaves enp3s0f0 enp3s0f1
  bond-mode 802.3ad
  bond-miimon 100
  bond-downdelay 200
  bond-updelay 200

But this config does not work on Bullseye, either!

A (weird mix) but kind of working config

By testing multiple configuration options, I finally was able to get LACP working with the following config:

auto enp4s0f0
iface enp4s0f0 inet manual
  bond-master bond0

auto enp4s0f1
iface enp4s0f1 inet manual
  bond-master bond0

auto bond0
iface bond0 inet static
  address 192.168.100.4/24
  gateway 192.168.100.1
  bond-slaves none
  bond-mode 802.3ad
  bond-miimon 100
  bond-downdelay 200
  bond-updelay 200

Here both the bond-master interface is defined in the physical interfaces, yet on the bond interface itself bond-slaves is set to none.

This kinda worked - but it still is a weird workaround mixing up master and slave configuration.

Bug report 990428

Hence I opened up Debian bug report #990428, mainly because of two facts:

  • Existing bonding configuration (from Debian 10) would not work anymore -> regression
  • The (Debian) documented way of bonding configuration does not work either

My hope was obviously that this would be fixed before the official Debian 11 Bullseye release. But here we are, a few months later with Debian 11.0 released and with the same bug still in place :-(.

In the meantime Oleander Reis shared some insights why this happens. Debian 11 comes with a newer version (2.12) of the ifenslave package. This package contains the code which configures interface slaves to a bond interface. Unfortunately this newer version introduces a couple of issues concerning interface bonding, failing on "bond-master" configurations, for example:

Sep 01 15:58:23 bullseye ifup[1251]: No iface stanza found for master bond0
Sep 01 15:58:23 bullseye ifup[1249]: run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1
Sep 01 15:58:23 bullseye ifup[1242]: ifup: failed to bring up enp3s0f0
Sep 01 15:58:23 bullseye ifup[1256]: No iface stanza found for master bond0
Sep 01 15:58:23 bullseye ifup[1254]: run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1
Sep 01 15:58:23 bullseye ifup[1242]: ifup: failed to bring up enp3s0f1

Another important hint from Oleander was to remove the physical interface stanzas from /etc/network/interfaces (only keep bond interfaces).

Working bond interface config in Debian 11 Bullseye

Using this information from Oleander leads to the following config:

root@bullseye:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# Frontend bond interface
auto bond0
iface bond0 inet static

  address 192.168.100.4/24
  gateway 192.168.100.1
  bond-slaves enp3s0f0 enp3s0f1
  bond-mode 802.3ad
  bond-miimon 100
  bond-lacp-rate 1

Note: There are no physical interfaces (enp3s0f0 and enp3s0f1) defined in the interfaces file!

This works surprisingly well, and systemctl status networking shows everything OK:

root@bullseye:~# systemctl status networking
- networking.service - Raise network interfaces
     Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2021-09-01 16:07:52 CEST; 50min ago
       Docs: man:interfaces(5)
    Process: 688 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=0/SUCCESS)
   Main PID: 688 (code=exited, status=0/SUCCESS)
        CPU: 135ms

Sep 01 16:07:51 bullseye systemd[1]: Starting Raise network interfaces...
Sep 01 16:07:52 bullseye systemd[1]: Finished Raise network interfaces.

And the bond interface is correctly up, which can be checked using /proc/net/bonding/bond0:

root@bullseye:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.10.0-8-amd64

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: XX:XX:XX:XX:XX:XX
Active Aggregator Info:
    Aggregator ID: 2
    Number of ports: 2
    Actor Key: 9
    Partner Key: 32787
    Partner Mac Address: XX:XX:XX:XX:XX:XX

Slave Interface: enp3s0f0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: XX:XX:XX:XX:XX:XX
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: XX:XX:XX:XX:XX:XX
    port key: 9
    port priority: 255
    port number: 1
    port state: 63
details partner lacp pdu:
    system priority: 32667
    system mac address: XX:XX:XX:XX:XX:XX
    oper key: 32787
    port priority: 32768
    port number: 16659
    port state: 61

Slave Interface: enp3s0f1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: XX:XX:XX:XX:XX:XX
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: XX:XX:XX:XX:XX:XX
    port key: 9
    port priority: 255
    port number: 2
    port state: 63
details partner lacp pdu:
    system priority: 32667
    system mac address: XX:XX:XX:XX:XX:XX
    oper key: 32787
    port priority: 32768
    port number: 275
    port state: 61

Fixed in ifenslave 2.13

Update October 17th 2021: The mentioned Debian bug #990428 was closed today and mentions that the bug has been fixed in the package ifenslave 2.13. The following changes are made to the new package version:

  • Fix MAC address setting messed up by udev for bond interfaces. (Closes: #949062). By Guillem Jover.
  • Use ifquery instead of example contrib script ifstate. (Closes: #991930). By Guillem Jover.
  • Fix ifquery redirections. By Guillem Jover.
  • Bump Standards-Version to 4.6.0 (no changed needed). By Guillem Jover.
  • Remove long supported Linux version requirements from Description. By Guillem Jover.
  • Use correct argument in setup_slave_device(). (Closes: #968368). By Sami Haahtinen.
  • Handle slave definitions of interfaces with no bond settings. (Closes: #990428). By Oleander Reis.
  • Delete bond interfaces on ifdown -a. (Closes: #992102). By Oleander Reis.

Right now, the current ifenslave version on Bullseye is still  2.12, but 2.13 should be available in the next few days.


Add a comment

Show form to leave a comment

Comments (newest first)

Olivier from France wrote on Nov 14th, 2021:

Nice work ! This is the perfect answer to my current concerns.
Even the Debian wiki shows a weird configuration, not working.
Updating to bullseye on my production server was a mess... but not anymore. Many thanks to you.

P.S: unfortunately the current stable version is 2.12 in bullseye. The testing version is needed.


Clay from wrote on Oct 30th, 2021:

Thanks for submitting the bug report and providing this excellent write-up. Really helped me out after I ran into this on a Debian 10 -> 11 upgrade, having missed it in the upgrade documentation. Thank you.


b1001101 from wrote on Oct 18th, 2021:

I stumbled across your article while I was investigating a bizarre behaviour of bonded (and, in my case, bridged) interfaces on one of my boxes; although it turned out to be something entirely unrelated, I thank you for emphasizing a couple of points I integrated in my configuration for which I thank you. No issue whatsoever since its original configuration (Debian 10) lacking the physical interface stanzas (fun fact: when defined, the individual interfaces ended up getting an unsought address from isc-dhcpd-server). Anyway, in the event it can be useful to anyone, this is me (Debian 11, ifenslave 2.12)

auto bond0
iface bond0 inet manual
bond-mode 4
bond-primary eno1
bond-slaves eno1 enp2s0
bond-miimon 100
bond-lacp-rate 1

auto br0
iface br0 inet static
address 192.168.16.16/24
gateway 192.168.16.1
dns-nameservers 192.168.16.32
dns-search whatever.lan
bridge_ports bond0
bridge_stp off
bridge_waitport 0
bridge_fd 0

No issue whatsoever since Debian 10. Keep up the good work, mate!


ck from Switzerland wrote on Oct 12th, 2021:

Hi Luka. Unfortunately I do not know why this setup would not work anymore. I suggest to ask the mailing list (debian-user@lists.debian.org). Maybe VLAN configuration has also changed in Bullseye.


Luka from Slovakia wrote on Oct 11th, 2021:

I have tried your example, but it is not still working. :( I have checked networking status:

"ifup: Failed to enslave eno1.100 to bond0.Is bond0 ready and a bonding interface?"

Thanks.


ck from Switzerland wrote on Oct 11th, 2021:

If I remember correctly, each VLAN needs a different bonding interface, tagged with a VLAN. For example for VLAN100:

auto bond0
iface bond0 inet manual
bond-slaves eno1.100 eno2.100

auto bond0.100
iface bond0.100 inet static
address x.x.x.x
netmask 255.255.255.0
gateway x.x.x.x
vlan-raw-device bond0

Something like this. I did not test the config above.


Lukas Mastilak from Slovakia wrote on Oct 10th, 2021:

Sorry, I removed the configuration of physical interfaces from /etc/network/interfaces. In my setup, I have two physical interfaces and two vlans. For example:
auto bond0
iface bond0 inet static
bond-slaves eno1.100 eno2.100
address x.x.x.x
netmask 255.255.255.0
gateway x.x.x.x


My notes:
1. without VLANs the networking is working after reboot
2. with VLANs, when I reboot server, the network is not working. I must add physical interfaces into /etc/network/interfaces and restart networking. Then I will again remove physical interfaces from config and restart networking. Subsequently, the networking already is working.

Also, I checked module bonding if it is loaded after reboot.


ck from Switzerland wrote on Oct 10th, 2021:

Side note, make sure that the "bonding" Kernel module is loaded during boot (initramfs). Maybe this is missing in your setup?


ck from Switzerland wrote on Oct 10th, 2021:

Luka, how do you want to make a bonding interface without physical interfaces? The config I shared in the post works and survives a reboot. Working correctly with Debian Bullseye on HP DL380 server (4 on board nics).


Luka from Slovakia wrote on Oct 10th, 2021:

Hi, I have tried to configure bond interface without physical interfaces. After the reboot system, this config is not working. :(