Edge Image Builder SLE-Micro 6.0 Network Issues for Static IP's and BONDs

[jmmckenz]

Edge Image Builder SLE-Micro 6.0 Network Issues
When attempting to setup network interfaces on hardware with multiple controllers, interface enumeration is inconsistent and causes issues with setting up NICS and BONDS.
Attempted to use predictable network interface names, but that doesn’t take effect until the system is rebooted (during combustion phase net.ifnames is still set to 0), so networking does not get applied.
As an example, a VM using 4 different network controllers was created to demonstrate this issue.
On each successive reboot, eth0 through eth3 would move around. When trying to bond eth0 and eth1 in this situation, there is a 1/12 chance that the NIC’s will match what is in the network setup and identify the hostname and config that need to be applied to the system.

Other Observations:
The nmconnection files being created during EIB are ambiguous as far as MAC address for the interface being created. For example for a non-bonded interface (generated config found in combustion files during EIB build):

[connection]
autoconnect=true
autoconnect-slaves=-1
id=eth0
interface-name=eth0
type=802-3-ethernet
uuid=dfd202f5-562f-5f07-8f2a-a7717756fb70

[ipv4]
address0=192.168.50.242/24
dhcp-timeout=2147483647
dns=10.0.0.94
dns-priority=40
method=manual
route0=0.0.0.0/0,192.168.50.1,100
route0_options=table=254

[ipv6]
addr-gen-mode=0
dhcp-timeout=2147483647
method=disabled

[ethernet]
cloned-mac-address=B6:59:B4:29:6E:91

The issue here is using the “cloned-mac-address” setting as opposed to “mac-address”. This directly relates to issue 66 (linked below). In a situation where the NIC order is unpredictable, it appears that NetworkManager is applying the “cloned-mac-address” to the first NIC it finds, regardless of what its’ permanent MAC is. This causes conflict with the NIC that has the real MAC, so that interface does not get setup. In the case of the bond, I ended up with a split bond where the first NIC was not what I was looking for and squashed the MAC of the NIC it was supposed to go to, and by chance the second NIC was on the correct network, but again the effective MAC was incorrect.

Related issues:
#19
#66
#105

How I worked around this:
Setup a custom script to populate system runonce.service and the script it needs to run. It also copies over the network folder for use after combustion is run.
https://github.com/jmmckenz/examples/blob/main/edge-image-builder/custom/scripts/19-force-network-setup
Setup os-files for runonce.service and associated script
https://github.com/jmmckenz/examples/blob/main/edge-image-builder/os-files/etc/systemd/system/runonce.service
https://github.com/jmmckenz/examples/blob/main/edge-image-builder/os-files/root/bin/combustion/bonded-network-fix-to-predictable-names.sh
Chose non NIC names for the NICs in my network file, to be used as labels instead of interface-names.
https://github.com/jmmckenz/examples/blob/main/edge-image-builder/network/elemental1.mclocal.yaml

Within the bonded-network-fix-to-predictable-names.sh script, note the changes to the nmconnection file I am doing. I am removing the “interface-name” line completely from the nmconnection file. I am also changing “clone-mac-address” to “mac-address”. The “clone-mac-address” change came after discovering the issue related to issue 66.

I am not sure this is a contributing factor or not, but I also notice at some point we lower-to-upper the MAC address during configuration. My network files all had lowercase MAC addresses, and the files that get created during EIB those MAC addresses are changed to UPPER.

Test Environment:
Harvester VM
4 NICS (NIC1 and NIC2 on one network, NIC3 and NIC4 on another network)
NIC1 – virtio
NIC2 – e1000
NIC3 – e1000g
NIC4 – rtl8139
EIB notes:
kernelArgs: net.ifnames=1
Network config / os-files / custom scripts as linked above.