Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

systemd networkd configuration broken #6655

Closed
afbjorklund opened this issue Feb 16, 2020 · 8 comments · Fixed by #6659
Closed

systemd networkd configuration broken #6655

afbjorklund opened this issue Feb 16, 2020 · 8 comments · Fixed by #6659
Labels
area/guest-vm General configuration issues with the minikube guest VM co/runtime/crio CRIO related issues co/runtime/docker Issues specific to a docker runtime kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@afbjorklund
Copy link
Collaborator

afbjorklund commented Feb 16, 2020 •
edited
Loading

The eth0 and eth1 don't come up correctly in networkctl, at least not in VirtualBox.

● systemd-networkd-wait-online.service - Wait for Network to be Configured
   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:systemd-networkd-wait-online.service(8)

$ networkctl
IDX LINK             TYPE               OPERATIONAL SETUP     
  1 lo               loopback           carrier     unmanaged 
  2 eth0             ether              routable    configuring
  3 eth1             ether              routable    configuring
  4 sit0             sit                off         unmanaged 
  5 docker0          bridge             routable    unmanaged 
  7 veth7da0c9b      ether              carrier     unmanaged 
  9 veth03151ca      ether              carrier     unmanaged 

7 links listed.

This is a problem when you have another systemd unit that waits for network-online

It will wait for 2 minutes (120 seconds), and then fail with a timeout - the "dead" above.

For this reason, CRI-O is slow to start.

See cri-o/cri-o@ebf507d

Our Docker unit just waits on "network"

minikube docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target  minikube-automount.service docker.socket
Requires= minikube-automount.service docker.socket

docker.com docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket

See moby/moby@dfa4e77
@afbjorklund afbjorklund added area/guest-vm General configuration issues with the minikube guest VM kind/bug Categorizes issue or PR as related to a bug. co/runtime/docker Issues specific to a docker runtime co/runtime/crio CRIO related issues labels Feb 16, 2020
@afbjorklund afbjorklund mentioned this issue Feb 16, 2020
Merged
@afbjorklund
Copy link
Collaborator Author 

$ SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/systemd-networkd-wait-online
Found link 9
Found link 7
Found link 5
Found link 4
Found link 3
Found link 2
Found link 1
ignoring: lo
link eth1 is being processed by networkd
Event loop failed: Connection timed out

@afbjorklund afbjorklund added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Feb 16, 2020
@afbjorklund
Copy link
Collaborator Author

Something is buggy with the current configuration, since it works OK without it...

$ cat /etc/systemd/network/10-eth1.network 
[Match]
Name=eth1
Virtualization=qemu

[Network]
DHCP=ipv4

[DHCP]
UseDNS=false
$ cat /etc/systemd/network/20-dhcp.network 
[Match]
Name=eth*

[Network]
DHCP=ipv4

[DHCP]
ClientIdentifier=mac

https://github.com/kubernetes/minikube/tree/master/deploy/iso/minikube-iso/board/coreos/minikube/rootfs-overlay/etc/systemd/network

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Feb 16, 2020 •
edited
Loading 

Feb 16 11:07:42 minikube systemd-networkd[28381]: eth1: DHCPv4 address 192.168.99.100/24
Feb 16 11:07:42 minikube systemd-networkd[28381]: eth0: DHCPv4 address 10.0.2.15/24 via 10.0.2.2
Feb 16 11:07:42 minikube systemd-networkd[28381]: eth1: DHCP: No gateway received from DHCP server: No data available

This is probably not the configuration, but more like bugs in systemd 240.

@afbjorklund afbjorklund changed the title systemd network configuration broken systemd networkd configuration broken Feb 16, 2020
@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Feb 16, 2020 •
edited
Loading

Needs to have fixes backported, won't get an updated systemd until next OS upgrade.

https://github.com/systemd/systemd/commits/master/src/network/networkd-dhcp4.c

@afbjorklund
Copy link
Collaborator Author

Well, it is a little happier at least:

Feb 16 13:20:27 minikube systemd-networkd[2025]: eth1: Gained carrier
Feb 16 13:20:27 minikube systemd-networkd[2025]: eth0: Gained carrier
Feb 16 13:20:27 minikube systemd-networkd[2025]: eth1: DHCPv4 address 192.168.99.102/24
Feb 16 13:20:27 minikube systemd-networkd[2025]: eth1: DHCP: No gateway received from DHCP server.
Feb 16 13:20:27 minikube systemd-networkd[2025]: eth0: DHCPv4 address 10.0.2.15/24 via 10.0.2.2

But it still times out and fails. :-(

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Feb 16, 2020 •
edited
Loading

Another important fix:

EDIT: Did not seem to help, must be something else that is blocking the configure/complete.

@afbjorklund afbjorklund mentioned this issue Feb 16, 2020
Merged
@afbjorklund
Copy link
Collaborator Author

Probably same as systemd/systemd#8686

@afbjorklund
Copy link
Collaborator Author

afbjorklund commented Feb 16, 2020 •
edited
Loading

As per workarounds, this made it start:

[Network]
DHCP=ipv4
LinkLocalAddressing=no
$ networkctl 
IDX LINK             TYPE               OPERATIONAL SETUP     
  1 lo               loopback           carrier     unmanaged 
  2 eth0             ether              routable    configured
  3 eth1             ether              routable    configured
  4 sit0             sit                carrier     unmanaged 
  5 mybridge         bridge             carrier     unmanaged 
  6 veth205f338c     ether              carrier     unmanaged 
  7 veth9a5b12ae     ether              carrier     unmanaged 

7 links listed.

Apparently it defaults to ipv6 😵

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
area/guest-vm General configuration issues with the minikube guest VM co/runtime/crio CRIO related issues co/runtime/docker Issues specific to a docker runtime kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add systemd patch for handling DHCP router afbjorklund/minikube
1 participant
@afbjorklund