Connect to network according to network priorities

[ndeloof]

This is a follow-up for docker/cli#1372

see https://gist.github.com/jfellus/cfee9efc1e8e1baf9d15314f16a46eca for a reproduction scenario

[ndeloof]

I was able to confirm we run NetworkConnect in order according to network priorities, but still eth0 is always connected to the first network by lexical order. It seems the actual binding takes place on ContainerStart, need more investigations.

created https://docker.atlassian.net/browse/IL-255 to track this issue internally

[nospam2000]

what is the reason for closing the issue? Has it been fixed? Any reference for tracking in which version it has been fixed?

[ndeloof]

sorry, my bad (was too fast doing issue triage and transfert)

[thediveo]

Any new findings? This is a bug affecting especially container workload developers working with, say, industrial protocols that need direct layer 2 access and it's easy to mess the vNIC ordering up, even when developers clearly spelt-out their deployment requirements in compose v2 using network attachment priorities.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nospam2000]

keep issue open unless resolved

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[vinsgithub]

this bug seems to be still present in compose v3!
Please keep issue open (hopefully solve this :-/ )

[thediveo]

this is a mess, as the priorities also get used in sorting endpoint lists for resolving service names in the embedded DNS resolver.

[Biepa]

Any update on this?

[rodriciru]

Hi. I think this is a big problem.
As Docker compose uses alphabetical order, no mater what priorities do you have, in iptables it only adds the first network, not all.
So if, like me, you have a UFW firewall and add allow the priority IP, it won't be reachable because docker adds other IP

Please fix this ASAP. I really think this is a big problem

[MobaLinux]

There is any update on this? This issue is open for long time now without any resolution ...

[thediveo]

"dead in the water" comes to my mind as an especially fitting metaphor here.

[lox]

Ugh, this is incredibly frustrating.

[lox]

Could at least the documentation on priority be updated to show that this feature is completely not working on either V1 or V2?

[rajil]

This doesn't work in V3 also.

[ndeloof]

After more investigation, it appears priority was introduced to control which network gets service-level configuration mac_address assigned. This is now obsolete as Docker Engine 1.25 will allow mac_address to be declared on a per-network basis.

While docker compose sort networks according to priority to invoke ContainerNetworkConnect API, this has no effect on network interface being assigned, as internally Docker Engine uses a lexical-ordering to store network config before container is created. I haven't found any https://github.com/moby/moby issue requesting this "feature" to be provided, but obviously if this is offered by engine we would adopt it for compose.

Closing this issue as "works as expected" (while not obvious until one dig into project history)

[thediveo]

just to clarify: the sorting bases on the (custom) network names, and not on the compose project network reference name, correct?

[ndeloof]

@thediveo indeed. Docker engine has no visibility on compose network names used for reference by services.

[oldium]

After more investigation, it appears priority was introduced to control which network gets service-level configuration mac_address assigned. This is now obsolete as Docker Engine 1.25 will allow mac_address to be declared on a per-network basis.

While docker compose sort networks according to priority to invoke ContainerNetworkConnect API, this has no effect on network interface being assigned, as internally Docker Engine uses a lexical-ordering to store network config before container is created. I haven't found any https://github.com/moby/moby issue requesting this "feature" to be provided, but obviously if this is offered by engine we would adopt it for compose.

Closing this issue as "works as expected" (while not obvious until one dig into project history)

I have found report mentioning order of network interfaces in moby actually: moby/moby#37201

[JellyKid]

So is networking just broken in Docker / Docker compose, because that's what it seems like if there is no way to properly assign networks to interfaces. That's like networking 101. There is too much "magic" happening behind the scenes. There needs to be a straight forward way to say, we need eth0 on this container to be connected to this network.

[cyberloof]

No, it's not "broken in compose", but in docker engine. Once moby/moby#37201 is implemented compose would be able to offer what you expect

[nospam2000]

The problem is that a fix could break existing configurations. If it is known how the fix would look like we could prepare for it, even if it never comes.

Would the fix use the priority to the order or how would it work?

How can we prepare for a potential fix?

[oldium]

There is a priority attribute, which kind of introduces sorting into the network interfaces world, but it is actually useless. Maybe it is time to fix this.

[nospam2000]

There is a priority attribute, which kind of introduces sorting into the network interfaces world, but it is actually useless. Maybe it is time to fix this.

Is it reliable to assume that this priority field would be used in future for the ordering of the interface assignment?
I would be even more happy if I could give a concrete name of a network interface in the container, e.g. "interface_name: profinet1" instead of auto-naming them eth0, eth1.

The order is not really good because when attaching network interfaces to a running container the names will depend on the order of execution.

For the first interface (the default gateway) it would be ok for me to name that "eth0" because that one has a special role.

Michael

[ndeloof]

priority defines the order container is connected to networks. This has impact on mac_address assignment, and this property was defined for this need. Internally Docker engine sorts network by name and so network interfaces are set accordingly to network names. Depending on the way moby/moby#37201 is fixed, more control could be granted to user for this and Docker Compose would also give you such flexibility

[nospam2000]

priority defines the order container is connected to networks. This has impact on mac_address assignment

IIRC the default MAC address is solely based on the IP address, that means the last 4 digits of the MAC address are the 4 digits of the ipv4 address.

Here an example, 172.17.0.2 = ac:11:00:02 :

64: eth0@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Or do you mean the --mac-address option? Command line parameters dealing with arrays of structures are really confusing. For such things json/yaml files are much better to deal with.

Michael

[oldium]

Yes, it does affect order of attaching network interfaces and does not affect attaching of network interfaces at the same time. I was really surprised that it does not do anything regarding the order of interfaces in the container actually.

[mutax]

It would be really great if the priority field would be used for ordering the interface name assignment.
It was stated earlier, that the order of interfaces also defines the DNS resolution order as well as the default route selection.

This introduces major issues when using external networks in docker compose, as service names that are defined in a compose file might not get resolved correctly and service addresses from other networks the container is connected to are returned.

For example when using a service name 'db' in two compose projects and connecting them via external network, it's undefined which host will be resolved when using the service name unless the host name is extended with the network name. This however depends on the compose project name, so a service configuration now has to be rewritten depending on the name of the project, including the configuration of applications.

This can lead to compose files being readily provided by third parties to break when adding a custom network, for example to add a reverse proxy via service network from another compose project.

In addition the compose spec states that the priority field defines the order of interface connection which for docker also defines the naming of the interfaces, as the first one being attached will be eth0 and get the default route.

All in all this is very frustrating, given that a google search gives me several issues that have been closed describing this exact problem over the last years.

While it might 'work as designed' from a technical standpoint, the documentation is misleading and having the order defined by lexical order of names is unexpected and completely removes the strict separation of docker compose projects when it comes to service name resolution.