[BUG] eth0 default route priority define outbound connection

[mrpops2ko]

Description

hi i've been reading through all the documentation and i'm struggling on how to define which networks in docker compose become the default gateway

from googling i can find various resources which say that its;

1. in the order in which it is defined in the compose file (but this seems not to be true)
2. that it is based upon the alphabetic naming of the networks, so A will begin before B etc (this seems to be true for customer docker networks, but it doesn't work similarly when combined with ipvlan networks)

I stumbled across a few different issues here which talked about priority and i thought that i could use that to define which network becomes the default outbound gateway but that doesn't appear to be the case either, i'm not sure what priority does - from reading the bug thread it mentioned it was something in relation to mac addresses.

Steps To Reproduce

    networks:
      intraNW:
        priority: 10
      br0:
        priority: 30
        ipv4_address: 192.168.1.145
      a-traefik-public:
        priority: 20

take for example this, br0 is an ipvlan network and the other 2 are custom docker networks

my use case scenario is that i want to have as eth0 the br0 interface. the reason for this is because i wish to do split tunnelling on the router and in order to do that i need ipvlan.

it works fine if i remove the other custom networks, but i can't for the life of me seem to be able to get the ipvlan br0 network to become the default eth0 network whilst using docker custom networks alongside it

one possible solution to this seems to be creating a container init script and running that on load, which will change the default route to the desired one but i think this is not a very clean solution compared to a native implementation which docker compose

Compose Version

Docker Compose version v2.29.2

Docker Environment

Client:
 Version:    24.0.9
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 79
  Running: 50
  Paused: 0
  Stopped: 29
 Images: 209
 Server Version: 24.0.9
 Storage Driver: btrfs
  Btrfs:
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.99-Unraid
 Operating System: Slackware 15.0 x86_64 (post 15.0 -current)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 78.56GiB
 Name: alexandria
 ID: xxx
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: hvrpride
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: No swap limit support

Anything else?

No response

[mrpops2ko]

these are some of the other threads on this

#4645
moby/moby#43518
#11229
#8561

this seems like a longstanding issue

[ndeloof]

priority indeed defines which network is connected first when creating container with moby engine, but AFAIK the actual engine implementation does not use it to define default gateway, but does some alphanumeric sorting
@akerouanton might better know the status of this in latest engine codebase

[akerouanton]

@mrpops2ko Could you share you network definitions please?

[mrpops2ko]

@mrpops2ko Could you share you network definitions please?

root@alexandria:~# docker network ls
NETWORK ID     NAME                       DRIVER    SCOPE
e05c170d5559   a-traefik-public           bridge    local
32974c0c0378   br0                        ipvlan    local
f16259b5cce2   bridge                     bridge    local
e09d59b70229   host                       host      local
e75d14eecf63   none                       null      local

root@alexandria:~# docker network inspect e05c170d5559
[
    {
        "Name": "a-traefik-public",
        "Id": "e05c170d55594e375f582aba5d35c06b34d4d86774cd86cf750a39de269faca1",
        "Created": "2024-05-15T00:39:20.032951453+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "0350ae8ef3d8a2b700ccec71d028725fe3e2684a784dbbd3163bc099e3d879ac": {
xxx
            }
        },
        "Options": {},
        "Labels": {}
    }
]

root@alexandria:~# docker network inspect 32974c0c0378
[
    {
        "Name": "br0",
        "Id": "32974c0c0378747271ab726429616985d953d9a2bac744eb94cf231a50ec8199",
        "Created": "2024-09-14T13:42:25.513278678+01:00",
        "Scope": "local",
        "Driver": "ipvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.0.0/20",
                    "Gateway": "192.168.1.1",
                    "AuxiliaryAddresses": {
                        "server": "192.168.1.3"
                    }
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "096fe31bbbf0d98864d108c1f6adab9cd9800451ef0ff8b76af75df77bed345e": {
xxx
            }
        },
        "Options": {
            "parent": "br0"
        },
        "Labels": {}
    }
]

the other networks are all created upon docker compose startup

[mrpops2ko]

i tried playing around with the routing from within the container but nothing seems to work

$ ip route del default via 172.18.0.1 dev eth0 && ip route add default via 192.168.0.1 dev eth1
RTNETLINK answers: Operation not permitted
$ sudo ip route del default via 172.18.0.1 dev eth0 && ip route add default via 192.168.0.1 dev eth1
RTNETLINK answers: Operation not permitted

i even included

    cap_add:
      - NET_ADMIN

and that still didn't allow me to do it, so yeah i'm kind of at a loss on how

[lejacobroy]

I'm having a very similar problem: a bridge network and an ipvlan network are attached to a container, the ipvlan gets set as the default route when I need it to be the bridge network.

Networks are defined as follow:

networks:
  kea-10-ipvlan:  # Network that binds container to host network interface.
    name: kea-10-ipvlan
    driver: ipvlan
    driver_opts:
      parent: ${ETH1}  # Host interface that Kea containers will use (eth1)
    enable_ipv6: false
    ipam:
      config:
        - subnet: ${SUBNET4}  # Subnet for Kea containers
          gateway: ${GATEWAY4}  # Gateway for Kea subnet

  kea-20-backend:  # Internal network for communication with database
    name: kea-20-backend

  access-wg:       
    driver: bridge

And the for the container:

networks:
      access-wg:
      kea-10-ipvlan:         
        ipv4_address: ${DNS_IP4}

The defined order as no impact on the default route, neither does the alphanumerical order.
The priority attribute has no impact on the route ordering.

[ndeloof]

AFAICT this is a docker engine issue. Let me know if you find a way to get the expected result by a combination of docker run.. and docker network connect calls, which compose could replicate

[robmry]

Every time a container is connected to a network, Docker Engine looks at the container's whole set of endpoints and picks one to use as the gateway. (And, the same after a network disconnect.)

The gateway endpoint it picks is the first in this ordering.

I don't think that function's comment is quite right... but, endpoints are sorted by priority, then the dedicated docker_gwbridge network is preferred (for swarm), then non-internal networks, then dual-stack over IPv4-only. Finally, if the networks are equal according to those criteria, they're sorted lexicographically by network name.

That means the order in which networks are connected shouldn't make any difference to the end result:

• Compose's priority only affects the order in which networks are added, the value isn't passed to the engine.
• The epPriority used in the engine's ordering of endpoints is unrelated and, weirdly, there's no way to configure it via the engine's API. (So, it's completely useless.)

But, in the two examples above - the networks are all equal (user-defined, non-internal, and IPv4-only). So, they should be sorted by network name ... I've tried to repro the problem, but haven't been able to. For me, the selected gateway is always based on the network name - including with dockerd 24.0, I don't think this logic has changed since before that release anyway.

I don't think there's anything compose can do to affect this. If we make the engine's priority field useful, we may need to add way for compose to specify it - but we can come back to that.

@mrpops2ko, please could you re-raise this in https://github.com/moby/moby/issues? (Unfortunately we can't move issues between projects.) Then we can dig in to it some more ... I think I'll need a minimal repro, if possible.

[mrpops2ko]

hi @robmry thank you for this detailed explanation, and it is working as you've described it, the problem is that people have many use case scenarios where they don't want it to work as you described.

A > B works and is great but there are scenarios where A < B are required or wanted.

can you explain more about this priority field in docker compose, with a use case scenario? i'm at a loss as to what it actually does if it isn't used to do the thing that a user at first glance would assume it is doing

some parts from reading your comment seem contradictory endpoints are sorted by priority yet i've provided priority and the default gateway doesn't change

maybe thats the delineation? i'm assuming that the first endpoint with the highest priority would result in becoming the eth0 default gateway for internet connectivity, would that be your assumption and understanding too?

that isn't the case as to what is happening and reading through the older bug threads, priority was never seemingly designed to do that #11229 (comment)

[ndeloof]

priority (as seen in https://github.com/moby/moby/blob/c14efb8deeb2cf5a716ab874c707f9a69228926c/libnetwork/sandbox.go#L677-L683) used to sort endpoints is not set by Docker Compose as API. So the result is you get sort applied by name (https://docs.docker.com/reference/api/engine/version/v1.45/#tag/Network/operation/NetworkConnect) does not offer such an option - I wonder this is a swarm-only feature ?

priority is only used in Docker Compose to connect to networks in priority order, but this doesn't have any impact on this algorithm.

[robmry]

A > B works and is great but there are scenarios where A < B are required or wanted.

Hi @mrpops2ko - yes, understood ... only having the network name to determine gateway selection isn't good.

But, network name is influencing gateway selection for you now, including for ipvlan?

In your original description, you said "2. that it is based upon the alphabetic naming of the networks, so A will begin before B etc (this seems to be true for customer docker networks, but it doesn't work similarly when combined with ipvlan networks)".

As far as I can see, the rule should apply equally to ipvlan and bridge networks - and I wasn't able to repro the problem, for me the name of the network always determined the gateway selection, all other things being equal. I'm reluctant to make any changes without understanding this, it'd be easy to break something that's relied on by others.

can you explain more about this priority field in docker compose, with a use case scenario? i'm at a loss as to what it actually does if it isn't used to do the thing that a user at first glance would assume it is doing

As @ndeloof says, it's just the order in which compose connects networks.

For the engine - I think it'll affect interface naming in the containers, they're numbered in order (eth0, eth1 etc). I can't think of anything else at the moment, but could be forgetting something.

some parts from reading your comment seem contradictory endpoints are sorted by priority yet i've provided priority and the default gateway doesn't change

Right. As I said, "The epPriority used in the engine's ordering of endpoints is unrelated and, weirdly, there's no way to configure it via the engine's API. (So, it's completely useless.)" ... it doesn't work, don't try to make more sense of it than that!

maybe thats the delineation? i'm assuming that the first endpoint with the highest priority would result in becoming the eth0 default gateway for internet connectivity, would that be your assumption and understanding too?

I tried to describe my understanding in my last comment, and it's not that. But, I agree more control is needed. I don't think we'd want to reuse compose's priority field to set the gateway priority in future ... there might be a reason to connect the gateway endpoint last, or to want interface numbering to be separate from gateway selection.

So, there's some missing plumbing, we need a way to be able to specify gateway priority for the engine. But, if the network name isn't currently having the expected effect, I'd like to understand that.

However, this isn't a compose issue. So, I've raised moby/moby#48868 ... let's continue the discussion over there.

(@ndeloof, please could you close this one?)