ipv6nat stop working after docker update #33

celogeek · 2019-05-25T21:25:00Z

Hi,

I have got this error since my docker upgrade:

ipv6nat_1              | Try `iptables -h' or 'iptables --help' for more information.
ipv6nat_1              | 2019/05/25 21:20:31 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 2: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory

My docker version:

docker version
Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:36:01 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       481bc77
  Built:            Sat May  4 01:59:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false

My nat table:

sudo iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE *** (all my rules here)

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere
DNAT *** (all my ipv4 dnat here)
# Warning: iptables-legacy tables present, use iptables-legacy to see them

I have disabled ipv6 for now. Can I do something to fix this?

thanks

The text was updated successfully, but these errors were encountered:

robbertkl · 2019-05-26T10:31:59Z

I'm not sure where those errors come from, ipv6nat runs ip6tables, not iptables. It seems something else is going on here.

I can take a look, but please give me your run command (or docker-compose section) for ipv6nat and output of both iptables-save and ip6tables-save.

celogeek · 2019-05-26T11:19:28Z

iptables-save:

# Generated by xtables-save v1.8.2 on Sun May 26 11:16:23 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-5c0b4b729dfe -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-5c0b4b729dfe -j DOCKER
-A FORWARD -i br-5c0b4b729dfe ! -o br-5c0b4b729dfe -j ACCEPT
-A FORWARD -i br-5c0b4b729dfe -o br-5c0b4b729dfe -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 10.20.0.4/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 10.20.0.5/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.6/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 10.20.0.6/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 465 -j ACCEPT
-A DOCKER -d 10.20.0.6/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 10.20.0.8/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.10/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 51415 -j ACCEPT
-A DOCKER -d 10.20.0.12/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.20/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 10.20.0.20/32 ! -i br-5c0b4b729dfe -o br-5c0b4b729dfe -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-5c0b4b729dfe ! -o br-5c0b4b729dfe -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-5c0b4b729dfe -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Sun May 26 11:16:23 2019
# Generated by xtables-save v1.8.2 on Sun May 26 11:16:23 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.20.0.0/24 ! -o br-5c0b4b729dfe -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.20.0.4/32 -d 10.20.0.4/32 -p tcp -m tcp --dport 993 -j MASQUERADE
-A POSTROUTING -s 10.20.0.5/32 -d 10.20.0.5/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.6/32 -d 10.20.0.6/32 -p tcp -m tcp --dport 587 -j MASQUERADE
-A POSTROUTING -s 10.20.0.6/32 -d 10.20.0.6/32 -p tcp -m tcp --dport 465 -j MASQUERADE
-A POSTROUTING -s 10.20.0.6/32 -d 10.20.0.6/32 -p tcp -m tcp --dport 25 -j MASQUERADE
-A POSTROUTING -s 10.20.0.8/32 -d 10.20.0.8/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.10/32 -d 10.20.0.10/32 -p tcp -m tcp --dport 51415 -j MASQUERADE
-A POSTROUTING -s 10.20.0.12/32 -d 10.20.0.12/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.20/32 -d 10.20.0.20/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 10.20.0.20/32 -d 10.20.0.20/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i br-5c0b4b729dfe -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 993 -j DNAT --to-destination 10.20.0.4:993
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 10025 -j DNAT --to-destination 10.20.0.5:22
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 587 -j DNAT --to-destination 10.20.0.6:587
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 465 -j DNAT --to-destination 10.20.0.6:465
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 25 -j DNAT --to-destination 10.20.0.6:25
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 10023 -j DNAT --to-destination 10.20.0.8:22
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 51415 -j DNAT --to-destination 10.20.0.10:51415
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 10024 -j DNAT --to-destination 10.20.0.12:22
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.20.0.20:443
-A DOCKER ! -i br-5c0b4b729dfe -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.20.0.20:80
COMMIT
# Completed on Sun May 26 11:16:23 2019

ip6tables-save:

# Generated by xtables-save v1.8.2 on Sun May 26 11:16:52 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sun May 26 11:16:52 2019
# Generated by xtables-save v1.8.2 on Sun May 26 11:16:52 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sun May 26 11:16:52 2019

docker-compose.yml:

networks:
  default:
    driver: bridge
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: 10.20.0.0/24
        - subnet: fd00:dead:beef::/48

services:
  ipv6nat:
    image: robbertkl/ipv6nat
    command: ["-retry", "-cleanup"]
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /lib/modules:/lib/modules:ro
    network_mode: host
    cap_add:
      - NET_ADMIN
      - SYS_MODULE

robbertkl · 2019-05-26T11:22:48Z

And this output is with ipv6nat running? Or did it exit at this time? I'm asking because you're using --cleanup which removes all the rules on exit.

Can you do a docker-compose up -d --force-recreate ipv6nat and after that send me the docker logs?

celogeek · 2019-05-26T11:25:38Z

I have removed the command section and run the recreate:

ipv6nat_1              | 2019/05/26 11:25:07 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 2: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory
ipv6nat_1              |
ipv6nat_1              | Try `iptables -h' or 'iptables --help' for more information.

celogeek · 2019-05-26T11:31:49Z

The ipv6nat wasn't running because it fails at the start with that log.

robbertkl · 2019-05-26T11:37:47Z

OK, it craps out trying to detect hairpin mode in the IPv4 firewall. Could you try:

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --cap-add=NET_ADMIN --cap-add=SYS_MODULE --entrypoint iptables-save robbertkl/ipv6nat

celogeek · 2019-05-26T12:15:14Z

sudo docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --cap-add=NET_ADMIN --cap-add=SYS_MODULE --entrypoint iptables-save robbertkl/ipv6nat
# Generated by iptables-save v1.6.2 on Sun May 26 12:14:56 2019
*nat
:PREROUTING ACCEPT [8392:426047]
:INPUT ACCEPT [2635:96038]
:OUTPUT ACCEPT [3987:318688]
:POSTROUTING ACCEPT [9744:648697]
COMMIT
# Completed on Sun May 26 12:14:56 2019

robbertkl · 2019-05-26T15:31:03Z

Can you try the same but with --privileged instead?

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --privileged --entrypoint iptables-save robbertkl/ipv6nat

celogeek · 2019-05-26T15:50:45Z

The output is empty.
When I run docker-ipv6nat again:

/ # ./docker-ipv6nat
2019/05/26 15:50:19 running [/sbin/iptables -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER --wait]: exit status 2: iptables v1.6.2: Couldn't load target `DOCKER':No such file or directory

robbertkl · 2019-05-26T16:18:56Z

So when you're running iptables-save from within a privileged, net=host ipv6nat container it shows no IPv4 rules, but when you run iptables-save on the host it does show IPv4 rules?

I've just tried the same on a test host running Docker 19.03.0-beta3, and I still see the IPv4 rules from within the container, so no way to reproduce it on my end.

celogeek · 2019-05-26T16:20:59Z

Sorry I run the ip6tables command.
Here the output:

# Generated by iptables-save v1.6.2 on Sun May 26 16:20:42 2019
*nat
:PREROUTING ACCEPT [3580:160229]
:INPUT ACCEPT [562:19464]
:OUTPUT ACCEPT [792:64734]
:POSTROUTING ACCEPT [1927:130179]
COMMIT
# Completed on Sun May 26 16:20:42 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:20:42 2019
*filter
:INPUT ACCEPT [3180:338688]
:FORWARD ACCEPT [50554:39971344]
:OUTPUT ACCEPT [3327:460405]
COMMIT
# Completed on Sun May 26 16:20:42 2019

robbertkl · 2019-05-26T16:25:07Z

Sorry, I'm a bit confused, this output is different than before again. Can you please post all 3 outputs:

iptables-save (on the host)
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --cap-add=NET_ADMIN --cap-add=SYS_MODULE --entrypoint iptables-save robbertkl/ipv6nat
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock:ro -v /lib/modules:/lib/modules:ro --net=host --privileged --entrypoint iptables-save robbertkl/ipv6nat

All commands use iptables, not ip6tables. The problem is your ipv6nat instance seems not to see the IPv4 rules, even though they're available on the host. I cannot reproduce this myself, however.

celogeek · 2019-05-26T16:30:08Z

Despite my provider give me only a /128 block, it seems I can have a /64 block anyway.
They are not limiting the ipv6 assignation on my /64 block.

I have reconfigured my docker to handle ipv6 block /80.
All my containers have a public IP address now, and it seems to connect to the host with ipv6 on a container port (SMTP for example) redirect properly but in a weird way.
HOST ipv6 -> Host docker ipv4 -> Container IPV4

Also, an issue is that they are no ipv6 isolation of the container. If I have a port listing (80) on a container but I don't open it on the host (don't have the "ports"), I can't reach it with ipv4, but I can with ipv6.
Also, ports redirect doesn't work well: ports: 2222:22, give ipv4: 2222, ipv6: 22

The ipv6 nat is far easier to configure.

Here the output I have (sorry for the change):
1.

# Generated by xtables-save v1.8.2 on Sun May 26 16:29:11 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-a040ef404950 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a040ef404950 -j DOCKER
-A FORWARD -i br-a040ef404950 ! -o br-a040ef404950 -j ACCEPT
-A FORWARD -i br-a040ef404950 -o br-a040ef404950 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 10.20.0.27/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 51415 -j ACCEPT
-A DOCKER -d 10.20.0.28/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.15/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.19/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 10.20.0.19/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 465 -j ACCEPT
-A DOCKER -d 10.20.0.19/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 10.20.0.21/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 22 -j ACCEPT
-A DOCKER -d 10.20.0.18/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 10.20.0.10/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 10.20.0.10/32 ! -i br-a040ef404950 -o br-a040ef404950 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-a040ef404950 ! -o br-a040ef404950 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-a040ef404950 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Sun May 26 16:29:11 2019
# Generated by xtables-save v1.8.2 on Sun May 26 16:29:11 2019
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.20.0.0/24 ! -o br-a040ef404950 -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.20.0.27/32 -d 10.20.0.27/32 -p tcp -m tcp --dport 51415 -j MASQUERADE
-A POSTROUTING -s 10.20.0.28/32 -d 10.20.0.28/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.15/32 -d 10.20.0.15/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.19/32 -d 10.20.0.19/32 -p tcp -m tcp --dport 587 -j MASQUERADE
-A POSTROUTING -s 10.20.0.19/32 -d 10.20.0.19/32 -p tcp -m tcp --dport 465 -j MASQUERADE
-A POSTROUTING -s 10.20.0.19/32 -d 10.20.0.19/32 -p tcp -m tcp --dport 25 -j MASQUERADE
-A POSTROUTING -s 10.20.0.21/32 -d 10.20.0.21/32 -p tcp -m tcp --dport 22 -j MASQUERADE
-A POSTROUTING -s 10.20.0.18/32 -d 10.20.0.18/32 -p tcp -m tcp --dport 993 -j MASQUERADE
-A POSTROUTING -s 10.20.0.10/32 -d 10.20.0.10/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 10.20.0.10/32 -d 10.20.0.10/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i br-a040ef404950 -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 51415 -j DNAT --to-destination 10.20.0.27:51415
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 10025 -j DNAT --to-destination 10.20.0.28:22
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 10023 -j DNAT --to-destination 10.20.0.15:22
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 587 -j DNAT --to-destination 10.20.0.19:587
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 465 -j DNAT --to-destination 10.20.0.19:465
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 25 -j DNAT --to-destination 10.20.0.19:25
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 10024 -j DNAT --to-destination 10.20.0.21:22
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 993 -j DNAT --to-destination 10.20.0.18:993
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.20.0.10:443
-A DOCKER ! -i br-a040ef404950 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.20.0.10:80
COMMIT
# Completed on Sun May 26 16:29:11 2019

# Generated by iptables-save v1.6.2 on Sun May 26 16:29:40 2019
*nat
:PREROUTING ACCEPT [4531:202007]
:INPUT ACCEPT [731:25374]
:OUTPUT ACCEPT [990:81507]
:POSTROUTING ACCEPT [2369:161300]
COMMIT
# Completed on Sun May 26 16:29:40 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:29:40 2019
*filter
:INPUT ACCEPT [3977:421198]
:FORWARD ACCEPT [85777:73490326]
:OUTPUT ACCEPT [4113:547670]
COMMIT
# Completed on Sun May 26 16:29:40 2019

# Generated by iptables-save v1.6.2 on Sun May 26 16:29:56 2019
*nat
:PREROUTING ACCEPT [4554:202959]
:INPUT ACCEPT [735:25502]
:OUTPUT ACCEPT [992:81688]
:POSTROUTING ACCEPT [2374:161665]
COMMIT
# Completed on Sun May 26 16:29:56 2019
# Generated by iptables-save v1.6.2 on Sun May 26 16:29:56 2019
*filter
:INPUT ACCEPT [3995:422953]
:FORWARD ACCEPT [86786:74429505]
:OUTPUT ACCEPT [4131:549825]
COMMIT
# Completed on Sun May 26 16:29:56 2019

celogeek · 2019-05-26T18:16:36Z

well, ipv6 sucks with docker. I will disable ipv6 on my host. It is not really useful.
If that evolves, I may reenable it.
Or I may look out for an alternative.

robbertkl · 2019-05-26T18:33:06Z

Yeah, it does suck big time. However, I've been running it with ipv6nat for years without any issues. Not sure what your issue is; your container not being able to reach the (IPv4) firewall rules seems to be the cause of ipv6nat crashing, but why this is happening I don't know. Are you by any chance using SELinux? I can't reproduce it myself, so it's hard to debug.

celogeek · 2019-05-26T19:14:10Z

I'm on debian/buster, the latest kernel on this system.
I also have the latest (non-beta) version of docker/docker-compose.
It seems to hide iptables rules indeed inside the container which cause the issue here.

robbertkl · 2019-05-26T19:24:19Z

Yeah, I'm curious to know what's causing it. Even on 19.03.0-beta3 I could not reproduce it, so it must not be a Docker thing. Perhaps it's something kernel-related.

celogeek · 2019-05-26T19:24:47Z

Can you test it with debian/buster ?

celogeek · 2019-05-26T20:23:32Z

Maybe it would be possible and safer to run your companion on the host directly? Something like a service that starts with docker?

celogeek · 2019-05-26T21:12:39Z

The kernel:

Linux ns363135 4.19.0-5-amd64 #1 SMP Debian 4.19.37-3 (2019-05-15) x86_64 GNU/Linux

robbertkl · 2019-05-26T21:14:50Z

Running on the host is a great idea, that should work since it should be able to access the firewall there.

I'll try to do some tests with buster / later kernels soon.

robbertkl · 2019-05-26T22:31:40Z

OK @celogeek, just did 2 clean Debian VM installs (1 stretch @ 4.9.0-9, 1 buster @ 4.19.0-5) and installed Docker stable (18.09.6).

Then tried reading iptables from within a (privileged, net=host) container on both:

on stretch it worked
on buster it did not (no rules)

It seems the problem is buster using nftables (see https://wiki.debian.org/nftables). I managed to get it working by switching to legacy version by running:

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
update-alternatives --set arptables /usr/sbin/arptables-legacy
update-alternatives --set ebtables /usr/sbin/ebtables-legacy
reboot

After the reboot, I can see the rules from inside the container again.

celogeek · 2019-05-26T23:37:29Z

Nice catch!

Did you try directly on the host? without a container?

robbertkl · 2019-05-27T00:37:21Z

Yes, you can either switch to legacy iptables like above, or run ipv6nat directly on the host. On the host it will use the iptables/ip6tables commands that are linked to the special nftables compatibility versions.

This same approach would also work from within the container (auto translating iptables to nftables), so I'm working on a fix for the Docker image to support this. I just need to figure out how to reliably detect which is used. After I'm done with this fix, you should be able to run it again as a container, without any changes.

robbertkl · 2019-05-27T16:55:40Z

Hi @celogeek , just released a new version of the docker images with support for nftables, so it should work again with your previous setup, unchanged. Could you test this?

celogeek · 2019-05-27T19:01:55Z

Yes of course.
I will need to reactivate ipv6 on my server.
I do that as soon as possible.

I guess you have try in debian/buster/nfptables condition ?

robbertkl · 2019-05-27T20:50:52Z

Yep, tried it and as far as I could see it starts just fine.

celogeek · 2019-05-28T18:52:15Z

test done
it doesn't work.

here the error I got:

ipv6nat_1              | 2019/05/28 18:50:57 running [/usr/local/bin/ip6tables -t filter -I DOCKER 1 -d fd00:dead:beef::2 ! -i br-f016aef740c8 -o br-f016aef740c8 -p tcp -m tcp --dport 22 -j ACCEPT --wait]: exit status 1: iptables: Invalid argument. Run `dmesg' for more information.

in dmesg:

[  557.022382] x_tables: ip6_tables: tcp match: only valid for protocol 6

Maybe used ip6tables-nft instead?

robbertkl · 2019-05-28T22:24:51Z

Yeah, I'm seeing this same error now. Seems like a bug in the compat scripts, because we're obviously using protocol 6. Doing an ip6tables-translate did not generate an error.

You're right about ip6tables-nft, this is what I wanted to use, however I'm using Alpine Linux as a base image and their iptables is running behind.

robbertkl · 2019-05-28T22:30:01Z

Wait, this is great, there's a Alpine ip6tables 1.8.2-r1 package in edge now, built just YESTERDAY 😄

I'll go check it out.

robbertkl · 2019-05-29T20:08:06Z

I got a bit further by using 1.8.1-r1 and ip6tables-nft, but now I ran into an issue with go-iptables, I reported it here: coreos/go-iptables#59

There's not really a workaround with the current version, so I'll fix the issue to get it working. Should be done soon, I'll let you know. I'll reopen this issue.

robbertkl · 2019-05-30T00:57:37Z

@celogeek Could you please test again with the latest release? Thanks!

celogeek · 2019-05-30T08:55:07Z

yes sure. testing it right now.

celogeek · 2019-05-30T09:55:03Z

It works !!! congrat.

Everything is back to normal now.

I will take a vps with ipv6 connectivity and play with real ipv6 config (no nat).
It seems to be not so easy to do a firewall with docker / ufw / ipv6.

This would be a good challenge ;)

A very great companion for docker would be something like this tools, that disable incoming connection and redirect port to the one inside but with real ipv6 ip addr.

As I said before if I have a real ipv6 on a docker, it is accessible directly without following the rules of dockers (port open/close or redirected).

robbertkl · 2019-05-30T10:22:03Z

Good to hear it's working! I'll close the issue now.

robbertkl closed this as completed May 26, 2019

robbertkl mentioned this issue May 29, 2019

ip6tables error #34

Closed

robbertkl reopened this May 29, 2019

robbertkl closed this as completed May 30, 2019

ipv6nat stop working after docker update #33

ipv6nat stop working after docker update #33

Comments

celogeek commented May 25, 2019

robbertkl commented May 26, 2019

celogeek commented May 26, 2019

robbertkl commented May 26, 2019

celogeek commented May 26, 2019

celogeek commented May 26, 2019

robbertkl commented May 26, 2019

celogeek commented May 26, 2019

robbertkl commented May 26, 2019

celogeek commented May 26, 2019

robbertkl commented May 26, 2019

celogeek commented May 26, 2019

robbertkl commented May 26, 2019

celogeek commented May 26, 2019

celogeek commented May 26, 2019

robbertkl commented May 26, 2019

celogeek commented May 26, 2019

robbertkl commented May 26, 2019

celogeek commented May 26, 2019

celogeek commented May 26, 2019

celogeek commented May 26, 2019

robbertkl commented May 26, 2019

robbertkl commented May 26, 2019

celogeek commented May 26, 2019

robbertkl commented May 27, 2019

robbertkl commented May 27, 2019

celogeek commented May 27, 2019

robbertkl commented May 27, 2019

celogeek commented May 28, 2019 • edited Loading

robbertkl commented May 28, 2019

robbertkl commented May 28, 2019

robbertkl commented May 29, 2019

robbertkl commented May 30, 2019

celogeek commented May 30, 2019

celogeek commented May 30, 2019

robbertkl commented May 30, 2019

celogeek commented May 28, 2019 •

edited

Loading