(Linux) Docker build cannot connect to the internet

[samsapti]

Bug report

Describe your environment

• Device: Lenovo Legion Y530
• OS name and version: Artix Linux (rolling release)
• IVPN app version: CLI v3.8.7 from AUR

Describe the problem

When running docker build ... or docker buildx build ..., Docker can't connect to the internet if fx. the Dockerfile includes RUN commands to clone a git repo or install additional packages. Issue doesn't occur when ivpn-service is disabled on boot, so this is definitely an IVPN issue. Unlike in #19, running ivpn firewall -lan_allow doesn't solve the issue. Even fully disabling the firewall and disconnecting from IVPN doesn't help. Excluding the dockerd subnet from the IVPN firewall doens't help either.

Steps to reproduce:

1. Start ivpn-service
2. Try to build a Docker image from a Dockerfile that requires internet connection
3. Observe that Dockerfile command that requires internet connection stalls

Observed Results:

docker build ... and docker buildx build ... can't connect to the internet.

Expected Results:

docker build ... and docker buildx build ... should be able to connect to the internet.

[samsapti]

Seems to be fixed by #19. Nevermind.

[samsapti]

Weirdly enough, this issue only occurs when building Docker images with Alpine as base image. Specifically, it hangs when trying to fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/aarch64/APKINDEX.tar.gz.

[ka-iy]

@theanonymousexyz I tried this on an arm64 SoC device running Ubuntu 22.04 (Jammy) with IVPN enabled and running, and while at first it looked like I had run into the same problem as you did with the alpine Docker build - that it appeared to hang for a good couple of minutes - it turned out that it just took time.

I started the alpine build, saw it seemingly hung while fetchng the APKINDEX file, waited about two minutes, then left it running while I walked away and heated up some grub, which took about 6-10 minutes.

When I returned I found that the docker build process had successfully fetched the APKINDEX file and that the docker build ran to completion.

So I don't think that this is a problem with IVPN, especially because you said:

Even fully disabling the firewall and disconnecting from IVPN doesn't help.

Since I don't know how long you waited and what your Dockerfile looks like, I have a couple of suggestions for you to run this with IVPN enabled:

1. Make sure you are running your test with alpine:latest - see this Alpine bug report.
2. Make sure you are not running Docker-in-Docker or something funky like that. See this bug report.
3. Make sure that nslookup dl-cdn.alpinelinux.org works and that you can manually wget the package (see my test steps below).
4. Wait about 10 mins or so to see whether it will eventually download. If it doesn't, Ctrl+C out of that first build and try again.

My test steps:

1. Enable multi-hop IVPN (IVPN built from source on the arm64 SoC machine) and make sure that IVPN is enabled and running.
2. Ensure that nslookup dl-cdn.alpinelinux.org works when IVPN is running and enabled (all steps done with IVPN running):

   nslookup dl-cdn.alpinelinux.org
   Server:		172.16.0.1
   Address:	172.16.0.1#53
   
   Non-authoritative answer:
   dl-cdn.alpinelinux.org	canonical name = dualstack.d.sni.global.fastly.net.
   Name:	dualstack.d.sni.global.fastly.net
   Address: 151.101.114.133
   Name:	dualstack.d.sni.global.fastly.net
   Address: 2a04:4e42:1b::645
   

3. Ensure that wget https://dl-cdn.alpinelinux.org/alpine/v3.15/main/aarch64/APKINDEX.tar.gz works
   • The wget was almost instantaneous.
4. Create a very simple alpine Dockerfile:

   # cat Dockerfile.alpine 
   FROM alpine:latest
   RUN apk update
   RUN apk add vim
   

5. Run the docker build (and walk away for a while):

   # docker build . -f Dockerfile.alpine 
   Sending build context to Docker daemon  6.144kB
   Step 1/3 : FROM alpine:latest
   latest: Pulling from library/alpine
   b3c136eddcbf: Pull complete 
   Digest: sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c
   Status: Downloaded newer image for alpine:latest
    ---> 6e30ab57aeee
   Step 2/3 : RUN apk update
    ---> Running in 37534b1e0b8f
   fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/aarch64/APKINDEX.tar.gz
   fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/aarch64/APKINDEX.tar.gz
   v3.16.0-29-gcd435d424c [https://dl-cdn.alpinelinux.org/alpine/v3.16/main]
   v3.16.0-35-gf9dbce14e4 [https://dl-cdn.alpinelinux.org/alpine/v3.16/community]
   OK: 16871 distinct packages available
   Removing intermediate container 37534b1e0b8f
    ---> 5013fe4abbbd
   Step 3/3 : RUN apk add vim
    ---> Running in ebffa2cd8c2c
   (1/5) Installing xxd (8.2.5000-r0)
   (2/5) Installing lua5.4-libs (5.4.4-r5)
   (3/5) Installing ncurses-terminfo-base (6.3_p20220521-r0)
   (4/5) Installing ncurses-libs (6.3_p20220521-r0)
   (5/5) Installing vim (8.2.5000-r0)
   Executing busybox-1.35.0-r13.trigger
   OK: 35 MiB in 19 packages
   Removing intermediate container ebffa2cd8c2c
    ---> 68712d7e4254
   Successfully built 68712d7e4254
   

So if your nslookup and wget work when the VPN is connected, the build should eventually succeed. If not, I suspect the problem may be with your environment. Try building IVPN from from source.

Funnily enough, the second time the docker build is run, fetching that APKINDEX file is almost instantaneous; I assume that IVPN caches it somewhere. I know it's not the docker cache because I ran docker system prune -a before re-running the alpine build just to be sure.

Bonus: It is interesting to see the results which come up when gurgling the search string docker alpine slow 😁

[samsapti]

@Kartikeya-IYER Thanks for the tip. Unfortunately, my builds just fail when building with IVPN on. But when I boot the system with IVPN disabled, it works fine.

[ka-iy]

Hmm... what happens if, instead of rebooting, you just stop the service using systemctl stop ivpn-service (or service ivpn-service stop if you don't have systemd)? Does it still hang?

[samsapti]

Hmm... what happens if, instead of rebooting, you just stop the service using systemctl stop ivpn-service (or service ivpn-service stop if you don't have systemd)? Does it still hang?

The problem persists. I think it has to do with the system firewall not being properly configured (nftables), which is why it works when I boot without IVPN ever running.

[stenya]

Hi,

The issue occurs only when VPN is connected (no issues when VPN and Firewall are disabled).

After some analysis, I came to the conclusion that it is not really related to the IVPN app itself.
It looks like the target server (which provides the Alpine Docker image) is blocking some IPs belonging to the IVPN servers.

As a workaround, please, try to use VPN servers from another location.
For example, I was able to build an Alpine-based docker image while I was connected to a server in Dallas (USA).

I am closing the ticket.
Feel free to open it again if necessary.