build arm and x86_64 docker containers

[Tweeticoats]

This changes travisci to use docker buildx to build arm and x86_64 containers.

Buildx is a alternate builder for docker that allows you to build many containers at once.
This uses binfmt and qemu to allow linux to run arm programs allowing you to build containers for arm on an x86 machine.

This will build containers for x86_64, arm 8 (arm 64), and arm 7 containers and push this to docker hub.
The user can then pull stash for thair cpu as needed.
This uses ubuntu and includes ffmpeg and python needed for some scrapers and plugins.

[WithoutPants]

The build Dockerfile should have its dependencies updated to be consistent with the other one. I think the proudction and develop ones are no longer used.

[WithoutPants]

Getting the following output when running the buildx process:

docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 --output type=image,name=stashapp/stash:development -f ./docker/ci/x86_64/Dockerfile dist/

[+] Building 544.6s (16/19)                                                    
 => [internal] load build definition from Dockerfile                      0.3s
 => => transferring dockerfile: 32B                                       0.1s
 => [internal] load .dockerignore                                         0.3s
 => => transferring context: 2B                                           0.1s
 => [linux/amd64 internal] load metadata for docker.io/library/ubuntu:20  2.7s
 => [linux/arm/v7 internal] load metadata for docker.io/library/ubuntu:2  2.8s
 => [linux/arm64 internal] load metadata for docker.io/library/ubuntu:20  2.6s
 => [internal] load build context                                         6.8s
 => => transferring context: 196.61MB                                     6.5s
 => CACHED [linux/arm/v7 app 1/3] FROM docker.io/library/ubuntu:20.04@sh  0.0s
 => => resolve docker.io/library/ubuntu:20.04@sha256:703218c0465075f4425  0.0s
 => CACHED [linux/amd64 prep 1/4] FROM docker.io/library/ubuntu:20.04@sh  0.0s
 => => resolve docker.io/library/ubuntu:20.04@sha256:703218c0465075f4425  0.1s
 => CACHED [linux/arm64 app 1/3] FROM docker.io/library/ubuntu:20.04@sha  0.0s
 => => resolve docker.io/library/ubuntu:20.04@sha256:703218c0465075f44  541.3s
 => CANCELED [linux/arm64 app 2/3] RUN apt update && apt install -y py  541.2s
 => ERROR [linux/arm/v7 app 2/3] RUN apt update && apt install -y pyth  541.1s
 => CANCELED [linux/amd64 app 2/3] RUN apt update && apt install -y py  541.2s
 => [linux/amd64 prep 2/4] COPY stash-*  /                               10.1s
 => [linux/amd64 prep 3/4] RUN if [ "linux/arm64" = "linux/arm/v6" ];     3.0s
 => [linux/amd64 prep 3/4] RUN if [ "linux/arm/v7" = "linux/arm/v6" ];    3.0s
 => [linux/amd64 prep 3/4] RUN if [ "linux/amd64" = "linux/arm/v6" ];     3.0s
------
 > [linux/arm/v7 app 2/3] RUN apt update && apt install -y python3 python3 python-is-python3 python3-requests ffmpeg && rm -rf /var/lib/apt/lists/*:          
#7 6.584                                                                       
#7 6.585 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                      
#7 6.586
#7 12.40 Get:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease [265 kB]
<snip>
#7 540.4 debconf: delaying package configuration, since apt-utils is not installed
#7 540.6 Fetched 93.2 MB in 7min 11s (216 kB/s)
#7 540.7 Error while loading /usr/sbin/dpkg-split: No such file or directory
#7 540.7 Error while loading /usr/sbin/dpkg-deb: No such file or directory
#7 540.7 dpkg: error processing archive /var/cache/apt/archives/libssl1.1_1.1.1f-1ubuntu2.1_armhf.deb (--unpack):
#7 540.7  dpkg-deb --control subprocess returned error exit status 1
#7 540.7 Error while loading /usr/sbin/dpkg-split: No such file or directory
#7 540.7 Error while loading /usr/sbin/dpkg-deb: No such file or directory
#7 540.7 dpkg: error processing archive /var/cache/apt/archives/libpython3.8-minimal_3.8.5-1~20.04_armhf.deb (--unpack):
#7 540.7  dpkg-deb --control subprocess returned error exit status 1
#7 540.7 Error while loading /usr/sbin/dpkg-split: No such file or directory
#7 540.7 Error while loading /usr/sbin/dpkg-deb: No such file or directory
#7 540.7 dpkg: error processing archive /var/cache/apt/archives/libexpat1_2.2.9-1build1_armhf.deb (--unpack):
#7 540.7  dpkg-deb --control subprocess returned error exit status 1
#7 540.7 Error while loading /usr/sbin/dpkg-split: No such file or directory
#7 540.7 Error while loading /usr/sbin/dpkg-deb: No such file or directory
#7 540.7 dpkg: error processing archive /var/cache/apt/archives/python3.8-minimal_3.8.5-1~20.04_armhf.deb (--unpack):
#7 540.7  dpkg-deb --control subprocess returned error exit status 1
#7 540.8 Errors were encountered while processing:
#7 540.8  /var/cache/apt/archives/libssl1.1_1.1.1f-1ubuntu2.1_armhf.deb
#7 540.8  /var/cache/apt/archives/libpython3.8-minimal_3.8.5-1~20.04_armhf.deb
#7 540.8  /var/cache/apt/archives/libexpat1_2.2.9-1build1_armhf.deb
#7 540.8  /var/cache/apt/archives/python3.8-minimal_3.8.5-1~20.04_armhf.deb
#7 540.9 E: Sub-process /usr/bin/dpkg returned an error code (1)
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkit_qemu_emulator /bin/sh -c apt update && apt install -y python3 python3 python-is-python3 python3-requests ffmpeg && rm -rf /var/lib/apt/lists/*]: exit code: 100

[Tweeticoats]

How are you running that? just on your computer or through travis?

I assume you have run the following to add binfmt support to docker:
docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64

Can you paste the output of the following?
docker info
docker buildx create --name builder --use
docker buildx inspect --bootstrap

The crash is from qemu crashing trying to emulate arm 7 that I suspect are with moby/buildkit.
buildx uses buildkit, buildkit uses binfmt in the kernel, binfmt pipes things through qemu.
I've encountered similar issues and I worked around this by getting buildx to ssh to a raspberry pi 4 to build arm 7 and 8.

The binfmt recommendation might also need to change the docker container was the one recommended by a blog post from docker but that is out dated. It looks like the recommended container is:
docker run --privileged --rm tonistiigi/binfmt --install all

I have tested travis running xenial and it successfully built containers and pushed this to my docker hub account so that combination of tools should be fine.

[WithoutPants]

I'm running this manually in a debian VM.

You're correct regarding the binfmt support. I've since ran that and now get the following when building:

 => CANCELED [linux/amd64 app 2/3] RUN apt update && apt install -y py  494.8s
 => ERROR [linux/arm/v7 app 2/3] RUN apt update && apt install -y pyth  494.7s
 => CACHED [linux/amd64 prep 2/4] COPY stash-*  /                         0.0s
 => CACHED [linux/amd64 prep 3/4] RUN if [ "linux/arm64" = "linux/arm/v6  0.0s
 => CACHED [linux/amd64 prep 3/4] RUN if [ "linux/amd64" = "linux/arm/v6  0.0s
 => CACHED [linux/amd64 prep 3/4] RUN if [ "linux/arm/v7" = "linux/arm/v  0.0s
 => CANCELED [linux/arm64 app 2/3] RUN apt update && apt install -y py  494.7s
------                  
 > [linux/arm/v7 app 2/3] RUN apt update && apt install -y python3 python3 python-is-python3 python3-requests ffmpeg && rm -rf /var/lib/apt/lists/*:          
#13 0.268  
<snip>
#13 494.7 Fetched 93.2 MB in 6min 42s (232 kB/s)
#13 494.7 E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/s/speex/libspeex1_1.2~rc1.2-1.1ubuntu1_armhf.deb  Undetermined Error [IP: 91.189.88.142 80]
#13 494.7 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
------
failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c apt update && apt install -y python3 python3 python-is-python3 python3-requests ffmpeg && rm -rf /var/lib/apt/lists/*]: exit code: 100

[bnkai]

That was probably a temporary network error / mirror issue while getting the specific package.
I tried to go through all the steps on a local vm also and the buildx build completes ok.
I had to remove the push from the buildx so while the images where build ok i couldn't test them (they should be ok though).
Is there a way to do that also or do i have to push to a local/new docker registry ?

[Tweeticoats]

As you are building multi platform images it needs to push the file system images and metadata so you need to push this to a registry.
This can be the docker registry running locally:

docker run -d -p 5000:5000 --restart always --name registry registry:2
docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 --push --output type=image,name=localhost:5000/stashapp/stash:$DOCKER_TAG,push=true -f docker/ci/x86_64/Dockerfile dist/

If you want to use a self signed certificate it gets a little more complex as you need to put the certificate in /usr/local/share/ca-certificates/ and run update-ca-certificates.

[bnkai]

I tried to finish my testing by pushing to a local registy but that didnt go well
The buildx is run inside a container so localhost wont work , a local address was needed eg 192.168.xxx.xxx and to use http/insecure https you have to create a new buillder instance and set the config for that registry docker/buildx#94 (comment). That throwed some more errors and after 5-6 attempts to push to the registry i gave up.
This doesnt affect the PR though given that every time i tried the build procedure was finished ok without problems , only the pushing to the local insecure registry was the issue. I propose we merge the PR and test the new development tagged containers from the docker hub. The current development-x86_64 container shouldnt be affected so it should be a low risk move.