How to use base image from build cache of docker-container driver

[agirault]

Context

• Using buildx to support multiple platforms
• Hoping to do multi-stages, but need to separate some stages in a different dockerfile (public vs private).

Prerequisites

base.dockerfile

# syntax=docker/dockerfile:1
FROM ubuntu:20.04
ENTRYPOINT [ "/bin/bash", "-c", "echo I am on $(uname -m)" ]

final.dockerfile

# syntax=docker/dockerfile:1
ARG BASE_IMAGE
FROM $BASE_IMAGE as base

Issue

Command

export BUILDER=multiarch
export PLATFORMS=linux/arm64,linux/amd64
export BASE_NAME=base:latest
docker buildx create --name $BUILDER --platform $PLATFORMS
docker buildx build --builder $BUILDER --platform $PLATFORMS -f base.dockerfile -t $BASE_NAME .
docker buildx build --builder $BUILDER --platform $PLATFORMS -f final.dockerfile --build-arg BASE_IMAGE=$BASE_NAME .

Log

multiarch
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 10.3s (9/9) FINISHED                                                                                                      
 => [internal] booting buildkit                                                                                                   3.2s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                2.3s
 => => creating container buildx_buildkit_multiarch0                                                                              0.9s
 => [internal] load build definition from base.dockerfile                                                                         0.0s
 => => transferring dockerfile: 150B                                                                                              0.0s
 => [internal] load .dockerignore                                                                                                 0.0s
 => => transferring context: 2B                                                                                                   0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                        2.8s
 => docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc          1.7s
 => => resolve docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc              0.0s
 => => sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546 9.94MB / 9.94MB                                    1.5s
 => => extracting sha256:1328b32c40fca9bcf9d70d8eccb72eb873d1124d72dadce04db8badbe7b08546                                         0.2s
 => [linux/amd64 internal] load metadata for docker.io/library/ubuntu:20.04                                                       1.9s
 => [linux/arm64 internal] load metadata for docker.io/library/ubuntu:20.04                                                       2.1s
 => [linux/arm64 1/1] FROM docker.io/library/ubuntu:20.04@sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61db  0.0s
 => => resolve docker.io/library/ubuntu:20.04@sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd             0.0s
 => [linux/amd64 1/1] FROM docker.io/library/ubuntu:20.04@sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61db  0.0s
 => => resolve docker.io/library/ubuntu:20.04@sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd             0.0s
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 1.5s (6/6) FINISHED                                                                                                       
 => [internal] load .dockerignore                                                                                                 0.0s
 => => transferring context: 2B                                                                                                   0.0s
 => [internal] load build definition from final.dockerfile                                                                        0.0s
 => => transferring dockerfile: 112B                                                                                              0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                        0.2s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc   0.0s
 => => resolve docker.io/docker/dockerfile:1@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc              0.0s
 => CANCELED [linux/amd64 internal] load metadata for docker.io/library/base:latest                                               1.1s
 => ERROR [linux/arm64 internal] load metadata for docker.io/library/base:latest                                                  1.1s
------
 > [linux/arm64 internal] load metadata for docker.io/library/base:latest:
------
final.dockerfile:3
--------------------
   1 |     # syntax=docker/dockerfile:1
   2 |     ARG BASE_IMAGE
   3 | >>> FROM $BASE_IMAGE as base
   4 |     
--------------------
ERROR: failed to solve: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

If doing the following change (own remote registry, not docker.io)

-  export BASE_NAME=base:latest
+  export BASE_NAME=myremoteregistry.io/$(git rev-parse --short HEAD)

The error...

 => ERROR [linux/arm64 internal] load metadata for docker.io/library/base:latest                                                  1.1s
 ERROR: failed to solve: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

... becomes

 => ERROR [linux/arm64 internal] load metadata for myremoteregistry.io/477a2b26    1.6s
 ERROR: failed to solve: myremoteregistry.io/477a2b26: not found

which makes sense since the base image was not pushed, it's only in the build cache of our docker-container builder.

Question

How to leverage build cache generated from another dockerfile as if this was just a new stage on top of a previous stage? Works fine when not using buildx.

[agirault]

FYI, this stackoverflow thread has recent answers recommending turning off BUILDKIT to be able to use a locally built image, which obviously isn't wanted with buildx for multiarch: https://stackoverflow.com/questions/20481225/how-can-i-use-a-local-image-as-the-base-image-with-a-dockerfile

That thread is the number 1 hit when googling "docker use local image as base", viewed 174k times.