Image vulnerability scanning updates have broken the Validate docker build

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

Validate builds have been failing since the docker vulnerability updates.

🕵️ Expected behavior

I expected the build to work

📜 To Reproduce

• Select one of the more recent build failures from https://github.com/NASA-PDS/validate/actions/workflows/unstable-cicd.yaml

🖥 Environment Info

GitHub Action env

📚 Version of Software Used

latest snapshot

⚙️ Engineering Details

NOTE: This is blocking tagging of the repo. We can merge rollback of changes if we cannot find a solution quickly.

🎉 Integration & Test

No response

[nutjob4life]

Whew! Wow, today I learned there's a reason the GitHub Action is called docker/build-push-action and not a separate build-action and push-action.

Going down the rabbit hole, I've learned that there's a difference between a Docker image and a Docker manifest file. When you do

docker image build --tag whatever .

you're building an image and loading it into your local Docker daemon. To do the same thing with docker buildx, you'd do

docker buildx build --load --tag whatever .

But you can't do that with multiplatform images:

$ docker buildx build --load --tag whatever --platform linux/arm64,linux/amd64 .
ERROR: docker exporter does not currently support exporting manifest lists

The difference is that there's really no such a thing as a "multiplatform image"; an image is always for a single platform. When you use --platform with two platforms linux/arm64,linux/amd64, you're creating three things:

1. An image for linux/arm64
2. An image for linux/amd64
3. A manifest list that points to the two images

The local Docker daemon has no idea what to do with "manifest lists" so it rejects it. A container registry does understand manifest lists, so the only thing you can do with a "multiplatform image" is push it:

docker buildx build --push --tag whatever --platform linux/arm64,linux/amd64 .  # Bye --load, hi --push!

Since GitHub Actions inherently supports multiplatform images, it's docker/build-push-action. You're expected to build and push to a container registry, not try any funny business by leaving the "image" in the local Docker daemon.

But we want to leave the "image" in the local Docker daemon so we can use the Grype scanning tool; that's why in the yaml we said:

uses: docker/build-push-action@v6
with:
    load: true
    push: false

I had assumed this would work fine with multiplatform images. Sadly, it does not.

What to do? Well there are several alternatives.

Use containerd

On Docker Desktop, you can use an enhancement to the local Docker daemon called containerd. Just check this box:

It has built-in support for local multiplatform images (which again are just multiple platform-specific images plus a manifest list).

The problem is that in GitHub Actions, containerd support is not yet official. You can make it happen with:

steps:
-
    name: Use containerd
    env:
        DOCKER_CLI_EXPERIMENTAL: enabled
    uses: crazy-max/ghaction-setup-docker@v2
    with:
        version: v24.0.6
        daemon-config: |
              {
                    "features": {
                      "containerd-snapshotter": true
                    }
              }

# rest of the workflow

This sets up Docker Community Edition (such as what Docker Desktop is based on) as the context for the workflow. The problem, though: do you trust the software from someone named @crazy-max? Don't answer that! After all, I'm @nutjob4life

Make a Single Platform Image for Scanning

Rather than build a multiplatform image from the onset, build a single platform image, scan it—and if it passes, then build-and-push a multiplatform image.

The problem with this is that sometimes issues are found specifically with dependencies on a specific platform. For example, an unbounded loop vulnerability might exist only on the arm64 version of libwhatever.so but not in the amd64. If you scan only the amd64 version of the image, you'd miss the problem in final published multiplatform image.

In this case, you can use the matrix feature of the GitHub Actions workflow to do the scans. However, this makes the stable-cicd.yaml and unstable-cicd.yaml files exceedingly complex.

Just Use Dependabot for Now

The work that Grype does with image scanning overlaps with what Dependabot does anyway. And Dependabot does a really nice job. Maybe we should just wait until containerd becomes mainstream.

[jordanpadams]

Status: Educational rabbithole. Couple workaround possible ☝️ . Going to go with @nutjob4life

Verdict: shelve it

[nutjob4life]

@jordanpadams I think we can safely close this, right?

[jordanpadams]

Closing