Revert "Upgrade pre-installed packages in docker images" and Pin docker images by digest

[triarius]

In #2410 I responded to a report that the system packages in our docker images were out of date by upgrading the system packages at build time. However, this has a notable disadvantage - the version in an image is now determined both by the commit of the build and the time that the build ran. So we have lost some reproducibility. So in this PR, we revert that.

However, we can do a bit better than the situation before. Previously, we only pinned the tags of the base image, for example we pinned the ubuntu image to the tag 22.04. Dependabot would only update the image when a newer tag emerged, which is very infrequent in the case of ubuntu. However, maintainers of the base image regularly push newer images to the tag that we have pinned. This mutates the "digest" of the image, and dependabot is capable of detecting updates to the digest. So in this PR, we pin all docker base images by digest.

Notably, this is MORE reproducible than the original situation, as it now two builds at the same commit will always use the same base image.

Reverts #2410

[linear]

Issue reopened: PDP-1799 Upgrade packages installed by default on docker images

[DrJosh9000]

LGTM 🥦

I gather that pinning versions of the packages installed with apk, apt, etc and then having Dependabot update those is not possible, otherwise we could do the same and close another hole in reproducibility.

[DrJosh9000]

Good catch here

[triarius]

LGTM 🥦

I gather that pinning versions of the packages installed with apk, apt, etc and then having Dependabot update those is not possible, otherwise we could do the same and close another hole in reproducibility.

Yeah, the story is a bit murky here. As noted in: dependabot/dependabot-core#2129 (comment), if we pinned the versions of apk/apt packages, the build will break after the package is upgraded in the repo as these repos typically only hold one version of each package. There may be some ways around this with caching docker layers, but that won't work consistently either with the way we currently keep the docker build cache local to each instance the builder agents run on.

In theory, dependabot could detect the update and create a PR that builds successfully. But everyone's branches will be broken until they incorporate the changes in the dependabot PR. And dependabot will never create the new PR immediately, there is always going to be some lag between when packages are updated in the repos and dependabot runs. I think this will be a bad development experience.