CI: Publish Docker image to Docker Hub #281

henrybear327 · 2023-11-26T09:55:15Z

Docker images will be built on merge requests.

As discussed, we will employ a multi-stage docker build to reduce build time and image size, but still, making keep the image reproducible when the environment needs to be changed.

jserv · 2023-11-26T10:06:34Z

How much time does it take to build a Docker image from scratch? I am concerned about the considerably longer build time in the CI pipeline, which could lead to potential problems.

henrybear327 · 2023-11-26T10:07:49Z

How much time does it take to build a Docker image from scratch? I am concerned about the considerably longer build time in the CI pipeline, which could lead to potential problems.

This is under testing now. I have no idea how long this will take for the CI on GitHub. Locally, it will be ~20 min IIRC.

jserv · 2023-11-26T10:15:23Z

This is under testing now. I have no idea how long this will take for the CI on GitHub. Locally, it will be ~20 min IIRC.

20 min is too long. Can we concentrate on x86-64 target instead?

henrybear327 · 2023-11-26T10:17:11Z

This is under testing now. I have no idea how long this will take for the CI on GitHub. Locally, it will be ~20 min IIRC.

20 min is too long. Can we concentrate on x86-64 target instead?

I think I would propose the following:

make a separate DockerHub image, called rv32emu-base
unless we change the base environment, we will just pull the base image and build on top of it, which I believe will only be several seconds to complete!

Dockerfile-base

dockerhub/build.sh

dockerhub/README.md

.github/workflows/docker.yml

jserv · 2023-11-28T00:26:31Z

Instead of a directory named "dockerhub," should we put the Docker related files in somewhere like docker directory?

Dockerfile

henrybear327 · 2023-11-29T12:16:04Z

The GitHub secrets aren't set properly. @jserv can you fix it so the CI can work properly :)

I do set the following:

DOCKERHUB_ACCESS_TOKEN

DOCKERHUB_USERNAME

Based on this comment, I think it's a security feature, that a pull request from another repo shouldn't be able to access the secret, which makes sense.

I have tested the Docker Hub login and push feature on my branch, so the PR should work when merged!

docs/base-image.md

jserv · 2023-11-29T15:19:31Z

The build process will take around 20 and 10 minutes when building a native image on the native arch for gcc and sail, respectively, if internet connection is fast

The proposed change is currently too lengthy to be accepted. We should consider building the Docker image only when files in the src directory are modified. Additionally, to optimize the process, we should implement incremental generation. This means that we should update only the rv32emu executable instead of rebuilding the entire gcc and sail components.

henrybear327 · 2023-11-29T15:24:04Z

The build process will take around 20 and 10 minutes when building a native image on the native arch for gcc and sail, respectively, if internet connection is fast

The proposed change is currently too lengthy to be accepted. We should consider building the Docker image only when files in the src directory are modified. Additionally, to optimize the process, we should implement incremental generation. This means that we should update only the rv32emu executable instead of rebuilding the entire gcc and sail components.

Sorry that the commit message is misleading. Please see the updated version.

To sum up, the incremental build is taking around 6 minutes to generate a new rv32emu image and push it to Docker Hub.

jserv · 2023-11-29T15:31:42Z

To sum up, the incremental build is taking around 6 minutes to generate a new rv32emu image and push it to Docker Hub.

A filter should be implemented to monitor modified files. See https://github.com/sysprog21/rv32emu/blob/master/.github/workflows/benchmark.yml

henrybear327 · 2023-11-29T19:06:26Z

To sum up, the incremental build is taking around 6 minutes to generate a new rv32emu image and push it to Docker Hub.

A filter should be implemented to monitor modified files. See https://github.com/sysprog21/rv32emu/blob/master/.github/workflows/benchmark.yml

The rule is set up to only produce a new rv32emu image when there are any changes in the following directories (as there are files that might impact code compilation, benchmarking, arch tests, etc.

src/**
build/**
mk/**
tests/**
tools/**

However, I think it might be better that all PR be merged to the master branch to produce a new image. In this case, any image that we pull from the Docker Hub will be the latest commit on the master branch.

jserv

Modify README.md and mention the instructions to pull built images from Docker Hub.

.github/workflows/codacy.yml

The rv32emu image build process will take around 6 minutes on GitHub CI. The base image build process will take around 20 and 10 minutes when building a native image on the native arch for gcc and sail, respectively, if internet connection is fast. The multi-arch base image build will take a long time (4 hours on a 8-core machine with fast internet) as we will use QEMU emulation for cross compilation. At the time of writing, we can only run multi-arch build successfully on x86 machines. The base image will need to be built using the script provided in the docker directory, manually. The rv32emu image will be built automatically on PR and merge into the master branch.

jserv · 2023-11-30T03:05:56Z

Thank @henrybear327 for contributing!

…mage_on_merge CI: Publish Docker image to Docker Hub

henrybear327 self-assigned this Nov 26, 2023

henrybear327 requested a review from jserv November 26, 2023 09:55

henrybear327 marked this pull request as draft November 26, 2023 10:27

henrybear327 force-pushed the feat/push_docker_image_on_merge branch 2 times, most recently from 164c492 to 511847f Compare November 26, 2023 11:50

jserv reviewed Nov 26, 2023

View reviewed changes

Dockerfile-base Outdated Show resolved Hide resolved

henrybear327 force-pushed the feat/push_docker_image_on_merge branch 5 times, most recently from 9bbbad7 to 812ed6f Compare November 27, 2023 10:13