Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[ci] increase build reproducibility #778

Closed
Shastick opened this issue Sep 10, 2024 · 2 comments
Closed

[ci] increase build reproducibility #778

Shastick opened this issue Sep 10, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request P2 Normal priority

Comments

@Shastick
Copy link
Contributor

Shastick commented Sep 10, 2024

Is your feature request related to a problem? Please describe.

Currently, some of the dependencies necessary to build and run the project are not pinned:

This means that there is a slight chance that older versions of the master branch may not be easily buildable (ie, check-out a commit, run make image and be done)

We would like to increase the general reproducibility of the monitoring images, with reproducibility defined as it is possible to check-out an older commit, run make image and obtain a usable image, for reasonable definitions of an older commit.

Describe the solution you'd like

In addition to pinning Python dependencies, we can start pinning the exact version of the apt packages that we install, as well as pinning the base images we use.

Describe alternatives you've considered

None: the alternative is to continue as we do today.

Additional context

Ideally we pick an option that allows us to easily automate the update dependencies with a tool such as dependabot.

@Shastick Shastick added enhancement New feature or request P2 Normal priority labels Sep 10, 2024
@Shastick Shastick self-assigned this Sep 10, 2024
@Shastick
Copy link
Contributor Author

Pinning apt-package dependencies may not be the silver-bullet we hope for. From a (still open) dependabot issue requesting to add automated updates for apt-packages in Dockerfiles:

This sounds great in theory, but for the vast majority of use cases it's probably a false hope. Why? Debian repos only maintain the latest version of a given package. Unless you are hosting your own package repo, you aren't going to be able to install arbitrary package versions. So the idea of committing a "dependencies.json" file to version control is essentially impossible, at least in the context of building Docker images.

The only exceptions I see are if you host your own package repo or rely on very careful Docker caching to retain an old "pinned" version of a package.

For the record: the python-3.12.X-slim images we currently depend on are debian-based.

Possible alternative

An option to allow users to easily rebuild older revisions of the repository may involve publishing our own base images. It could work like this:

  1. At regular intervals, release a base image in which all required apt-get install commands have been run.
  2. Always update master so it depends on the latest released base image
  3. Keep base images as we keep any other released artifacts
  4. Users needing to rebuild an older version of the repository will depend on an existing base-image with the required dependencies rather than needing to reinstall them.

Using multi-stage docker builds we can likely come up with something that is reasonably easy to maintain and use, but we should probably only do so if there is a real need.

@BenjaminPelletier & @barroco I won't explore this further until we've clearly determined a need for it. We can discuss it at an upcoming community meeting.

@Shastick
Copy link
Contributor Author

As discussed during the last community meeting, pinning the base image to a minor version, as well as the setuptools version installed with pip outside of the requirements is enough for now.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request P2 Normal priority
Projects
None yet
Development

No branches or pull requests

1 participant