Explore adding a reproducibility test to rust test infrastructure.

[biabbas]

Fixes #75362
Trying to add a reproducibility check on rust infrastructure itself to detect reproducibility issues.
These were the issues encountered till now:

1. On ci only nightly build is supported because of an issue here: https://github.com/rust-lang/rust/blob/master/src/build_helper/src/git.rs#L143
2. Because this is a nightly build, llvm is built from sources. So space was an issue. This was resolved by deleting the copy of source and all build contents except for stage1 artifacts.

Questions:

1. The check takes about 5hours to complete for now. Is this sustainable for each pr?
2. Can we add this as a post merge workflow?
   This is also related to issue: Rust reproducibility issue - Finding the proper fix #134589

Zulip discussion thread:
#t-infra > Add reproducibility check to detect reproducibility issues

[rustbot]

r? @marcoieni

rustbot has assigned @marcoieni.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[biabbas]

@rustbot label -T-bootstrap

[biabbas]

@rustbot label +T-infra

[marcoieni]

2. space was an issue. This was resolved by deleting the copy of source and all build contents except for stage1 artifacts.

please write this as a comment in the code 👍

[marcoieni]

The PR looks good overall, the comments are great, thanks!
I left a few more minor comments.

I want to involve @Kobzol to ask him if he agrees with the overall strategy here. I.e.:

• do we want to add this workflow?
• does it solve the problem in the right way?

[Kobzol]

While it would be great to have some reproducibility checks on CI, I don't think that the current approach used by this PR is the right one, for a few reasons.

• If we do any tests post-merge, it will be quite difficult to act upon them. We should IMO either do this on a best-effort basis out-of-tree (or only in PR CI), or make it actually blocking for merging PRs and run it on auto builds.
• When a reproducibility test fails, it can be quite difficult to understand what went wrong. We should first invest in at least some basic tooling that will display some useful form of a diff between the binary artifacts so that we can actually figure out what went wrong.
• A related aspect to that is reproducibility of the test itself. If the test will be implemented in a separate CI workflow, it will be impossible to run it locally. It would be great to have the option to run the test locally, likely either as a bootstrap test or a run-make test. Both would require some code changes and test infra improvements to make this possible, probably.
• I don't think that (re)building the compiler twice from scratch, even including (sccached) LLVM builds has to take 5 hours. Building the stage 2 compiler on x86_64-gnu, which runs on the free 4-core runner, take ~2000s, so doing that twice should take at most 1.5 hours, if we have proper caching configured. This is also why I'd like to do this in our normal CI workflow, not in a separate workflow that doesn't have any of our supporting infrastructure. I think that we can also start with something simpler, as was proposed in CI for deterministic / reproducible builds #75362, e.g. do the test only for stdlib as a start. Or, as an alternative, maybe we could do a stage 2 build, then move the sources to a different location, do a stage3 build, and just compare stage 2 and stage 3? There are likely several approaches that we can try.

I think that this is something that we should ideally discuss with t-infra first to gather consensus and come up with some plan, before starting with the implementation.

[biabbas]

@Kobzol

If we do any tests post-merge, it will be quite difficult to act upon them. We should IMO either do this on a best-effort basis out-of-tree (or only in PR CI), or make it actually blocking for merging PRs and run it on auto builds.

I'm not fully sure of this. The test seems too slow to be run on each PR. If we do this as a post merge, this would require some effort again to be fixed. Also there are reproducibility issues already with rustdoc, cargo-clippy etc. I suggest that this test be run as post merge test and reproducibility issues to be fixed before releases.

When a reproducibility test fails, it can be quite difficult to understand what went wrong. We should first invest in at least some basic tooling that will display some useful form of a diff between the binary artifacts so that we can actually figure out what went wrong.

I did try adding diffoscope for this, but diffoscope takes up too much time. When I tried adding this test diffoscope always got cancelled, I think this is because the test had already crossed 5 hours.

A related aspect to that is reproducibility of the test itself. If the test will be implemented in a separate CI workflow, it will be impossible to run it locally. It would be great to have the option to run the test locally, likely either as a bootstrap test or a run-make test. Both would require some code changes and test infra improvements to make this possible, probably.

This is also why I'd like to do this in our normal CI workflow, not in a separate workflow that doesn't have any of our supporting infrastructure.

This sounds good. We should provide infrastructure to run this test locally.
Integrating this into test suite, would reduce duplicate steps that don't change the build, for example downloading of components.

Or, as an alternative, maybe we could do a stage 2 build, then move the sources to a different location, do a stage3 build, and just compare stage 2 and stage 3?

This could work.

[Kobzol]

I'm not fully sure of this. The test seems too slow to be run on each PR. If we do this as a post merge, this would require some effort again to be fixed. Also there are reproducibility issues already with rustdoc, cargo-clippy etc. I suggest that this test be run as post merge test and reproducibility issues to be fixed before releases.

As I noted above, I don't expect the test to take more than 1.5 hours on Linux, if we simply do two stage 2 build with proper caching. So form this point of view I think it should be fine to run on every merge.

The binaries distributed would be built by the customer with the sources ( On a similar target configuration) and then compared. So the above suggestion would not suffice as a reproducibility check.

Could you clarify why it wouldn't be enough?

[biabbas]

Or, as an alternative, maybe we could do a stage 2 build, then move the sources to a different location, do a stage3 build, and just compare stage 2 and stage 3?

Could you clarify why it wouldn't be enough?

Sorry, I missed reading the part where you moved the sources to a new location, so now this does make sense.