Dist build manifest #77762

pietroalbini · 2020-10-09T18:20:21Z

This PR makes two changes that should remove a significant chunk of the time spent in our release process: cloning the rust-lang/rust monorepo, all its submodules, and building bootstrap to then invoke build-manifest:

build-manifest doesn't rely on a clone of the monorepo being present anymore. The only remaining bit of information it fetched from it (the Rust version) is instead bundled in the binary.
A new "component" is added, build-manifest. That component includes a prebuilt version of the tool, and it's not included in the Rustup manifest. This will allow promote-release to directly invoke the tool without interacting with our build system.
The Linux x86_64 CI is changed to also build the component mentioned above. It's the only CI builder tasked to do so, and to cleanly support this a new --include-default-paths flag was added to ./x.py.
The BUILD_MANIFEST_NUM_THREADS environment variable is added to configure the number of threads at runtime.

This PR is best reviewed commit-by-commit.
r? @Mark-Simulacrum

Mark-Simulacrum · 2020-10-09T19:35:31Z

Can't we just include the rust version in this component? (Or all components, whatever)

pietroalbini · 2020-10-12T09:05:24Z

We need the version to extract the component's tarball anyway, so promote-release will have to fetch it regardless. I guess we could const VERSION: &str = include!("src/version");, but I'm not sure that's worth it.

pietroalbini · 2020-10-12T15:49:21Z

This also added the BUILD_MANIFEST_NUM_THREADS environment variable to be able to configure the number of threads at runtime. r? @Mark-Simulacrum

src/tools/build-manifest/src/main.rs

Mark-Simulacrum · 2020-10-12T16:22:16Z

I think the const VERSION is what I'd personally like to see FWIW, it seems like we're intending promote-release to be tightly coupled anyway.

Mark-Simulacrum · 2020-10-12T16:22:42Z

r=me though on the PR as-is, we can worry about how we get version separately if you want

pietroalbini · 2020-10-12T17:42:03Z

@Mark-Simulacrum pushed two commits to do the changes you requested.

Mark-Simulacrum · 2020-10-12T17:44:18Z

I think you've rebased since "Note: this PR builds on top of #77336. The first commit is already approved in that PR, and it will be rebased away as soon as that PR is merged." -- can you remove that and more broadly update description as necessary for the version changes? r=me with that done.

This commit changes the way build-manifest is invoked, to let it accept the Rust version directly instead of requiring the path of the Rust monorepo and letting build-manifest figure out the path on its own. This allows to run build-manifest without a clone of the monorepo.

This will prevent the tool mistakenly ignoring the variables if they happen to contain non-utf8 data.

pietroalbini · 2020-10-12T17:55:25Z

@bors r=Mark-Simulacrum

bors · 2020-10-12T17:55:26Z

📌 Commit 0b7ee9d has been approved by Mark-Simulacrum

@Mark-Simulacrum

…=Mark-Simulacrum Dist build manifest This PR makes two changes that should remove a significant chunk of the time spent in our release process: cloning the `rust-lang/rust` monorepo, all its submodules, and building `bootstrap` to then invoke `build-manifest`: * `build-manifest` doesn't rely on a clone of the monorepo being present anymore. The only remaining bit of information it fetched from it (the Rust version) is instead bundled in the binary. * A new "component" is added, `build-manifest`. That component includes a prebuilt version of the tool, and it's *not* included in the Rustup manifest. This will allow `promote-release` to directly invoke the tool without interacting with our build system. * The Linux x86_64 CI is changed to also build the component mentioned above. It's the only CI builder tasked to do so, and to cleanly support this a new `--include-default-paths` flag was added to `./x.py`. * The `BUILD_MANIFEST_NUM_THREADS` environment variable is added to configure the number of threads at runtime. This PR is best reviewed commit-by-commit. r? @Mark-Simulacrum

pietroalbini · 2020-10-13T10:02:25Z

@bors try

Getting a CI build to allow me to continue developing locally.

bors · 2020-10-13T10:02:27Z

🙅 Please do not try after a pull request has been r+ed. If you need to try, unapprove (r-) it first.

pietroalbini · 2020-10-13T10:03:34Z

@bors r-

pietroalbini · 2020-10-13T10:03:38Z

@bors try

bors · 2020-10-13T10:03:50Z

⌛ Trying commit 0b7ee9d with merge acb9961a544903a7530fcd72e71a3e50518a230f...

bors · 2020-10-13T11:54:18Z

☀️ Try build successful - checks-actions, checks-azure
Build commit: acb9961a544903a7530fcd72e71a3e50518a230f (acb9961a544903a7530fcd72e71a3e50518a230f)

pietroalbini · 2020-10-13T11:59:17Z

@bors r=Mark-Simulacrum

bors · 2020-10-13T11:59:19Z

📌 Commit 0b7ee9d has been approved by Mark-Simulacrum

bors · 2020-10-13T12:11:54Z

⌛ Testing commit 0b7ee9d with merge d772879...

bors · 2020-10-13T14:11:59Z

☀️ Test successful - checks-actions, checks-azure
Approved by: Mark-Simulacrum
Pushing d772879 to master...

[beta] Rollup backports Cherry-picked: * Always use the Rust version in package names rust-lang#77336 * rustc_parse: More precise spans for `tuple.0.0` rust-lang#77774 * Update crossbeam-channel to avoid UB rust-lang#77819 * build-manifest: stop generating numbered channel names except for stable rust-lang#77854 * Dist build manifest rust-lang#77762 * bootstrap: set correct path for the build-manifest binary rust-lang#77909 r? `@ghost`

rust-highfive assigned Mark-Simulacrum Oct 9, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 9, 2020

Mark-Simulacrum reviewed Oct 12, 2020

View reviewed changes

src/tools/build-manifest/src/main.rs Outdated Show resolved Hide resolved

pietroalbini added 7 commits October 12, 2020 19:53

bootstrap: add disabled by default build-manifest dist component

2f387e9

bootstrap: add --include-default-paths to ./x.py

60ae018

ci: also build the build-manifest component on dist-x86_64-linux

24d04cc

build-manifest: allow configuring the number of threads

f3d07b3

build-manifest: use var_os instead of var to check if vars exist

cbded3e

This will prevent the tool mistakenly ignoring the variables if they happen to contain non-utf8 data.

build-manifest: bundle the rustc version in the binary

0b7ee9d

pietroalbini force-pushed the dist-build-manifest branch from 498bda3 to 0b7ee9d Compare October 12, 2020 17:54

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 12, 2020

Dylan-DPC-zz mentioned this pull request Oct 13, 2020

Rollup of 9 pull requests #77881

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 13, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 13, 2020

bors added the merged-by-bors This PR was explicitly merged by bors. label Oct 13, 2020

bors merged commit d772879 into rust-lang:master Oct 13, 2020

rustbot added this to the 1.49.0 milestone Oct 13, 2020

pietroalbini deleted the dist-build-manifest branch October 14, 2020 09:05

pietroalbini added beta-nominated Nominated for backporting to the compiler in the beta channel. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. beta-accepted Accepted for backporting to the compiler in the beta channel. labels Oct 14, 2020

pietroalbini mentioned this pull request Oct 20, 2020

[beta] Rollup backports #78141

Merged

pietroalbini modified the milestones: 1.49.0, 1.48.0 Oct 20, 2020

pietroalbini removed the beta-nominated Nominated for backporting to the compiler in the beta channel. label Oct 20, 2020

Dist build manifest #77762

Dist build manifest #77762

Uh oh!

Conversation

pietroalbini commented Oct 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mark-Simulacrum commented Oct 9, 2020

Uh oh!

pietroalbini commented Oct 12, 2020

Uh oh!

pietroalbini commented Oct 12, 2020

Uh oh!

Uh oh!

Mark-Simulacrum commented Oct 12, 2020

Uh oh!

Mark-Simulacrum commented Oct 12, 2020

Uh oh!

pietroalbini commented Oct 12, 2020

Uh oh!

Mark-Simulacrum commented Oct 12, 2020

Uh oh!

pietroalbini commented Oct 12, 2020

Uh oh!

bors commented Oct 12, 2020

Uh oh!

pietroalbini commented Oct 13, 2020

Uh oh!

bors commented Oct 13, 2020

Uh oh!

pietroalbini commented Oct 13, 2020

Uh oh!

pietroalbini commented Oct 13, 2020

Uh oh!

bors commented Oct 13, 2020

Uh oh!

bors commented Oct 13, 2020

Uh oh!

pietroalbini commented Oct 13, 2020

Uh oh!

bors commented Oct 13, 2020

Uh oh!

bors commented Oct 13, 2020

Uh oh!

bors commented Oct 13, 2020

Uh oh!

Uh oh!

pietroalbini commented Oct 9, 2020 •

edited

Loading