rustc_codegen_llvm: add support for writing summary bitcode #125345
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
r? @Nadrieril rustbot has assigned @Nadrieril. Use |
rustbot
added
S-waiting-on-review
This comment has been minimized.
This comment has been minimized.
durin42
force-pushed
the
thin-link-bitcode
branch
from
74ad3bd to
c34660e
Compare
|
Some changes occurred in compiler/rustc_codegen_cranelift cc @bjorn3 |
This comment has been minimized.
This comment has been minimized.
|
r? compiler |
|
r? compiler |
durin42
force-pushed
the
thin-link-bitcode
branch
from
c34660e to
01c1ffd
Compare
|
Some changes occurred in compiler/rustc_codegen_gcc |
This comment has been minimized.
This comment has been minimized.
|
Latest CI run failed with |
|
This is enabled with The code in the PR looks ok to me, though LTO experts might have opinions -- @bjorn3? @mw? But I think that adding a new output type is a significant enough change that it should be done via the Major Change Proposal mechanism. |
|
|
Ok, thanks for the extra info. @bjorn3 do you want to give the final approval here? You know what's going on here much better than I do. |
bjorn3
reviewed
May 22, 2024
This comment has been minimized.
This comment has been minimized.
Typical uses of ThinLTO don't have any use for this as a standalone file, but distributed ThinLTO uses this to make the linker phase more efficient. With clang you'd do something like `clang -flto=thin -fthin-link-bitcode=foo.indexing.o -c foo.c` and then get both foo.o (full of bitcode) and foo.indexing.o (just the summary or index part of the bitcode). That's then usable by a two-stage linking process that's more friendly to distributed build systems like bazel, which is why I'm working on this area. I talked some to @teresajohnson about naming in this area, as things seem to be a little confused between various blog posts and build systems. "bitcode index" and "bitcode summary" tend to be a little too ambiguous, and she tends to use "thin link bitcode" and "minimized bitcode" (which matches the descriptions in LLVM). Since the clang option is thin-link-bitcode, I went with that to try and not add a new spelling in the world. Per @dtolnay, you can work around the lack of this by using `lld --thinlto-index-only` to do the indexing on regular .o files of bitcode, but that is a bit wasteful on actions when we already have all the information in rustc and could just write out the matching minimized bitcode. I didn't test that at all in our infrastructure, because by the time I learned that I already had this patch largely written.
This was needed in an older version of this patch, but never got edited out when it became obsolete.
|
I have confirmed that, somehow, the llvm-17 regression here is real and seemingly caused by my change. I'll investigate soon. |
If we don't do this, some versions of LLVM (at least 17, experimentally) will double-emit some error messages, which is how I noticed this. Given that it seems to be costing some extra work, let's only request the summary bitcode production if we'll actually bother writing it down, otherwise skip it.
I did this in the user-facing logic, but I noticed while fixing a minor defect that I had missed it in a few places in the internal details.
durin42
force-pushed
the
thin-link-bitcode
branch
from
ad0ef0c to
3ea4941
Compare
|
Fixed the llvm 17 failure. Sorry for the force-push, that ended up being hard to avoid once I got into the weeds debugging. I also cleaned up a couple of variable names so things are generally consistent in using |
This comment has been minimized.
This comment has been minimized.
|
@bors r+ |
bors
added
S-waiting-on-bors
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
May 23, 2024
…llaumeGomez Rollup of 6 pull requests Successful merges: - rust-lang#125263 (rust-lld: fallback to rustc's sysroot if there's no path to the linker in the target sysroot) - rust-lang#125345 (rustc_codegen_llvm: add support for writing summary bitcode) - rust-lang#125362 (Actually use TAIT instead of emulating it) - rust-lang#125412 (Don't suggest adding the unexpected cfgs to the build-script it-self) - rust-lang#125445 (Migrate `run-make/rustdoc-with-short-out-dir-option` to `rmake.rs`) - rust-lang#125452 (Cleanup check-cfg handling in core and std) r? `@ghost` `@rustbot` modify labels: rollup
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
May 24, 2024
…llaumeGomez Rollup of 6 pull requests Successful merges: - rust-lang#125263 (rust-lld: fallback to rustc's sysroot if there's no path to the linker in the target sysroot) - rust-lang#125345 (rustc_codegen_llvm: add support for writing summary bitcode) - rust-lang#125362 (Actually use TAIT instead of emulating it) - rust-lang#125412 (Don't suggest adding the unexpected cfgs to the build-script it-self) - rust-lang#125445 (Migrate `run-make/rustdoc-with-short-out-dir-option` to `rmake.rs`) - rust-lang#125452 (Cleanup check-cfg handling in core and std) r? `@ghost` `@rustbot` modify labels: rollup
rust-timer
added a commit
to rust-lang-ci/rust
that referenced
this pull request
May 24, 2024
Rollup merge of rust-lang#125345 - durin42:thin-link-bitcode, r=bjorn3 rustc_codegen_llvm: add support for writing summary bitcode Typical uses of ThinLTO don't have any use for this as a standalone file, but distributed ThinLTO uses this to make the linker phase more efficient. With clang you'd do something like `clang -flto=thin -fthin-link-bitcode=foo.indexing.o -c foo.c` and then get both foo.o (full of bitcode) and foo.indexing.o (just the summary or index part of the bitcode). That's then usable by a two-stage linking process that's more friendly to distributed build systems like bazel, which is why I'm working on this area. I talked some to `@teresajohnson` about naming in this area, as things seem to be a little confused between various blog posts and build systems. "bitcode index" and "bitcode summary" tend to be a little too ambiguous, and she tends to use "thin link bitcode" and "minimized bitcode" (which matches the descriptions in LLVM). Since the clang option is thin-link-bitcode, I went with that to try and not add a new spelling in the world. Per `@dtolnay,` you can work around the lack of this by using `lld --thinlto-index-only` to do the indexing on regular .o files of bitcode, but that is a bit wasteful on actions when we already have all the information in rustc and could just write out the matching minimized bitcode. I didn't test that at all in our infrastructure, because by the time I learned that I already had this patch largely written.
bjorn3
pushed a commit
to bjorn3/rust
that referenced
this pull request
Jun 30, 2024
rustc_codegen_llvm: add support for writing summary bitcode Typical uses of ThinLTO don't have any use for this as a standalone file, but distributed ThinLTO uses this to make the linker phase more efficient. With clang you'd do something like `clang -flto=thin -fthin-link-bitcode=foo.indexing.o -c foo.c` and then get both foo.o (full of bitcode) and foo.indexing.o (just the summary or index part of the bitcode). That's then usable by a two-stage linking process that's more friendly to distributed build systems like bazel, which is why I'm working on this area. I talked some to `@teresajohnson` about naming in this area, as things seem to be a little confused between various blog posts and build systems. "bitcode index" and "bitcode summary" tend to be a little too ambiguous, and she tends to use "thin link bitcode" and "minimized bitcode" (which matches the descriptions in LLVM). Since the clang option is thin-link-bitcode, I went with that to try and not add a new spelling in the world. Per `@dtolnay,` you can work around the lack of this by using `lld --thinlto-index-only` to do the indexing on regular .o files of bitcode, but that is a bit wasteful on actions when we already have all the information in rustc and could just write out the matching minimized bitcode. I didn't test that at all in our infrastructure, because by the time I learned that I already had this patch largely written.
GuillaumeGomez
added a commit
to GuillaumeGomez/rust
that referenced
this pull request
Jul 10, 2024
rustc_codegen_llvm: add support for writing summary bitcode Typical uses of ThinLTO don't have any use for this as a standalone file, but distributed ThinLTO uses this to make the linker phase more efficient. With clang you'd do something like `clang -flto=thin -fthin-link-bitcode=foo.indexing.o -c foo.c` and then get both foo.o (full of bitcode) and foo.indexing.o (just the summary or index part of the bitcode). That's then usable by a two-stage linking process that's more friendly to distributed build systems like bazel, which is why I'm working on this area. I talked some to `@teresajohnson` about naming in this area, as things seem to be a little confused between various blog posts and build systems. "bitcode index" and "bitcode summary" tend to be a little too ambiguous, and she tends to use "thin link bitcode" and "minimized bitcode" (which matches the descriptions in LLVM). Since the clang option is thin-link-bitcode, I went with that to try and not add a new spelling in the world. Per `@dtolnay,` you can work around the lack of this by using `lld --thinlto-index-only` to do the indexing on regular .o files of bitcode, but that is a bit wasteful on actions when we already have all the information in rustc and could just write out the matching minimized bitcode. I didn't test that at all in our infrastructure, because by the time I learned that I already had this patch largely written.
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Typical uses of ThinLTO don't have any use for this as a standalone file, but distributed ThinLTO uses this to make the linker phase more efficient. With clang you'd do something like
clang -flto=thin -fthin-link-bitcode=foo.indexing.o -c foo.cand then get both foo.o (full of bitcode) and foo.indexing.o (just the summary or index part of the bitcode). That's then usable by a two-stage linking process that's more friendly to distributed build systems like bazel, which is why I'm working on this area.I talked some to @teresajohnson about naming in this area, as things seem to be a little confused between various blog posts and build systems. "bitcode index" and "bitcode summary" tend to be a little too ambiguous, and she tends to use "thin link bitcode" and "minimized bitcode" (which matches the descriptions in LLVM). Since the clang option is thin-link-bitcode, I went with that to try and not add a new spelling in the world.
Per @dtolnay, you can work around the lack of this by using
lld --thinlto-index-onlyto do the indexing on regular .o files of bitcode, but that is a bit wasteful on actions when we already have all the information in rustc and could just write out the matching minimized bitcode. I didn't test that at all in our infrastructure, because by the time I learned that I already had this patch largely written.