Support auto linking on Apple targets #121293
Comments
rustbot
added
the
needs-triage
|
CC people whom I know have worked with macOS linking before: @BlackHoleFox, @bjorn3, @simlay |
|
This would be great to have! For ELF I recently found the |
bjorn3
added
A-linkage
|
On Windows we could embed static dependencies in static libraries (using the |
Noratrieb
added
T-compiler
|
Seems neat to me. Is auto-linking available in |
Just checked, |
Agreed.
I think I'd argue that:
In this specific case, it's a bit more difficult, as we'd have to choose one of these options ( |
Hmm, I don't know much about Windows, are there different linkers there, and what are their names? Seems like LLVM's |
Rustc doesn't know which linker will be used and the build system that invokes the linker likely allows picking a linker which doesn't support this feature. As such it has to respect |
Passing the `-weak-l` / `-weak_framework`, and makes the dynamic linker `dyld` wait with resolving symbols until they're actually used. This is somewhat niche, but I have at least three different reasons to do this: - If we ever want to support rust-lang#121293 in some shape or form, or just generally want `rustc` to have a more global view of the linking, we need to support the `-weak_framework` linker flag. Supporting `-weak_framework` natively - Never makes sense together with `as_needed`. https://github.com/denoland/deno/blob/ab18dac09d9e5b00afee55ae0108f7c98bb2e2d3/.cargo/config.toml#L14-L24
Make `#[used]` work when linking with `ld64` To make `#[used]` work in static libraries, we use the `symbols.o` trick introduced in rust-lang#95604. However, the linker shipped with Xcode, ld64, works a bit differently from other linkers; in particular, [it completely ignores undefined symbols by themselves](https://github.com/apple-oss-distributions/ld64/blob/ld64-954.16/src/ld/parsers/macho_relocatable_file.cpp#L2455-L2468), and only consider them if they have relocations (something something atoms something fixups, I don't know the details). So to make the `symbols.o` file work on ld64, we need to actually insert a relocation. That's kinda cumbersome to do though, since the relocation must be valid, and hence must point to a valid piece of machine code, and is hence very architecture-specific. Fixes rust-lang#133491, see that for investigation. --- Another option would be to pass `-u _foo` to the final linker invocation. This has the problem that `-u` causes the linker to not be able to dead-strip the symbol, which is undesirable. (If we did this, we would possibly also want to do it by putting the arguments in a file by itself, and passing that file via ``@`,` e.g. ``@undefined_symbols.txt`,` similar to rust-lang#52699, though that [is only supported since Xcode 12](https://developer.apple.com/documentation/xcode-release-notes/xcode-12-release-notes#Linking), and I'm not sure we wanna bump that). Various other options that are probably all undesirable as they affect link time performance: - Pass `-all_load` to the linker. - Pass `-ObjC` to the linker (the Objective-C support in the linker has different code paths that load more of the binary), and instrument the binaries that contain `#[used]` symbols. - Pass `-force_load` to libraries that contain `#[used]` symbols. Failed attempt: Embed `-u _foo` in the object file with `LC_LINKER_OPTION`, akin to rust-lang#121293. Doesn't work, both because `ld64` doesn't read that from archive members unless it already has a reason to load the member (which is what this PR is trying to make it do), and because `ld64` only support the `-l`, `-needed-l`, `-framework` and `-needed_framework` flags in there. --- TODO: - [x] Support all Apple architectures. - [x] Ensure that this works regardless of the actual type of the symbol. - [x] Write up more docs. - [x] Wire up a few proper tests. `@rustbot` label O-apple
Problem statement
Using a statically compiled Rust library from another language / with other toolchains can be somewhat troublesome, since the Rust library may have linking requirements that the user then has to manually pass to the linker.
Specific example: I have a static Rust library compiled for macOS that uses Winit and Wgpu and exposes a
rust_mainfunction, which I then pass to Xcode to link together with a small C main file that calls into Rust. Under the hood, Winit/Wgpu requires linking toAppKit,Metaland other such system frameworks. I'm required to also specify in Xcode the frameworks that my binary requires.Apart from being difficult to set up (as I, the user, has to know enough about linking to understand the inscrutable error messages), this workflow is also a possible SemVer hazard, as a library like Winit cannot add a dependency on a new the system library in a minor version without possibly breaking the user's build.
This can be fixed by using the
--print=native-static-libsrustcflag, but that probably requires more work for the user to integrate into the workflow than the quick fix of "just link to the broken library", and is also not very discoverable.Proposed solution
Swift and Clang have the concept of "auto linking", where the compiler will instruct the linker to link to the correct external libraries if the user imports code from one of these libraries. This is enabled in Clang with
-fmodules, and can be further controlled with the-fno-autolinkflag, or manually inserted with-Xclang --linker-option=xyz.On the surface, this provides much the same functionality as Rust with its
#[link(...)]attribute, but there's an important distinction: the dependency information is embedded in the object file, and understood by the linker itself, which allows it to work without Swift/Clang driving the linker invocation!This would fix #110143.
Implementation notes
Mach-O binaries uses the
LC_LINKER_OPTIONload command for this, which is understood by LLVM's lld, and Apple's ld64. A few resources on that:ELF binaries linking with LLVM's lld can use the
.linker-optionssection or the.deplibssection. LLVM also has thellvm.linker.optionsnamed metadata.Additional complication:
ld64will not pick these up from archive members unless it's already loading the archive member, see also #133832. Sincerustcuses a lot of codegen units, and as such the specific codegen unit -> object file / archive member that contains the linker options might not be loaded by the linker.As you can see, this is unfortunately quite platform-specific, and depends on the capabilities of the linker, so it probably isn't solvable in the general case; but I'd argue that this is still something that we could slowly improve support for, since this has a clear graceful fallback.
I'd be willing to (attempt to) implement this if you think this is desired?
Drawbacks / Unknowns
More complex linking integration, would this work for "Rust lib depending on Rust static lib" use-case, if that's even possible today?
Linker arguments are inherently ordered, and may be unexpectedly jumbled by this transformation? Would have to be properly researched and tested.
Sometimes, the user may want to opt-out of this behaviour. This can be done with the linker flag
-ignore_auto_link, though we should probably document that somewhere once this has been implemented.The text was updated successfully, but these errors were encountered: