Subprofile support ((cross) toolchains, native binaries during cross builds)

[chris-se]

We are looking into conan to replace a set of home-grown bash scripts that are not really maintainable anymore (thank you very much for this project by the way!) and there is one crucial feature missing in conan that we'd need to be able to switch over:

We sometimes build our own toolchains, and conan poses two very relatd problems here:

1. In the native case on Linux system we'd like to build a current gcc ourselves instead of using the distribution's native compiler. However, to build gcc we already need a compiler.

   From reading the docs and playing around with conan a bit we've come to the following solution:

   • create a bunch of packages for the compiler and its dependencies (gcc, gmp, isl, etc.)

   • have the gcc package inject CC/CXX variables

   • create a profile that sets [settings] compiler = ... and has [build_requires] gcc

   • compile our software with this profile

   This is doable, but has the following problems:

   • The compiler setting is wrong when compiling gcc, as we are using the system compiler - and while that is possible because of not setting CC/CXX in the profile itself (but rather injecting them), the package is passed the wrong settings. This is fine for the compiler itself (where we'd remove the compiler settings from the package and build id because it doesn't matter with what it was compiled), but the libraries on which the compiler depends on is another matter.

   • Any compiler-specific flags we set in the profile would also be applied when building the compiler and its dependencies. For example, if we want to set CXXFLAGS=-std=c++2a in the profile (because we want the other software to be compiled for the next C++ standard) but the system compiler doesn't support that setting, we won't be able to build the compiler anymore.

     This is even worse if we decide to build a cross-compiler: if we want to cross-compile for arm, for example, and set something like -march=something_arm_specific, that will immediately fail when building the cross-compiler itself. (See also the next point.)

     Of course one could build the compiler manually (with a different profile but that results in the same settings), but that removes a lot of the possible automation that conan brings.

2. The more pressing case: cross-compiling. It's great that conan does support cross-compiling, but there's one crucial feature missing here: some software requires some programs to be built for the build system to work in a cross environment.

   For example, Qt needs to build its own qmake for the build architecture in order to be built, regardless of whether it's cross-compiled or not. But there are many other examples where the same thing occurs. (Let alone the fact that e.g. Qt will need to provide tools such as moc and uic in the build architecture for its dependencies.)

   Obviously you could set some specific variables for the build arch compiler and flags etc. in the profile itself, and have packages use those if present, but that wouldn't really be a very generic solution, because that would depend strongly on the convention chosen how to name those variables.

   This gets worse if you have further dependencies. This is not the case for Qt (as far as I remember), but if you have the following constellation:

   • Software package A should be cross-compiled, but needs a self-built tool during the build
     process that must be run on the build architecture.

   • Said build tool depends on an external library - but in that case the library has to be built for the build architecture.

   • Worse: in some cases the same library is needed both in the build architecture and the cross architecture.

   Then you have the problem that you need to specify a dependency for the external library package - but compiled for the build arcchitecture, not the actual target cross architecture.

I'd therefore like to propose the following scheme:

• When specifying dependencies in profiles, a user should have the possibility of specifying a different profile with whih the dependency (and all its sub-dependencies) should be added.

  This would solve the toolchain building problem. For example, if a user could specify in the profile that the profile has an additional build_requires on gcc, with profile 'default', conan could then just do the right thing.

• Profiles should have an additional option specifying another profile that is to be used for building binaries for the build architecture. (e.g. profile 'cross-rpi' sets 'default' as the profile to use for native compiling)

• When specifying dependencies in packages and conan files, the user should not be able to select a specific profile, as profiles are user-specific and the only thing that packages should care about are settings themselves. But one should be able to specify that a specific dependency should be loaded with the current subprofile that is to be used for the build architecture (set in the main profile, see previous point) and not the current profile itself. (For example, in a conanfile.txt one should be able to specify something like $PKG/$VER/build@$USER/$CHANNEL. (Feel free to bikeshed about the syntax.)

  By default (when not cross-compiling) this would be treated as a normal dependency.

• For packages: the Conan base class gets additional dictionaries self.build_env and self.build_settings that contain the environment and settings for the build architecture. If one isn't cross-compiling (or isn't using a subprofile) these will be identical to self.env and self.settings. Both are purely informative (they should never be part of the build id, build_os and build_arch are already available for that purpose), but may be used to pass the appropriate options to the package's build system.

• Generators / build system integration:

  • Some, such as CMake, don't support this kind of "build for two different architectures during the same build" at all (the CMake wiki mentions work-arounds for this specific issue). So they should just ignore these additional settings. (It's then up to the package creator to figure out how the upstream build system is integrated here.) So no changes should be required here.

  • But others such as autotools do. (You can pass the build CC as CC_FOR_BUILD, similarly with CXX, CFLAGS, etc.) They should be updated to support this mode.

To summarize what I'd like to have added:

• A setting to profiles that specify a 'build arch/os' profile that is to be used when native tools are needed during the build process of some software.

  e.g. 'cross-rpi' has 'default' as it's "build arch/os profile"

• A proper name and place for that setting in the current profille

• A way to specify arbitrary different profiles to use for added dependencies within profiles

  e.g. when a profile adds a toolchain package, it should be able to say that the toolchain should be built with profile 'x' instead of itself

• A proper syntax for that setting

• A way to specify that specific dependencies of packages are to use the current "build arch/os profile" instead of the current main one. It should not be possible to specify an arbitrary profile though, as profiles are external to packages, one should just be able to switch between the main profile and the build profile.

• A proper syntax for that setting

• Additional support for env and settings in the base class for conanfile.py packages so that builds that need binaries that run on the build system do work.

• Integration of the "build profile" into build system generators and/or integrations that support this (such as autotools).

I know this isn't a trivial request, and I'd be willing to do the actual implementation work - but first I would like some feedback, especially when it comes to the naming and syntax of these things.

It would be great if this could be added to conan, this is the main show-stopper for us to switch to it at the moment.

Thanks!

[lasote]

Thank you very very much for this issue, this morning we are in the middle of a great discussion about all this. We have answers for many of your concerns, but let us read and analyze all your use cases carefully and will go back to you ;)

I'm very glad you like Conan by the way, a great effort with the great support of the community, for example, with issues like this one, with gold information to think about.

[chris-se]

Many thanks for your very quick initial reply. 👍 And it really appears to be great timing that you're having that discussion right now.

I forgot to link this in my original report, but the following wiki page describes how CMake can work around the issue of building executables with a non-cross compiler during cross builds:

https://cmake.org/Wiki/CMake_Cross_Compiling#Using_executables_in_the_build_created_during_the_build

But as I said, autotools supports that directly, see https://www.gnu.org/software/autoconf-archive/ax_prog_cxx_for_build.html for example. And meson also supports it directly, see http://mesonbuild.com/Cross-compilation.html (you can specify whether you want a native or non-native compiler there, native being non-cross ("build") here).

Just in case this helps a bit during your discussion.

[lasote]

Thanks for the info. We introduced some new settings in Conan 1.0 to support
cross building: http://docs.conan.io/en/latest/systems_cross_building/cross_building.html , but we know that still, something is missing to cover all the use cases, especially when build_requires are introduced to cross build. We are trying to gather all use cases and see how can be solved, so many thanks for your help again.
By the way, note also that with conan/profiles you can specify env vars, settings, etc to a concrete package:

[env]
gmp.CXX=whatever
CXX=otherthing

[niosHD]

Hi all,

I am not quite sure if my use case contributes to this discussion. However, since I also toy around with packaging our cross-compilation toolchain with conan I think that describing it may be helpful.

In more detail, what I try to achieve is to package our llvm-based toolchain into its various components instead of having one huge monolithic toolchain package. For example, there should be a llvm package which contains the optimization and code generation parts of the compiler, the clang package with the C/C++ frontend, the lld package with the linker, and the lldb package with the debugger. Additionally, there should be packages for the runtime libraries like libunwind and compiler-rt as well as for the newlib C library and the libcxx C++ library. Note that these library packages have to be built for the target architectures. Furthermore, given that llvm is inherently a cross compiler, potentially many different architecture specific packages have to be build. A simplified build dependency graph for a C toolchain with only one supported target architecture may looks then something like the following:

llvm (HOST) <-----\---- clang (HOST)  <--/--- newlib (TARGET) <--- compiler-rt (TARGET)
                   \--- lld (HOST)    <-/
                    \-- lldb (HOST)
other deps. (HOST) <-\- other tools (HOST)

Whats interesting about the setup is that all components marked as HOST can provide binaries for the final toolchain as well as libraries for other packages. Unfortunately, the example in the documentation currently does not build anything for HOST or BUILD and only repackages an already existing toolchain. I therefore tried to use different settings versions (os+arch, os_build+arch_build, os+arch+os_build+arch_build) for my HOST packages but no configuration yielded the expected result. In the end, always when I try to compile the first TARGET package also its dependencies (the HOST packages) are rebuilt.

Is such a fine granular packaging and building of a toolchain currently supported by conan? If yes, whats the correct way to do it? If not, would the proposed changes allow to support such an flow?

Btw, is it a bug that the CMake helper does not work when arch is not defined as setting?

Best,
Mario

[mpdelbuono]

Just adding in here another viewpoint (repeating from #2497 to get these all in the same place):

I have the same kind of problem. While I don't make any modifications to clang directly, I do have a clang package (that I might end up submitting to conan-center at some point) because we have a plugin that we develop for clang. So we build clang, and all of its libraries are available in that package, which is then consumed by the plugin.

This plugin is a static analysis plugin, and as such it's a build dependency for other consumer packages. However, because it's a plugin for the compiler, the ABI compatibility is not relevant to consumers of the plugin. So there are two cases:

• If I am a plugin developer, I might want the debug version of clang so that I can troubleshoot my plugin
• If I am a regular consumer, I never want the debug version of clang, because it is slow, but I do want to build my own packages in debug mode

However, consumers of the clang package will likely set build_type=Debug because for their own project they do want debugging information. Similarly, consumers of the clang package care care about matching the build OS/arch, but clang would have been developed against the target OS/arch.

Effectively, the meaning of these settings changes depending upon whether you're in the "plugin developer" context or the "general developer" context.

For now I'm using the workaround of just telling plugin developers to use a special option to trigger the clang debug build, and the clang package ignores all target settings. However, eventually it would be nice if all this information could be tracked by conan. I do recognize, of course, that building this graph is definitely a challenge.

[KerstinKeller]

The possibility to require packages in the host architecture would also be essential for us, if we want to use Conan productively. This is mainly because we're using code generators (such as QT moc, Protobuf protoc, ...) in our build process.

I like the suggestion that was made in #2505, to distinguish between requires and host_requires. This is very similar to how native dependencies are handled in Yocto:
DEPENDS += "protobuf protobuf-native"
I'd like to see a way, to mark a dependeny as native.

Maintaining different packages (protoc vs protobuf) might quickly become complicated, and it doesn't address the issue that some of the host/native packages have build and runtime dependencies on other packages, as mentioned before. E.g. one of our code generators requires boost and we also require boost on the target, so what to do in that case?

I am aware that mixing cross and native builds make things a lot more complicated, especially regarding setting executable and library paths for runtime and especially using CMake's Find / Config logic.

But using Conan over a ton of custom build scripts / docker containers would be a huge improvement for us, so I hope to see some work in this direction.

[GeorgeScrivener]

I agree with @KerstinKeller, and I'm having a very similar issue at work which is .
I think it would actually be necessary to have 2 profiles so we can also build the build tools with a custom profile as well.
This actually becomes even more complicated when tools like thrift are introduced, that build both runtime libraries and build time tools. This might necessitate being able to swap between profiles from inside the conanfile.

[pvdhoff]

Dear Conan developer,
first thank you for your create application. We would really like to use this tool in production. But as we develop embedded systems on various architectures we really need this feature to cross compile our projects. Beside the compiler it selfe especially popular projects like Qt and Google flatbuffers need such tools running host machine. So is there any timeline for this feature?

[sztomi]

@pvdhoff Cross compiling is already possible, this GHI is about redesigning/improving that feature.

[pvdhoff]

@sztomi That‘s only part of the truth. As soon as you need to build tools running on the build platform (such as the compiler or some tools needed to build QT or flatc) you are somehow lost or need some hacks. So to be able to do cross compiling with all it‘s facets one need this feature. And sorry for that question, but what is a GHI

[sztomi]

GHI = github issue :)

As soon as you need to build tools running on the build platform (such as the compiler or some tools needed to build QT or flatc) you are somehow lost

Not sure I follow - are implying that you would be running the compiler on the target system?

[radonish]

Bumping this up - my need is related to using a host (x86_64) version of the Google Protocol Buffer compiler to covert the *.proto files to C++ code which will then get cross-compiled into a library targeted for ARM.

It seems like any 'build_requires' components should be using a specified "host profile" and 'requires' components should be using the selected target profile.

Eagerly looking forward for this feature to be implemented.

[DavidZemon]

You can't just use the host profile for all build_requires. For instance, I have a project that uses Google Test and the Google Test dependency is listed as a build_requires. It must be cross-compiled with the same profile as the rest of the application, but it should not propagate to other applications that depend on this one.

[memsharded]

This issue is very outdated. The new cross-compilation with 2 profiles is the default now in Conan 2.0, and has been long time active in 1.X. I am closing this as outdated, please create new tickets for any further question, thanks!