armel: Compiling a project using tbb via cmake yields: undefined reference to symbol '__atomic_fetch_add_8@@LIBATOMIC_1.0'

[josch]

Summary

On armel, compiling cmake projects using onetbb fails because of missing -latomic. We worked around this in Debian using CMake snippets like this:

# link with -latomic if necessary
# on armel the build will otherwise fail with
# /usr/bin/ld: ../bin/libvcmi.so: undefined reference to `__atomic_fetch_add_8'
# /usr/bin/ld: ../bin/libvcmi.so: undefined reference to `__atomic_load_8'
if(CMAKE_LIBRARY_ARCHITECTURE STREQUAL "arm-linux-gnueabi")
	file(WRITE ${CMAKE_BINARY_DIR}/test_atomics.cpp
	"#include <stdint.h>\nint main(void){uint64_t x=0,y=0;return (int)__atomic_fetch_add_8(&x,y,0);}\n")
	try_compile(ATOMICS_BUILD_SUCCEEDED ${CMAKE_BINARY_DIR} ${CMAKE_BINARY_DIR}/test_atomics.cpp)
	message(STATUS "Found __sync_add_and_fetch_8(): ${ATOMICS_BUILD_SUCCEEDED}")
	if (NOT ATOMICS_BUILD_SUCCEEDED)
		target_link_options(vcmiserver PRIVATE "-Wl,--push-state,--no-as-needed,-latomic,--pop-state")
	endif ()
	file(REMOVE ${CMAKE_BINARY_DIR}/test_atomics.cpp)
endif()

One instance of this problem is vcmi: vcmi/vcmi#3109

The "armel" architecture is the only platform exhibiting that problem amongst the ones that we build vcmi for in Debian. Other platforms like armhf, arm64, mips, powerpc, riscv, s390x etc do not have this problem and build just fine.

I talked with the Debian porters for the ARM platform and was told that the problem should only happen on platforms without native atomics, so maybe the problem also happens on parisc or sh.

Version

2021.12.0-1

Environment

• Hardware: MNT Reform with Banana Pi CM4 A311D
• OS name and version: Debian GNU/Linux 13 (trixie, unstable)
• Compiler version: 4:13.2.0-7

Observed Behavior

Compiling a test executable on armel yielded an "undefined reference to symbol" error.

Expected Behavior

Successful compilation like on all the other architectures.

Steps To Reproduce

CMakeLists.txt:

cmake_minimum_required(VERSION 3.16.0)
project(tbbtest)
find_package(TBB REQUIRED)
add_executable(hello hello.cc)
target_link_libraries(hello PRIVATE TBB::tbb)

hello.cc

#include "tbb/parallel_for.h"
#include <iostream>
int main(int,char**) {
  tbb::parallel_for(
    tbb::blocked_range<size_t>(0,10),
    [&](const tbb::blocked_range<size_t>& r) {
      for (size_t i=r.begin();i<r.end();++i) std::cout << "hello" << std::endl;
    }
  );
  return 0;
}

Compile it:

$ cmake .
$ make
[ 50%] Building CXX object CMakeFiles/hello.dir/hello.cc.o
[100%] Linking CXX executable hello
/usr/bin/ld: CMakeFiles/hello.dir/hello.cc.o: undefined reference to symbol '__atomic_fetch_add_8@@LIBATOMIC_1.0'
/usr/bin/ld: /lib/arm-linux-gnueabi/libatomic.so.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/hello.dir/build.make:98: hello] Error 1
make[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/hello.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

[dnmokhov]

Thank you for reporting this.

Have you considered using the CMAKE_EXE_LINKER_FLAGS option, e.g.,

cmake -DCMAKE_EXE_LINKER_FLAGS="-Wl,--push-state,--no-as-needed,-latomic,--pop-state" .

[josch]

But that does not resolve the issue. It hides it.

Would a real solution not involve changing the TBBTargets.cmake such that it automatically adds the correct linker flags via the TBB::tbb interface properties?

[dnmokhov]

Hi @josch,

Sorry for the delayed response.

We already have an open pull request to address this issue: #987

Could you please see if it resolves it for you? If so, we could complete and merge the patch, once all the comments are addressed. Please fell free to contribute.

[josch]

Sorry for the delayed response.

No worries! I assume you are also just a volunteer doing this in your free time -- I know the problem. Thank you for getting back to me! 🙂

We already have an open pull request to address this issue: #987

The changes from that PR do not solve this issue. I patched the onetbb Debian package with the patch by @glaubitz and then compiled vcmi (the package where I observed the issue) with it. I'm getting the same error. I also tried my minimal reproducer from above without success.

I think (with my limited cmake knowledge) that the reason why it does not work is, that the atomic library gets added to TBB_COMMON_LINK_LIBS and those are used in src/tbb/CMakeLists.txt but with target_link_libraries(tbb PRIVATE .... The PRIVATE prevents the atomic library to get propagated to programs linked against tbb.

With that explanation in mind I tried out building tbb with the following patch:

--- a/src/tbb/CMakeLists.txt
+++ b/src/tbb/CMakeLists.txt
@@ -124,6 +124,8 @@ target_link_libraries(tbb
     Threads::Threads
     ${TBB_LIB_LINK_LIBS}
     ${TBB_COMMON_LINK_LIBS}
+    PUBLIC
+    atomic
 )
 
 tbb_install_target(tbb)

And with that I can compile my minimal reproducer and I get this:

$ make VERBOSE=1
[...]
[100%] Linking CXX executable //hello
/usr/bin/cmake -E cmake_link_script //CMakeFiles/hello.dir/link.txt --verbose=1
/usr/bin/c++ CMakeFiles/hello.dir/hello.cc.o -o //hello  usr/lib/arm-linux-gnueabi/libtbb.so.12.12 -latomic

As you can see, by adding atomic to PUBLIC, cmake projects that link against tbb will also link against atomic and this will make the build succeed. So maybe the patch from #987 could be adjusted to add atomic to PUBLIC link libraries instead of to the PRIVATE ones?

But that does not resolve the issue. It hides it.

Just pointing out that even this is hiding the issue. The real issue seems to be in gcc: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81358