curand and cusparse errors while compiling the LinearSolver tutorials using the CMake with CUDA and HYPRE

[ruohai0925]

We are trying to compile the LinearSolver tutorials using the CMake with CUDA and HYPRE enabled under the Linux system. During the link time, we get the curand and cusparse errors.

For example,

[ 90%] Linking CUDA executable NodalPoisson
[ 90%] Linking CUDA device code CMakeFiles/NodeTensorLap.dir/cmake_device_link.o
/usr/bin/ld: /home/zengx372/hypre/src/hypre/lib/libHYPRE.a(device_utils.c.o): in function `hypre_DeviceDataCurandGenerator(hypre_DeviceData*) [clone .part.0]':
tmpxft_00030c87_00000000-6_device_utils.cudafe1.cpp:(.text+0xec6): undefined reference to `curandCreateGenerator'
/usr/bin/ld: tmpxft_00030c87_00000000-6_device_utils.cudafe1.cpp:(.text+0xedd): undefined reference to `curandSetPseudoRandomGeneratorSeed'
/usr/bin/ld: tmpxft_00030c87_00000000-6_device_utils.cudafe1.cpp:(.text+0xef1): undefined reference to `curandSetGeneratorOffset'
/usr/bin/ld: tmpxft_00030c87_00000000-6_device_utils.cudafe1.cpp:(.text+0xf22): undefined reference to `curandSetStream'
/usr/bin/ld: /home/zengx372/hypre/src/hypre/lib/libHYPRE.a(device_utils.c.o): in function `hypre_DeviceDataDestroy(hypre_DeviceData*)':
tmpxft_00030c87_00000000-6_device_utils.cudafe1.cpp:(.text+0x10e2): undefined reference to `curandDestroyGenerator'
/usr/bin/ld: tmpxft_00030c87_00000000-6_device_utils.cudafe1.cpp:(.text+0x10f8): undefined reference to `cusparseDestroy'
/usr/bin/ld: tmpxft_00030c87_00000000-6_device_utils.cudafe1.cpp:(.text+0x1183): undefined reference to `cusparseGetErrorString'

If we use the GNUMake to compile the tutorials with CUDA and HYPRE, everything is fine.
If we use CMake to compile the tutorials only with CUDA, it is also fine. The bug appears when we compile the tutorials using the CMake with CUDA and HYPRE.

[ruohai0925]

@etpalmer63 Here is the way to reproduce the above error.

1. Fix the bug A bug related to the data type #2896,

2. Follow Segfault while running the hypre tutorial using CUDA GPU #2898 (comment) to set up the HYPRE

3. Follow https://amrex-codes.github.io/amrex/tutorials_html/Hypre_Install.html#building-with-hypre-via-cmake to build the tutorial with HYPRE via CMake. Be sure to call CMake with the following configuration,

cmake .. -DAMReX_HYPRE=ON -DHYPRE_LIBRARIES=${HYPRE_DIR}/lib/libHYPRE.a -DHYPRE_INCLUDE_DIRS=${HYPRE_DIR}/include -DAMReX_LINEAR_SOLVERS=ON -DAMReX_FORTRAN=ON -DAMReX_PRECISION=SINGLE -DAMReX_GPU_BACKEND=CUDA -DAMReX_MPI=OFF -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_CUDA_ARCHITECTURES="60"

4. The above error appears when build the executable with

cmake --build . -j8

[etpalmer63]

@ruohai0925 , thanks for putting this up. Do you also build HYPRE the same way you did in #2898?

[ruohai0925]

@etpalmer63 Yes, exactly the same way.

[ruohai0925]

As @WeiqunZhang suggested in the slack channel, here is one way to bypass this error.

1. There is a file cmbuild/LinearSolvers/ABecLaplacian_C/CMakeFiles/ABecLaplacian_C.dir/link.txt generated by cmake configuration. Just edit that file to add
   -lcusparse -lcurand
   in the end. It seems that CMake appends the library file (using lowercase L) before the search path (using uppercase L) in the link line.

2. The above error disappears when build the executable with
   cmake --build . -j8

[etpalmer63]

@ruohai0925 , I looked into this and I have more questions than answers. When I follow your steps, I can't get past the AMReX build step. To help me out, can you tell me:

1. Do you also have the environment variable, HYPRE_HOME defined?
2. What is the result of echo $HYPRE_DIR? Does that match ~/hypre/src/hypre?
3. What is the result of find ${HYPER_DIR}/include -name "*hpp"?

Thanks!

[ruohai0925]

@etpalmer63

1. I defined HYPRE_HOME in .bashrc: export HYPRE_DIR=/home/zengx372/hypre/src/hypre
2. echo $HYPRE_DIR -> /home/zengx372/hypre/src/hypre. Please change ~/hypre/src/hypre to your $HYPRE_DIR
3. This is a hypre related issue (https://github.com/hypre-space/hypre/pull/710/files). We submitted a PR in the hypre repo to fix that.

Thanks!

[etpalmer63]

@ruohai0925 , RE:3 Yes! That was the file preventing me from compiling. Thanks! 💪