Compiling error fir GROMACS 2023.1 with GPU CUDA at libcudart.so.11.0

User discussions
,
1

GROMACS version: 2023.1
GROMACS modification: No

I am trying to compile Gromacs on a computer with NVIDIA GeForce RTX 3090 (sm86), and there is always an error towards the end of make, even though cmake passes successfully! The current error I have is at the bottom, and is related to “/home/pablo/miniconda3/bin/…/lib/gcc/x86_64-conda-linux-gnu/11.4.0/…/…/…/…/x86_64-conda-linux-gnu/bin/ld: /home/pablo/miniconda3/lib/libcudart.so.11.0: undefined reference to `memcpy@GLIBC_2.14’”. If I compile without GPU support, I am able to make a fully functional gmx executable

current versions:

Blockquote
(base) pablo@dl:~/repos/gromacs-2023.1/build$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal

Blockquote
(base) pablo@dl:~/repos/gromacs-2023.1/build$ gcc --version; g++ --version
gcc (conda-forge gcc 11.4.0-0) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
g++ (conda-forge gcc 11.4.0-0) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Blockquote
(base) pablo@dl:~/repos/gromacs-2023.1/build$ conda --version
conda 23.5.0
(base) pablo@dl:~/repos/gromacs-2023.1/build$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
(base) pablo@dl:~/repos/gromacs-2023.1/build$ cmake --version
cmake version 3.26.4
CMake suite maintained and supported by Kitware (kitware.com/cmake).

Blockquote
(base) pablo@dl:~/repos/gromacs-2023.1/build$ ls /home/pablo/miniconda3/lib/libcufft.so.10
/home/pablo/miniconda3/lib/libcufft.so.10
(base) pablo@dl:~/repos/gromacs-2023.1/build$ ls /home/pablo/miniconda3/lib/libcudart.so.11.0
/home/pablo/miniconda3/lib/libcudart.so.11.0

full story:

I tried running:
**cmake** .. -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_EXTERNAL_BLAS=OFF -DGMX_EXTERNAL_LAPACK=OFF -DGMX_GPU=CUDA -DCMAKE_INSTALL_PREFIX=/home/pablo/repos/gromacs_tar -DGMX_CUDA_TARGET_SM="86" -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc

and got libcudart.so.11.0 not found when building:

Blockquote
[ 96%] Linking CXX executable …/…/…/bin/argon-forces-integration
/home/pablo/miniconda3/bin/…/lib/gcc/x86_64-conda-linux-gnu/11.4.0/…/…/…/…/x86_64-conda-linux-gnu/bin/ld: warning: libcudart.so.11.0, needed by …/…/…/lib/libnblib_gmx.so.0.1.0, not found (try using -rpath or -rpath-link)
/home/pablo/miniconda3/bin/…/lib/gcc/x86_64-conda-linux-gnu/11.4.0/…/…/…/…/x86_64-conda-linux-gnu/bin/ld: warning: libcufft.so.10, needed by /home/pablo/repos/gromacs-2023.1/build/lib/libgromacs.so.8, not found (try using -rpath or -rpath-link)
/home/pablo/miniconda3/bin/…/lib/gcc/x86_64-conda-linux-gnu/11.4.0/…/…/…/…/x86_64-conda-linux-gnu/bin/ld: /home/pablo/repos/gromacs-2023.1/build/lib/libgromacs.so.8: undefined reference to cufftSetStream@libcufft.so.10' /home/pablo/miniconda3/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/pablo/repos/gromacs-2023.1/build/lib/libgromacs.so.8: undefined reference to cufftExecR2C@libcufft.so.10’
/home/pablo/miniconda3/bin/…/lib/gcc/x86_64-conda-linux-gnu/11.4.0/…/…/…/…/x86_64-conda-linux-gnu/bin/ld: /home/pablo/repos/gromacs-2023.1/build/lib/libgromacs.so.8: undefined reference to cufftExecC2R@libcufft.so.10' /home/pablo/miniconda3/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/pablo/repos/gromacs-2023.1/build/lib/libgromacs.so.8: undefined reference to cufftDestroy@libcufft.so.10’
/home/pablo/miniconda3/bin/…/lib/gcc/x86_64-conda-linux-gnu/11.4.0/…/…/…/…/x86_64-conda-linux-gnu/bin/ld: /home/pablo/repos/gromacs-2023.1/build/lib/libgromacs.so.8: undefined reference to `cufftPlanMany@libcufft.so.10’
collect2: error: ld returned 1 exit status
make[2]: *** [api/nblib/samples/CMakeFiles/argon-forces-integration.dir/build.make:100: bin/argon-forces-integration] Error 1
make[1]: *** [CMakeFiles/Makefile2:3764: api/nblib/samples/CMakeFiles/argon-forces-integration.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

So then I conda installed that and libcufft.so.10 with:

- conda install -c anaconda cudatoolkit=11.8
- `conda install -c "nvidia/label/cuda-11.8.0" libcufft`

after Installing, I did export LD_LIBRARY_PATH=/home/pablo/miniconda3/lib, and reran

cmake … -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_EXTERNAL_BLAS=OFF -DGMX_EXTERNAL_LAPACK=OFF -DGMX_GPU=CUDA -DCMAKE_INSTALL_PREFIX=/home/pablo/repos/gromacs_tar -DGMX_CUDA_TARGET_SM=“86” -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc

but am now getting:

Blockquote
[ 96%] Linking CXX executable …/…/…/bin/argon-forces-integration
/home/pablo/miniconda3/bin/…/lib/gcc/x86_64-conda-linux-gnu/11.4.0/…/…/…/…/x86_64-conda-linux-gnu/bin/ld: /home/pablo/miniconda3/lib/libcudart.so.11.0: undefined reference to memcpy@GLIBC_2.14' /home/pablo/miniconda3/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld: /home/pablo/miniconda3/lib/libcudart.so.11.0: undefined reference to clock_gettime@GLIBC_2.17’
collect2: error: ld returned 1 exit status
make[2]: *** [api/nblib/samples/CMakeFiles/argon-forces-integration.dir/build.make:100: bin/argon-forces-integration] Error 1
make[1]: *** [CMakeFiles/Makefile2:3764: api/nblib/samples/CMakeFiles/argon-forces-integration.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

Could someone please help me fix this error because it seems inside of libcudart.so.11.0?? Thank you!!

2

You are mixing your system install of CUDA (/usr/local/cuda/bin/nvcc), which uses your system compiler and glibc, with the conda-installed version of the compiler toolchain (/home/pablo/miniconda3/bin/../lib/gcc/x86_64-conda-linux-gnu/11.4.0/../../../../x86_64-conda-linux-gnu/bin/ld), with a different glibc version (hence the *@GLIBC_2.17). As it happen, the clock_gettime symbol has changed from librt to libc around that version, hence the missing symbol.

I would suggest choosing either one or the other, and setting the following flags to clear possible mix-ups. Something like this for local install:

-DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.1/
-DCMAKE_CUDA_COMPILER=/usr/local/cuda-11.1/bin/nvcc
-DCMAKE_C_COMPILER=/usr/bin/gcc
-DCMAKE_CXX_COMPILER=/usr/bin/g++

(you may need to install cufft at the system level)

or find where the conda gcc/g++ compiler and CUDA are installed (somehwere in /home/pablo/miniconda3/) and use that for the above CMake flags with that.

1 Like
3

Thank you for your prompt reply. When I use the flags with local install paths you mention, I get not as far (93% instead of 96%):

cmake … -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_EXTERNAL_BLAS=OFF -DGMX_EXTERNAL_LAPACK=OFF -DGMX_GPU=CUDA -DCMAKE_INSTALL_PREFIX=/home/pablo/repos/gromacs_tar -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-11.8/bin/nvcc -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++

[ 93%] Linking CXX shared library …/…/lib/libgromacs.so
/usr/bin/ld: cannot find /lib64/libpthread.so.0
/usr/bin/ld: cannot find /usr/lib64/libpthread_nonshared.a
collect2: error: ld returned 1 exit status
make[2]: *** [src/gromacs/CMakeFiles/libgromacs.dir/build.make:13519: lib/libgromacs.so.9.0.0] Error 1
make[1]: *** [CMakeFiles/Makefile2:4236: src/gromacs/CMakeFiles/libgromacs.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

and when I try to see what is inside of the /lib64/ library. I do not see the libpthread_nonshared.a ANYWHERE. For the libpthread.so.0 file, I found it in /lib/x86_64-linux-gnu/libpthread.so.0. Do I have to install something? Would you know how to change the path to this? or would I have to ask an admin on my computer to make a symlink at /lib64/libpthread.so.0 that points to the path in the previous sentence? Would the -DCMAKE_PREFIX_PATH be able to help somehow?

(base) pablo@dl:~/repos/gromacs_gpu/build$ ls /lib64/
ld-linux-x86-64.so.2
(base) pablo@dl:~/repos/gromacs_gpu/build$ ls /usr/lib64
ld-linux-x86-64.so.2
(base) pablo@dl:~/repos/gromacs_gpu/build$ ls /usr/*/*libpthread*
/usr/lib32/libpthread-2.31.so /usr/lib32/libpthread.so.0
(base) pablo@dl:~/repos/gromacs_gpu/build$ ls /usr/*/*/*libpthread*
/usr/lib/i386-linux-gnu/libpthread-2.31.so
/usr/lib/x86_64-linux-gnu/libpthread-2.31.so
/usr/lib/x86_64-linux-gnu/libpthread.so
/usr/lib/i386-linux-gnu/libpthread.so.0
/usr/lib/x86_64-linux-gnu/libpthread.a
/usr/lib/x86_64-linux-gnu/libpthread.so.0
(base) pablo@dl:~/repos/gromacs_gpu/build$ ls /lib/*/*libpthread*
/lib/i386-linux-gnu/libpthread-2.31.so
/lib/x86_64-linux-gnu/libpthread-2.31.so
/lib/x86_64-linux-gnu/libpthread.so
/lib/i386-linux-gnu/libpthread.so.0
/lib/x86_64-linux-gnu/libpthread.a
/lib/x86_64-linux-gnu/libpthread.so.0
(base) pablo@dl:~/repos/gromacs_gpu/build$ ls /lib/*/*/*libpthread*
ls: cannot access '/lib/*/*/*libpthread*': No such file or directory

alternatively, when I try to use the conda paths, I get to the same place with the *@GLIBC_2.17 error.

4

Find which pthread you should use:

/sbin/ldconfig -p | grep pthread

Quick fix would then be something like: LDFLAGS=-L/usr/lib/x86_64-linux-gnu/ cmake [...], or whichever directory the pthread.so.0 was contained in, then build as normal.

5

The make completed with your suggestions. THANKS! however there were still some errors with make check that I will have to investigate.

(base) pablo@dl:~/repos/gromacs_gpu/build$ /sbin/ldconfig -p | grep pthread
libpthread.so.0 (libc6,x86-64, OS ABI: Linux 3.2.0) => /lib/x86_64-linux-gnu/libpthread.so.0
libpthread.so.0 (libc6, OS ABI: Linux 3.2.0) => /lib/i386-linux-gnu/libpthread.so.0
libpthread.so.0 (libc6, OS ABI: Linux 3.2.0) => /lib32/libpthread.so.0
libpthread.so (libc6,x86-64, OS ABI: Linux 3.2.0) => /lib/x86_64-linux-gnu/libpthread.so

LDFLAGS=-L/lib/x86_64-linux-gnu/ cmake … -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_EXTERNAL_BLAS=OFF -DGMX_EXTERNAL_LAPACK=OFF -DGMX_GPU=CUDA -DCMAKE_INSTALL_PREFIX=/home/pablo/repos/gromacs_tar -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-11.8/bin/nvcc -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++
Success!

However, the make check failed at a free energy check, which is what I wanted to use GROMACS for. I will find and look at the “checkforce.out” error file and can report back what happens, or I’m not sure if that would have to be on a new card…

85/87 Test #85: regressiontests/complex …Failed 62.32 sec
Re-running aminoacids using only CPU-based non-bonded kernels
Re-running awh_multidim using only CPU-based non-bonded kernels
Re-running awh_multidim using CPU-based update
Re-running butane using only CPU-based non-bonded kernels
Re-running dd121 using only CPU-based non-bonded kernels
Re-running distance_restraints using only CPU-based non-bonded kernels
Re-running distance_restraints using CPU-based update
Re-running ethyleenglycol using only CPU-based non-bonded kernels
Re-running field using only CPU-based non-bonded kernels
FAILED. Check checkforce.out (1 errors) file(s) in nacl for nacl
Re-running nacl using only CPU-based non-bonded kernels
Re-running nacl using CPU-based update
Re-running nbnxn-free-energy using only CPU-based non-bonded kernels
[…]
1 out of 99 complex tests FAILED
Start 86: regressiontests/freeenergy
86/87 Test #86: regressiontests/freeenergy … Passed 24.34 sec
Start 87: regressiontests/essentialdynamics
87/87 Test #87: regressiontests/essentialdynamics … Passed 4.84 sec
99% tests passed, 1 tests failed out of 87
Label Time Summary:
GTest = 108.63 secproc (82 tests)
IntegrationTest = 45.07 secproc (26 tests)
MpiTest = 75.37 secproc (19 tests)
QuickGpuTest = 17.65 secproc (18 tests)
SlowGpuTest = 176.63 secproc (17 tests)
SlowTest = 60.73 secproc (13 tests)
UnitTest = 2.83 secproc (43 tests)
Total Test time (real) = 200.19 sec
The following tests FAILED:
85 - regressiontests/complex (Failed)
Errors while running CTest
make[3]: *** [CMakeFiles/run-ctest-nophys.dir/build.make:71: CMakeFiles/run-ctest-nophys] Error 8
make[2]: *** [CMakeFiles/Makefile2:3271: CMakeFiles/run-ctest-nophys.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:3304: CMakeFiles/check.dir/rule] Error 2
make: *** [Makefile:628: check] Error 2

7

Hello! I had forgotten to do the make install!

now my gmx --version works well, even though there is that “failed check with nacl”. All the output files in /home/pablo/repos/gromacs_gpu/build/tests/complex/nacl look correct, so I will just continue to use this gmx executable as long as it keeps working! Thank you for all the help. As I mentioned, I had been trying to compile this for one whole week and your recommendations were extremely clear and helpful! Best.

Final cmake:

LDFLAGS=-L/lib/x86_64-linux-gnu/ cmake … -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_GPU=CUDA -DCMAKE_INSTALL_PREFIX=/home/pablo/repos/gromacs_tar -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-11.8 -DCMAKE_CUDA_COMPILER=/usr/local/cuda-11.8/bin/nvcc -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++

final result:

(base) pablo@dl:~/repos/gromacs_gpu/build$ gmx --version
:-) GROMACS - gmx, 2024-dev-20230613-db9d71ddbd (-:

Executable: /home/pablo/repos/gromacs_tar/bin/gmx
Data prefix: /home/pablo/repos/gromacs_tar
Working dir: /home/pablo/repos/gromacs_gpu/build
Command line:
gmx --version

GROMACS version: 2024-dev-20230613-db9d71ddbd
GIT SHA1 hash: db9d71ddbd84bd1c81f39a4fae0f8cdc627751ab
Precision: mixed
Memory model: 64 bit
MPI library: thread_mpi
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support: CUDA
NB cluster size: 8
SIMD instructions: AVX2_256
CPU FFT library: fftw-3.3.8-sse2-avx-avx2-avx2_128
GPU FFT library: cuFFT
Multi-GPU FFT: none
RDTSCP usage: enabled
TNG support: enabled
Hwloc support: disabled
Tracing support: disabled
C compiler: /usr/bin/gcc GNU 9.4.0
C compiler flags: -Wno-array-bounds -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wall -Wno-unused -Wunused-value -Wunused-parameter -Wextra -Wno-sign-compare -Wpointer-arith -Wundef -Werror=stringop-truncation -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler: /usr/bin/g++ GNU 9.4.0
C++ compiler flags: -Wno-array-bounds -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wall -Wextra -Wpointer-arith -Wmissing-declarations -Wundef -Wstringop-truncation -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp -O3 -DNDEBUG
BLAS library:
LAPACK library:
CUDA compiler: /usr/local/cuda-11.8/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2022 NVIDIA Corporation;Built on Wed_Sep_21_10:33:58_PDT_2022;Cuda compilation tools, release 11.8, V11.8.89;Build cuda_11.8.r11.8/compiler.31833905_0
CUDA compiler flags:-std=c++17;–generate-code=arch=compute_35,code=sm_35;–generate-code=arch=compute_37,code=sm_37;–generate-code=arch=compute_50,code=sm_50;–generate-code=arch=compute_52,code=sm_52;–generate-code=arch=compute_60,code=sm_60;–generate-code=arch=compute_61,code=sm_61;–generate-code=arch=compute_70,code=sm_70;–generate-code=arch=compute_75,code=sm_75;–generate-code=arch=compute_80,code=sm_80;–generate-code=arch=compute_86,code=sm_86;–generate-code=arch=compute_89,code=sm_89;–generate-code=arch=compute_90,code=sm_90;-Wno-deprecated-gpu-targets;–generate-code=arch=compute_53,code=sm_53;–generate-code=arch=compute_80,code=sm_80;-use_fast_math;-Xptxas;-warn-double-usage;-Xptxas;-Werror;-D_FORCE_INLINES;-Wno-array-bounds -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wall -Wextra -Wpointer-arith -Wmissing-declarations -Wundef -Wstringop-truncation -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp -O3 -DNDEBUG
CUDA driver: 12.10
CUDA runtime: 11.80

8

Note that you are using the (unstable) main branch, you should consider using a stable released version from here https://manual.gromacs.org/ (or the most up-to-date stable you can find in the git repository release-2023 branch).

Cheers,
Szilárd