Make check GpuUtilsUnitTests fails - CUDA 11 + Titan V

GROMACS version: 2020.03
GROMACS modification: No

Hi all,
I’m trying to build the latest gromacs on an Ubuntu 18.04 machine that has an NVidia Titan V using CUDA 11. The compile proceeds fine but unfortunately I get three errors during ‘make check’:

[ RUN ] HostAllocatorTestCopyable/0.ManualPinningOperationsWorkWithCuda
/path/to/gromacs-2020.3/src/gromacs/gpu_utils/tests/hostallocator.cpp:341: Failure
Value of: isPinned(input)
Actual: true
Expected: false
[ FAILED ] HostAllocatorTestCopyable/0.ManualPinningOperationsWorkWithCuda, where TypeParam = int (109 ms)

[ RUN ] HostAllocatorTestCopyable/1.ManualPinningOperationsWorkWithCuda
/path/to/gromacs-2020.3/src/gromacs/gpu_utils/tests/hostallocator.cpp:341: Failure
Value of: isPinned(input)
Actual: true
Expected: false
[ FAILED ] HostAllocatorTestCopyable/1.ManualPinningOperationsWorkWithCuda, where TypeParam = float (109 ms)

[ RUN ] HostAllocatorTestCopyable/2.ManualPinningOperationsWorkWithCuda
/path/to/gromacs-2020.3/src/gromacs/gpu_utils/tests/hostallocator.cpp:341: Failure
Value of: isPinned(input)
Actual: true
Expected: false
[ FAILED ] HostAllocatorTestCopyable/2.ManualPinningOperationsWorkWithCuda, where TypeParam = gmx::BasicVector (110 ms)

My cmake parameters are as follows:
cmake … -DGMX_BUILD_OWN_FFTW=ON -DGMX_CUDA_TARGET_SM=70 -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_GPU=on

and ‘nvidia-smi’ output is as follows:
Sat Jul 25 09:18:48 2020
±----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 TITAN V On | 00000000:65:00.0 On | N/A |
| 31% 47C P2 39W / 250W | 345MiB / 12063MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

Are there any known issues with CUDA 11 and/or compiling GROMACS with support for a Volta GPU? Or have I set up the build incorrectly? I’ve included the results of the cmake command below (for the sake of completeness). Happy to provide any additional information that could help troubleshoot.

Many thanks,

Stephen

$ cmake … -DGMX_BUILD_OWN_FFTW=ON -DGMX_CUDA_TARGET_SM=70 -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_GPU=on
– The C compiler identification is GNU 7.5.0
– The CXX compiler identification is GNU 7.5.0
– Check for working C compiler: /usr/bin/cc
– Check for working C compiler: /usr/bin/cc – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Check for working CXX compiler: /usr/bin/c++
– Check for working CXX compiler: /usr/bin/c++ – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Looking for NVIDIA GPUs present in the system
– Number of NVIDIA GPUs detected: 1
– Looking for pthread.h
– Looking for pthread.h - found
– Looking for pthread_create
– Looking for pthread_create - not found
– Looking for pthread_create in pthreads
– Looking for pthread_create in pthreads - not found
– Looking for pthread_create in pthread
– Looking for pthread_create in pthread - found
– Found Threads: TRUE
– Found CUDA: /usr/local/cuda (found suitable version “11.0”, minimum required is “9.0”)
– Found OpenMP_C: -fopenmp (found version “4.5”)
– Found OpenMP_CXX: -fopenmp (found version “4.5”)
– Found OpenMP: TRUE (found version “4.5”)
– Performing Test CFLAGS_EXCESS_PREC
– Performing Test CFLAGS_EXCESS_PREC - Success
– Performing Test CFLAGS_COPT
– Performing Test CFLAGS_COPT - Success
– Performing Test CFLAGS_NOINLINE
– Performing Test CFLAGS_NOINLINE - Success
– Performing Test CXXFLAGS_EXCESS_PREC
– Performing Test CXXFLAGS_EXCESS_PREC - Success
– Performing Test CXXFLAGS_COPT
– Performing Test CXXFLAGS_COPT - Success
– Performing Test CXXFLAGS_NOINLINE
– Performing Test CXXFLAGS_NOINLINE - Success
– Looking for include file unistd.h
– Looking for include file unistd.h - found
– Looking for include file pwd.h
– Looking for include file pwd.h - found
– Looking for include file dirent.h
– Looking for include file dirent.h - found
– Looking for include file time.h
– Looking for include file time.h - found
– Looking for include file sys/time.h
– Looking for include file sys/time.h - found
– Looking for include file io.h
– Looking for include file io.h - not found
– Looking for include file sched.h
– Looking for include file sched.h - found
– Looking for include file xmmintrin.h
– Looking for include file xmmintrin.h - found
– Looking for gettimeofday
– Looking for gettimeofday - found
– Looking for sysconf
– Looking for sysconf - found
– Looking for nice
– Looking for nice - found
– Looking for fsync
– Looking for fsync - found
– Looking for _fileno
– Looking for _fileno - not found
– Looking for fileno
– Looking for fileno - found
– Looking for _commit
– Looking for _commit - not found
– Looking for sigaction
– Looking for sigaction - found
– Performing Test HAVE_BUILTIN_CLZ
– Performing Test HAVE_BUILTIN_CLZ - Success
– Performing Test HAVE_BUILTIN_CLZLL
– Performing Test HAVE_BUILTIN_CLZLL - Success
– Looking for clock_gettime in rt
– Looking for clock_gettime in rt - found
– Looking for feenableexcept in m
– Looking for feenableexcept in m - found
– Looking for fedisableexcept in m
– Looking for fedisableexcept in m - found
– Checking for sched.h GNU affinity API
– Performing Test sched_affinity_compile
– Performing Test sched_affinity_compile - Success
– Looking for include file mm_malloc.h
– Looking for include file mm_malloc.h - found
– Looking for include file malloc.h
– Looking for include file malloc.h - found
– Checking for _mm_malloc()
– Checking for _mm_malloc() - supported
– Looking for posix_memalign
– Looking for posix_memalign - found
– Looking for memalign
– Looking for memalign - not found
– Check if the system is big endian
– Searching 16 bit integer
– Looking for sys/types.h
– Looking for sys/types.h - found
– Looking for stdint.h
– Looking for stdint.h - found
– Looking for stddef.h
– Looking for stddef.h - found
– Check size of unsigned short
– Check size of unsigned short - done
– Using unsigned short
– Check if the system is big endian - little endian
– Looking for HWLOC
– Looking for hwloc_topology_init
– Looking for hwloc_topology_init - found
– hwloc version:
– Found HWLOC: /usr/lib/x86_64-linux-gnu/libhwloc.so (found suitable version “1.11.6”, minimum required is “1.5”)
– Looking for C++ include pthread.h
– Looking for C++ include pthread.h - found
– Atomic operations found
– Performing Test PTHREAD_SETAFFINITY
– Performing Test PTHREAD_SETAFFINITY - Success
– Adding work-around for issue compiling CUDA code with glibc 2.23 string.h
– Check for working NVCC/C++ compiler combination with nvcc ‘/usr/local/cuda/bin/nvcc’
– Check for working NVCC/C++ compiler combination - works
– Checking for GCC x86 inline asm
– Checking for GCC x86 inline asm - supported
– Detected build CPU vendor - Intel
– Detected build CPU brand - Intel® Core™ i7-9800X CPU @ 3.80GHz
– Detected build CPU family - 6
– Detected build CPU model - 85
– Detected build CPU stepping - 4
– Detected build CPU features - aes apic avx avx2 avx512f avx512cd avx512bw avx512vl clfsh cmov cx8 cx16 f16c fma hle htt intel lahf mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp rtm sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
– Enabling RDTSCP support - detected on the build host
– Checking for 64-bit off_t
– Checking for 64-bit off_t - present
– Checking for fseeko/ftello
– Checking for fseeko/ftello - present
– Checking for SIGUSR1
– Checking for SIGUSR1 - found
– Checking for pipe support
– Checking for system XDR support
– Checking for system XDR support - present
– Detecting best SIMD instructions for this CPU
– Performing Test C_xCORE_AVX512_qopt_zmm_usage_high_FLAG_ACCEPTED
– Performing Test C_xCORE_AVX512_qopt_zmm_usage_high_FLAG_ACCEPTED - Failed
– Performing Test C_xCORE_AVX512_FLAG_ACCEPTED
– Performing Test C_xCORE_AVX512_FLAG_ACCEPTED - Failed
– Performing Test C_mavx512f_mfma_FLAG_ACCEPTED
– Performing Test C_mavx512f_mfma_FLAG_ACCEPTED - Success
– Performing Test C_mavx512f_mfma_COMPILE_WORKS
– Performing Test C_mavx512f_mfma_COMPILE_WORKS - Success
– Performing Test CXX_xCORE_AVX512_qopt_zmm_usage_high_FLAG_ACCEPTED
– Performing Test CXX_xCORE_AVX512_qopt_zmm_usage_high_FLAG_ACCEPTED - Failed
– Performing Test CXX_xCORE_AVX512_FLAG_ACCEPTED
– Performing Test CXX_xCORE_AVX512_FLAG_ACCEPTED - Failed
– Performing Test CXX_mavx512f_mfma_FLAG_ACCEPTED
– Performing Test CXX_mavx512f_mfma_FLAG_ACCEPTED - Success
– Performing Test CXX_mavx512f_mfma_COMPILE_WORKS
– Performing Test CXX_mavx512f_mfma_COMPILE_WORKS - Success
– Detected best SIMD instructions for this CPU - AVX_512
– Enabling 512-bit AVX-512 SIMD instructions using CXX flags: -mavx512f -mfma
– Performing Test _callconv___vectorcall
– Performing Test _callconv___vectorcall - Failed
– Performing Test _callconv___regcall
– Performing Test _callconv___regcall - Failed
– Performing Test callconv
– Performing Test callconv - Success
– The GROMACS-managed build of FFTW 3 will configure with the following optimizations: --enable-sse2;–enable-avx;–enable-avx2;–enable-avx512
– Using external FFT library - FFTW3 build managed by GROMACS
– A library with BLAS API not found. Please specify library location.
– Using GROMACS built-in BLAS.
– LAPACK requires BLAS
– A library with LAPACK API not found. Please specify library location.
– Using GROMACS built-in LAPACK.
– Checking for dlopen
– Performing Test HAVE_DLOPEN
– Performing Test HAVE_DLOPEN - Success
– Checking for dlopen - found
– Using dynamic plugins (e.g VMD-supported file formats)
– Checking for suitable VMD version
– VMD plugins not found. Path to VMD can be set with VMDDIR.
– Using default binary suffix: “”
– Using default library suffix: “”
– Could not convert sample image, ImageMagick convert can not be used. A possible way to fix it can be found here: https://alexvanderbist.com/posts/2018/fixing-imagick-error-unauthorized
Traceback (most recent call last):
File “”, line 1, in
ModuleNotFoundError: No module named ‘pygments’
– Could NOT find Sphinx (missing: SPHINX_EXECUTABLE pygments) (Required is at least version “1.6.1”)
– Performing Test HAVE_NO_DEPRECATED_COPY
– Performing Test HAVE_NO_DEPRECATED_COPY - Success
– Performing Test HAS_NO_STRINGOP_TRUNCATION
– Performing Test HAS_NO_STRINGOP_TRUNCATION - Success
– Performing Test HAS_NO_UNUSED_MEMBER_FUNCTION
– Performing Test HAS_NO_UNUSED_MEMBER_FUNCTION - Success
– Performing Test HAS_NO_REDUNDANT_MOVE
– Performing Test HAS_NO_REDUNDANT_MOVE - Success
– Performing Test HAS_NO_UNUSED
– Performing Test HAS_NO_UNUSED - Success
– Performing Test HAS_NO_UNUSED_PARAMETER
– Performing Test HAS_NO_UNUSED_PARAMETER - Success
– Performing Test HAS_NO_MISSING_DECLARATIONS
– Performing Test HAS_NO_MISSING_DECLARATIONS - Success
– Performing Test HAS_NO_NULL_CONVERSIONS
– Performing Test HAS_NO_NULL_CONVERSIONS - Success
– Performing Test HAS_DECL_IN_SOURCE
– Performing Test HAS_DECL_IN_SOURCE - Failed
– Performing Test HAS_NO_CLASS_MEMACCESS
– Performing Test HAS_NO_CLASS_MEMACCESS - Success
– Check if the system is big endian
– Searching 16 bit integer
– Using unsigned short
– Check if the system is big endian - little endian
– Looking for inttypes.h
– Looking for inttypes.h - found
– Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
– Doxygen not found. Documentation targets will not be generated.
Downloading: http://gerrit.gromacs.org/download/regressiontests-2020.3.tar.gz
– [download 100% complete]
– [download 0% complete]

– [download 100% complete]
– Configuring done
– Generating done
– Build files have been written to: /path/to/gromacs-2020.3/build

Gromacs doesn’t support CUDA 11 yet. I have same issue and haven’t found any solution yet. See this thread, might help in your case.

Great - thanks. (My extensive googling of the error message didn’t find your previous post!). I’ll give it a go compiling with CUDA 10.

Please let us know if using CUDA 10 fixes your problem. I’m hesitant to switch downgrade CUDA but if it does fix the problem I might consider it myself.

Yep, building with CUDA 10.2 worked smoothly and the build now passes all the regression tests. I’ll update if we have any problems when running mdrun, but otherwise I think we can mark this as solved.

I’ve the same problem, Cuda 11 vs cuda 10.2, but have not been able to uninstall cuda 11 and replace with cuda 10. I’ve used ```
sudo apt-get purge nvidia*
sudo apt-get autoremove
sudo apt-get autoclean
sudo rm -rf /usr/local/cuda* from:

the instructions from:
https://medium.com/@DataCrunch.io/installing-cuda-on-your-datacrunch-io-server-8bbc1cf571ee
produce the same result.

Purging is pretty serious stuff yet it does not clean everything. What are the tricks ?
----
It may be easier to use the beta version:  if i can find the githup master

thanks
pb

To address the Gromacs 2020.3 and CUDA 11 issues. Do i need to downgrade the NVIDIA drivers also, or just installing CUDA 10.2 will solve the problem. Iam asking think because i tried with
NVIDIA-SMI 450.51.06 Driver Version: 450.51.06 CUDA Version: 11.0
CUDA toolkit 10.1
but it did not work out

Thank you

I can’t say for sure, but I removed the NVidia driver and CUDA at the same time when trying to trouble-shoot this. I’d guess that using a matched pairing of CUDA generation and NVidia driver is a good idea…

Cheers,

Stephen