# Make check failing when GPU enabled

**URL:** <https://gromacs.bioexcel.eu/t/make-check-failing-when-gpu-enabled/1837>\
**Category:** User discussions\
**Tags:** installation-error\
**Created:** [March 26, 2021, 3:56pm UTC](https://gromacs.bioexcel.eu/t/make-check-failing-when-gpu-enabled/1837 "2021-03-26T15:56:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mikewallis](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@mikewallis](https://gromacs.bioexcel.eu/u/mikewallis)\
**Post date:** [March 26, 2021, 3:56pm UTC](https://gromacs.bioexcel.eu/t/make-check-failing-when-gpu-enabled/1837/1 "2021-03-26T15:56:10Z")

</div>

GROMACS version: 2021.1  
GROMACS modification: No  
Here post your question

Hi folks,  
I’ve been trying to get a CUDA-enabled gmx to pass `make check` but it’s timing out on a _lot_ of tests. They’re (mostly) passing when run CPU-only, however.

> cmake -DCMAKE\_INSTALL\_PREFIX=${PWD}/…/gromacs\_final -DREGRESSIONTEST\_DOWNLOAD=ON -DCMAKE\_C\_COMPILER=gcc -DCMAKE\_CXX\_COMPILER=g++ …/ -DGMX\_GPU=CUDA -DCUDA\_TOOLKIT\_ROOT\_DIR=/exports/applications/apps/SL7/cuda/10.1.105 -DGMX\_CUDA\_TARGET\_SM=“35;37;60;61;62”

(the targets are explicitly set as the binary could run on either K80 or Titan-X GPUs)

> gmx mdrun -version
> 
> GROMACS version: 2021.1  
> Verified release checksum is 8c24bff5d3f78b0a9afb16e880b5667e5affe9a686d462482bac20ce975492c6  
> Precision: mixed  
> Memory model: 64 bit  
> MPI library: thread\_mpi  
> OpenMP support: enabled (GMX\_OPENMP\_MAX\_THREADS = 64)  
> GPU support: CUDA  
> SIMD instructions: AVX2\_256  
> FFT library: fftw-3.3.3-sse2  
> RDTSCP usage: enabled  
> TNG support: enabled  
> Hwloc support: disabled  
> Tracing support: disabled  
> C compiler: /exports/applications/apps/community/roslin/gcc/7.3.0/bin/gcc GNU 7.3.0  
> C compiler flags: -mavx2 -mfma -Wno-missing-field-initializers -fexcess-precision=fast -funroll-all-loops -pthread -O3 -DNDEBUG  
> C++ compiler: /exports/applications/apps/community/roslin/gcc/7.3.0/bin/g++ GNU 7.3.0  
> C++ compiler flags: -mavx2 -mfma -Wno-missing-field-initializers -fexcess-precision=fast -funroll-all-loops -pthread -fopenmp -O3 -DNDEBUG  
> CUDA compiler: /exports/applications/apps/SL7/cuda/10.1.105/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2019 NVIDIA Corporation;Built on Fri\_Feb\_\_8\_19:08:17\_PST\_2019;Cuda compilation tools, release 10.1, V10.1.105  
> CUDA compiler flags:-std=c++14;-gencode;arch=compute\_35,code=sm\_35;-gencode;arch=compute\_37,code=sm\_37;-gencode;arch=compute\_60,code=sm\_60;-gencode;arch=compute\_61,code=sm\_61;-gencode;arch=compute\_62,code=sm\_62;-use\_fast\_math;;-mavx2 -mfma -Wno-missing-field-initializers -fexcess-precision=fast -funroll-all-loops -pthread -fopenmp -O3 -DNDEBUG

When running:  
On GPU:

> ```
> The following tests FAILED:
> 12 - MdlibUnitTest (Timeout)
> 19 - DomDecMpiTests (Timeout)
> 20 - EwaldUnitTests (Timeout)
> 22 - GpuUtilsUnitTests (Timeout)
> 23 - HardwareUnitTests (Timeout)
> 53 - MdrunOutputTests (Timeout)
> 54 - MdrunModulesTests (Timeout)
> 55 - MdrunIOTests (Timeout)
> 56 - MdrunTests (Timeout)
> 57 - MdrunPmeTests (Timeout)
> 58 - MdrunNonIntegratorTests (Timeout)
> 59 - MdrunTpiTests (Timeout)
> 60 - MdrunMpiTests (Timeout)
> 61 - MdrunMpiPmeTests (Timeout)
> 62 - MdrunMpiCoordinationTestsOneRank (Timeout)
> 63 - MdrunMpiCoordinationTestsTwoRanks (Timeout)
> 64 - MdrunFEPTests (Timeout)
> 66 - GmxapiExternalInterfaceTests (Timeout)
> 67 - GmxapiInternalInterfaceTests (Timeout)
> 68 - regressiontests/complex (Timeout)
> 69 - regressiontests/freeenergy (Timeout)
> 70 - regressiontests/rotation (Timeout)
> 71 - regressiontests/essentialdynamics (Timeout)
> 
> ```

with `GMX_DISABLE_GPU_DETECTION=1 make check`

> The following tests FAILED:  
> 63 - MdrunMpiCoordinationTestsTwoRanks (Timeout)

I am running this on a shared (gridengine) HPC cluster so there will be other jobs running on the node at the same time, but I should be getting exclusive use of a core and a GPU. It almost looks as if the tests aren’t being passed onto the GPU.

Any ideas?

Thanks,  
MIke

---

<div class="post-metadata">

**Author:** ![mikewallis](https://avatars.discourse-cdn.com/v4/letter/m/f19dbf/32.png) [@mikewallis](https://gromacs.bioexcel.eu/u/mikewallis)\
**Post date:** [March 26, 2021, 4:08pm UTC](https://gromacs.bioexcel.eu/t/make-check-failing-when-gpu-enabled/1837/2 "2021-03-26T16:08:27Z")

</div>

Just for clarity, I’ve made sure that `CUDA_VISIBLE_DEVICES` is set but as it’s a shared machine with up to 8 GPUs it won’t necessarily be 0. In the example above it was 1.

---

<div class="post-metadata">

**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\
**Post date:** [March 29, 2021, 4:07pm UTC](https://gromacs.bioexcel.eu/t/make-check-failing-when-gpu-enabled/1837/3 "2021-03-29T16:07:17Z")

</div>

That the tests are timing out suggests that either the CPU or GPU you are using to execute the test is busy. Make sure that you are not using resourced that are already busy. Note that unless restricted, the test may try to use all cores / GPUs it detects, but if you set up the job correctly restricting the resources allocated and assuming your schedules is set up correctly, this should not happen.