# Cannot figure out why gmx\_mpi cannot detect nvidia GPU **URL:** \ **Category:** User discussions\ **Created:** [November 14, 2024, 9:59am UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551 "2024-11-14T09:59:21Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ansel0250](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/ansel0250/32/3971_2.png) [@ansel0250](https://gromacs.bioexcel.eu/u/ansel0250)\ **Post date:** [November 14, 2024, 9:59am UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/1 "2024-11-14T09:59:21Z")
GROMACS version: 2024.4 GROMACS modification: No I had been compiling the gromacs for a while and use CUDA 12.5 and the card can be detected using nvidia-smi I have also checked the gmx --version to see if the CUDA driver and runtime is present in the gmx GROMACS version: 2024.4 Precision: mixed Memory model: 64 bit MPI library: MPI MPI library version: Intel(R) MPI Library 2021.9 for Linux\* OS OpenMP support: enabled (GMX\_OPENMP\_MAX\_THREADS = 128) GPU support: CUDA NBNxM GPU setup: super-cluster 2x2x2 / cluster 8 SIMD instructions: AVX2\_256 CPU FFT library: fftw-3.3.8-sse2-avx-avx2-avx2\_128 GPU FFT library: cuFFT Multi-GPU FFT: none RDTSCP usage: enabled TNG support: enabled Hwloc support: disabled Tracing support: disabled C compiler: /bin/cc GNU 11.4.1 C compiler flags: -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -O3 -DNDEBUG C++ compiler: /bin/c++ GNU 11.4.1 C++ compiler flags: -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp -O3 -DNDEBUG BLAS library: External - detected on the system LAPACK library: External - detected on the system CUDA compiler: /usr/local/cuda-12.5/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2024 NVIDIA Corporation;Built on Thu\_Jun\_\_6\_02:18:23\_PDT\_2024;Cuda compilation tools, release 12.5, V12.5.82;Build cuda\_12.5.r12.5/compiler.34385749\_0 CUDA compiler flags:-std=c++17;–generate-code=arch=compute\_89,code=sm\_89;-use\_fast\_math;-Xptxas;-warn-double-usage;-Xptxas;-Werror;-D\_FORCE\_INLINES;-Xcompiler;-fopenmp;-fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp -O3 -DNDEBUG CUDA driver: 12.50 CUDA runtime: 12.50 strangely the GMX cannot detect the GPU as it says in the log file Running on 1 node with total 24 cores, 32 processing units (GPU detection failed) Hardware detected on host gpu5 (the node of MPI rank 0): CPU info: Vendor: Intel Brand: 13th Gen Intel(R) Core™ i9-13900K Family: 6 Model: 183 Stepping: 1 Features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt intel lahf mmx msr nonstop\_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sha sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic Hardware topology: Basic Packages, cores, and logical processors: [indices refer to OS logical processors] Package 0: [0 1] [2 3] [4 5] [6 7] [8 9] [10 11] [12 13] [14 15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] CPU limit set by OS: -1 Recommended max number of threads: 32 the compilation is using intel oneAPI with with-gpu option , with-mpi option. I guess I need some help here. ---
**Author:** ![MagnusL](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/magnusl/32/2380_2.png) [@MagnusL](https://gromacs.bioexcel.eu/u/MagnusL)\ **Post date:** [November 15, 2024, 6:27am UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/2 "2024-11-15T06:27:52Z")
Is this via a queuing system/workload management system, such as slurm? Have you requested any GPUs (the slurm `-G` option)? ---
**Author:** ![ansel0250](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/ansel0250/32/3971_2.png) [@ansel0250](https://gromacs.bioexcel.eu/u/ansel0250)\ **Post date:** [November 17, 2024, 2:55pm UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/3 "2024-11-17T14:55:20Z")
there is SLURM and we have also applied -gres option into that , the main thing is the RTX4090 cannot be found in the system. but other software itself like LAMMPS could accelerate the calculation. for debugging purpose , the mpirun has also been run and the gmx\_mpi still cannot find the GPU, -nb cpu is fine for calculation. ---
**Author:** ![MagnusL](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/magnusl/32/2380_2.png) [@MagnusL](https://gromacs.bioexcel.eu/u/MagnusL)\ **Post date:** [November 18, 2024, 7:40am UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/4 "2024-11-18T07:40:12Z")
I’m afraid I don’t know what might be the problem then. @pszilard or @al42and, do you have any ideas? ---
**Author:** ![al42and](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/al42and/32/1393_2.png) [@al42and](https://gromacs.bioexcel.eu/u/al42and)\ **Post date:** [November 18, 2024, 11:15am UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/5 "2024-11-18T11:15:28Z")
Could you please share the `md.log` file from a problematic run (feel free to redact the usernames etc if you want)? ---
**Author:** ![ansel0250](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/ansel0250/32/3971_2.png) [@ansel0250](https://gromacs.bioexcel.eu/u/ansel0250)\ **Post date:** [November 18, 2024, 4:25pm UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/6 "2024-11-18T16:25:40Z")
[log.txt](https://gromacs.bioexcel.eu/uploads/short-url/mOM3NCza9wowmMcZtwrYo7c2Bfq.txt) (14.7 KB) How about the log here? ---
**Author:** ![al42and](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/al42and/32/1393_2.png) [@al42and](https://gromacs.bioexcel.eu/u/al42and)\ **Post date:** [November 20, 2024, 12:03pm UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/7 "2024-11-20T12:03:55Z")
Ok, thanks for sharing. Nothing wrong seen in the log file. As Magnus said, such behavior (device visible with `nvidia-smi`, driver detected by GROMACS, but still getting “GPU detection failed” message) usually means that `CUDA_VISIBLE_DEVICES` is not set correctly (either by the batch system or by the MPI library, although, as far as I recall, IntelMPI does not limit the visibility of NVIDIA devices). Could you try adding the lines below to your batch script right before `gmx_mpi mdrun` is called, and share the output? ```auto echo "CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES-unset}" env | grep '^I_MPI_OFFLOAD' export I_MPI_OFFLOAD_DEVICES=all export I_MPI_DEBUG=3 ``` ---
**Author:** ![ansel0250](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/ansel0250/32/3971_2.png) [@ansel0250](https://gromacs.bioexcel.eu/u/ansel0250)\ **Post date:** [November 29, 2024, 6:48am UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/8 "2024-11-29T06:48:08Z")
[ss.txt](https://gromacs.bioexcel.eu/uploads/short-url/7KU2eXYp89VQnt3oozcK5tzxSIr.txt) (17.1 KB) Please find the log after applying CUDA visible devices as ALL , IMPI offload as all , turned on verbose 3 of IMPI debug of the calculation (mdrun -nb gpu) ---
**Author:** ![al42and](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/al42and/32/1393_2.png) [@al42and](https://gromacs.bioexcel.eu/u/al42and)\ **Post date:** [December 2, 2024, 5:30pm UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/9 "2024-12-02T17:30:18Z")
Thanks. Quite strange. Could you try compiling this simple device detection utility in a similar SLURM session on the GPU node and see what it prints? ```console $ wget https://raw.githubusercontent.com/al42and/cuda-smi/refs/heads/master/cuda-smi.cpp $ /usr/local/cuda-12.5/bin/nvcc cuda-smi.cpp -lcudart_static -lnvidia-ml -o cuda-smi $ ./cuda-smi $ mpirun --perhost 1 ./cuda-smi ``` ---
**Author:** ![ansel0250](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/ansel0250/32/3971_2.png) [@ansel0250](https://gromacs.bioexcel.eu/u/ansel0250)\ **Post date:** [December 5, 2024, 9:27am UTC](https://gromacs.bioexcel.eu/t/cannot-figure-out-why-gmx-mpi-cannot-detect-nvidia-gpu/10551/10 "2024-12-05T09:27:48Z")
OK , after looking the cuda-smi , we found out the CUDAquery function isn’t functioning and we further dig in the process using modinfo , re-install the RPM of CUDA. but it turns out the chroot environment of the deployment node images can’t build the nvidia\_uvm etc modules and we discovered that problem (and remedy) in a month ago , undocumented , seems not discussed in nvidia forum also Now it is functional :)