# Mdrun detects GPUs but no GPU utilization
**URL:** https://gromacs.bioexcel.eu/t/mdrun-detects-gpus-but-no-gpu-utilization/11860
**Category:** User discussions
**Tags:** mdrun, gpu
**Created:** [April 2, 2025, 8:45pm UTC](https://gromacs.bioexcel.eu/t/mdrun-detects-gpus-but-no-gpu-utilization/11860 "2025-04-02T20:45:19Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@sheppard](https://gromacs.bioexcel.eu/u/sheppard)
#### Post date: [April 2, 2025, 8:45pm UTC](https://gromacs.bioexcel.eu/t/mdrun-detects-gpus-but-no-gpu-utilization/11860/1 "2025-04-02T20:45:19Z")
GROMACS version: 2025.0
GROMACS modification: No
Here post your question
Hello everyone,
I’m been trying to run REMD simulations on a single-node GPU “cluster” recently purchased by my lab. I have a working gmx\_mpi executable:
```auto
gmx_mpi -version
GROMACS version: 2025.0
Precision: mixed
Memory model: 64 bit
MPI library: MPI
MPI library version: Open MPI v5.0.6, package: Open MPI sheppard@sn4622123733 Distribution, ident: 5.0.6, repo rev: v5.0.6, Nov 15, 2024
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support: CUDA
NBNxM GPU setup: super-cluster 2x2x2 / cluster 8 (cluster-pair splitting on)
SIMD instructions: AVX_512
CPU FFT library: fftw-3.3.10-sse2-avx-avx2-avx2_128-avx512
GPU FFT library: cuFFT
Multi-GPU FFT: none
RDTSCP usage: enabled
TNG support: enabled
Hwloc support: disabled
Tracing support: disabled
C compiler: /usr/bin/cc GNU 11.5.0
C compiler flags: -fexcess-precision=fast -funroll-all-loops -march=skylake-avx512 -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler: /usr/bin/c++ GNU 11.5.0
C++ compiler flags: -fexcess-precision=fast -funroll-all-loops -march=skylake-avx512 -Wno-missing-field-initializers -Wno-old-style-cast -Wno-cast-qual -Wno-suggest-override -Wno-suggest-destructor-override -Wno-zero-as-null-pointer-constant -Wno-cast-function-type-strict SHELL:-fopenmp -O3 -DNDEBUG
BLAS library: Internal
LAPACK library: Internal
CUDA compiler: /home/sheppard/miniconda3/envs/cuda_env/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2024 NVIDIA Corporation;Built on Thu_Mar_28_02:18:24_PDT_2024;Cuda compilation tools, release 12.4, V12.4.131;Build cuda_12.4.r12.4/compiler.34097967_0
CUDA compiler flags: -O3 -DNDEBUG
CUDA driver: 12.70
CUDA runtime: 12.40
```
But my REMD simulations with 64 replicas are running very slow - roughly 5 ns/day when I would have expected at least 50 ns/day. I have tried to force bonded/nonbonded and PME calculations onto the GPUs using:
```auto
mpirun -np $SLURM_NTASKS gmx_mpi mdrun -v -multidir <64 replicas> -replex 1500 -dlb yes -nb gpu -bonded gpu -pme gpu -pmefft gpu
```
And I do see the following in one of the replica’s log file:
```auto
Running on 1 node with total 104 cores, 208 processing units, 4 compatible GPUs
Hardware detected on host cs.ucsb.edu (the node of MPI rank 0):
GPU info:
Number of GPUs detected: 4
#0: NVIDIA NVIDIA H200 MIG 1g.18gb, compute cap.: 9.0, ECC: yes, stat: compatible
#1: NVIDIA NVIDIA H200 MIG 1g.18gb, compute cap.: 9.0, ECC: yes, stat: compatible
#2: NVIDIA NVIDIA H200 MIG 1g.18gb, compute cap.: 9.0, ECC: yes, stat: compatible
#3: NVIDIA NVIDIA H200 MIG 1g.18gb, compute cap.: 9.0, ECC: yes, stat: compatible
```
And a grep for `PME`:
```auto
$ grep -i PME md.log
coulombtype = PME
pme-order = 4
lj-pme-comb-rule = Geometric
PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3
PME tasks will do all aspects on the GPU
Will do PME sum in reciprocal space for electrostatic interactions.
step 600: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3062.6 M-cycles
step 800: timed with pme grid 60 60 60, coulomb cutoff 1.350: 3572.5 M-cycles
step 1000: timed with pme grid 64 64 64, coulomb cutoff 1.266: 3665.3 M-cycles
step 1200: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3208.5 M-cycles
optimal pme grid 72 72 72, coulomb cutoff 1.200
PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3
PME tasks will do all aspects on the GPU
Will do PME sum in reciprocal space for electrostatic interactions.
step 4889900: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3804.6 M-cycles
step 4890100: timed with pme grid 60 60 60, coulomb cutoff 1.350: 4386.0 M-cycles
step 4890300: timed with pme grid 64 64 64, coulomb cutoff 1.266: 4930.7 M-cycles
step 4890500: timed with pme grid 72 72 72, coulomb cutoff 1.200: 4243.6 M-cycles
optimal pme grid 72 72 72, coulomb cutoff 1.200
PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3
PME tasks will do all aspects on the GPU
Will do PME sum in reciprocal space for electrostatic interactions.
step 9975100: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3882.0 M-cycles
step 9975300: timed with pme grid 60 60 60, coulomb cutoff 1.350: 4269.5 M-cycles
step 9975500: timed with pme grid 52 52 52, coulomb cutoff 1.558: 7345.5 M-cycles
step 9975700: timed with pme grid 56 56 56, coulomb cutoff 1.447: 3737.7 M-cycles
step 9975900: timed with pme grid 64 64 64, coulomb cutoff 1.266: 3896.2 M-cycles
step 9976100: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3512.4 M-cycles
step 9976300: timed with pme grid 56 56 56, coulomb cutoff 1.447: 3871.3 M-cycles
step 9976500: timed with pme grid 64 64 64, coulomb cutoff 1.266: 3566.4 M-cycles
step 9976700: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3386.8 M-cycles
step 9976900: timed with pme grid 56 56 56, coulomb cutoff 1.447: 3864.3 M-cycles
step 9977100: timed with pme grid 64 64 64, coulomb cutoff 1.266: 3583.7 M-cycles
step 9977300: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3538.1 M-cycles
optimal pme grid 72 72 72, coulomb cutoff 1.200
PME GPU mesh 1 3 15025501 95917.664 575521.615 36.4
Breakdown of PME mesh activities
Wait PME GPU gather 1 3 15025501 78961.413 473781.347 30.0
Reduce GPU PME F 1 3 15025501 2149.764 12898.935 0.8
Launch PME GPU ops. 1 3 120204021 951.779 5710.831 0.4
```
So GROMACS appears to “see” the GPUs, and it think it is running on them. However, if I run `nvidia-smi` on the node running my job, I see “No running processes found”. Additionally, `top` reveals roughly 100% CPU for each of my replicas. And finally, `dcgmi` shows:
```auto
$ dcgmi dmon -e 100,203,252
#Entity SMCLK GPUTL FBUSD ID GPU 3 1980 0 63494 GPU 2 1980 N/A 48950 GPU 1 1980 N/A 32088
GPU 0 1980 N/A 9329
GPU 3 1980 0 63494
GPU 2 1980 N/A 48950
GPU 1 1980 N/A 32088
GPU 0 1980 N/A 9329
```
This all makes me think the GPUs are not getting used, which is causing the slow run times. Has anyone come across issues like this?
Please let me know if I can provide any follow-up details, or if you believe I have misinterpreted the logs above.
Thank you all so much!
Jackson
---
### Author:  [@hess](https://gromacs.bioexcel.eu/u/hess)
#### Post date: [April 10, 2025, 10:59am UTC](https://gromacs.bioexcel.eu/t/mdrun-detects-gpus-but-no-gpu-utilization/11860/2 "2025-04-10T10:59:10Z")
You are using an enormous amount of MPI ranks, I guess one per core. You should use as many ranks as there are replicas. I don’t know if that is the only performance issue though.
---
### Author:  [@sheppard](https://gromacs.bioexcel.eu/u/sheppard)
#### Post date: [April 10, 2025, 5:47pm UTC](https://gromacs.bioexcel.eu/t/mdrun-detects-gpus-but-no-gpu-utilization/11860/3 "2025-04-10T17:47:01Z")
Thank you for your insight. My understanding was that I was indeed requesting 64 MPI ranks, one for each of my replicas. My SLURM directives are:
```auto
#SBATCH -o 07REMD_50-100ns/output.o%j
#SBATCH -e 07REMD_50-100ns/error.e%j
#SBATCH --gres=gpu:h200-1g.18gb:4
#SBATCH --partition=styx # Queue (partition) name
#SBATCH --ntasks=64 # Total number of mpi tasks requested
#SBATCH --cpus-per-task=2
#SBATCH -t 80:00:00 # Run time (hh:mm:ss)
```
Do my `--cpus-per-task` double the number of MPI ranks? Or does the long list of PP:0, PME:0, … indicate a large number of ranks? I think I got a few log files mixed up, but for my 64-replica simulations I’m seeing:
```auto
Mapping of GPU IDs to the 128 GPU tasks in the 64 ranks on this node:
```
And the list that follows (`PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,...`) does indeed contain 128 elements.
Would it be better to keep --cpus-per-task=1?
Thank you again, and please let me know if I can add any further detail.
---
### Author:  [@hess](https://gromacs.bioexcel.eu/u/hess)
#### Post date: [April 11, 2025, 7:15am UTC](https://gromacs.bioexcel.eu/t/mdrun-detects-gpus-but-no-gpu-utilization/11860/4 "2025-04-11T07:15:50Z")
I realize now that the PP PME assignment line probably shows the assignment for all replicas. Then I don’t know what the issue is. Can you attach the timing tables that appear at the end of a log file?