Mdrun detects GPUs but no GPU utilization

User discussions
,
1

GROMACS version: 2025.0
GROMACS modification: No
Here post your question

Hello everyone,

I’m been trying to run REMD simulations on a single-node GPU “cluster” recently purchased by my lab. I have a working gmx_mpi executable:

  gmx_mpi -version

GROMACS version:     2025.0
Precision:           mixed
Memory model:        64 bit
MPI library:         MPI
MPI library version: Open MPI v5.0.6, package: Open MPI sheppard@sn4622123733 Distribution, ident: 5.0.6, repo rev: v5.0.6, Nov 15, 2024
OpenMP support:      enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support:         CUDA
NBNxM GPU setup:     super-cluster 2x2x2 / cluster 8 (cluster-pair splitting on)
SIMD instructions:   AVX_512
CPU FFT library:     fftw-3.3.10-sse2-avx-avx2-avx2_128-avx512
GPU FFT library:     cuFFT
Multi-GPU FFT:       none
RDTSCP usage:        enabled
TNG support:         enabled
Hwloc support:       disabled
Tracing support:     disabled
C compiler:          /usr/bin/cc GNU 11.5.0
C compiler flags:    -fexcess-precision=fast -funroll-all-loops -march=skylake-avx512 -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler:        /usr/bin/c++ GNU 11.5.0
C++ compiler flags:  -fexcess-precision=fast -funroll-all-loops -march=skylake-avx512 -Wno-missing-field-initializers -Wno-old-style-cast -Wno-cast-qual -Wno-suggest-override -Wno-suggest-destructor-override -Wno-zero-as-null-pointer-constant -Wno-cast-function-type-strict SHELL:-fopenmp -O3 -DNDEBUG
BLAS library:        Internal
LAPACK library:      Internal
CUDA compiler:       /home/sheppard/miniconda3/envs/cuda_env/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2024 NVIDIA Corporation;Built on Thu_Mar_28_02:18:24_PDT_2024;Cuda compilation tools, release 12.4, V12.4.131;Build cuda_12.4.r12.4/compiler.34097967_0
CUDA compiler flags: -O3 -DNDEBUG
CUDA driver:         12.70
CUDA runtime:        12.40

But my REMD simulations with 64 replicas are running very slow - roughly 5 ns/day when I would have expected at least 50 ns/day. I have tried to force bonded/nonbonded and PME calculations onto the GPUs using:

mpirun -np $SLURM_NTASKS gmx_mpi mdrun -v -multidir <64 replicas> -replex 1500 -dlb yes -nb gpu -bonded gpu -pme gpu -pmefft gpu

And I do see the following in one of the replica’s log file:

Running on 1 node with total 104 cores, 208 processing units, 4 compatible GPUs
Hardware detected on host cs.ucsb.edu (the node of MPI rank 0):
  GPU info:
    Number of GPUs detected: 4
    #0: NVIDIA NVIDIA H200 MIG 1g.18gb, compute cap.: 9.0, ECC: yes, stat: compatible
    #1: NVIDIA NVIDIA H200 MIG 1g.18gb, compute cap.: 9.0, ECC: yes, stat: compatible
    #2: NVIDIA NVIDIA H200 MIG 1g.18gb, compute cap.: 9.0, ECC: yes, stat: compatible
    #3: NVIDIA NVIDIA H200 MIG 1g.18gb, compute cap.: 9.0, ECC: yes, stat: compatible

And a grep for PME:

$ grep -i PME md.log
   coulombtype                    = PME
   pme-order                      = 4
   lj-pme-comb-rule               = Geometric
  PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3
PME tasks will do all aspects on the GPU
Will do PME sum in reciprocal space for electrostatic interactions.
step  600: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3062.6 M-cycles
step  800: timed with pme grid 60 60 60, coulomb cutoff 1.350: 3572.5 M-cycles
step 1000: timed with pme grid 64 64 64, coulomb cutoff 1.266: 3665.3 M-cycles
step 1200: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3208.5 M-cycles
              optimal pme grid 72 72 72, coulomb cutoff 1.200
  PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3
PME tasks will do all aspects on the GPU
Will do PME sum in reciprocal space for electrostatic interactions.
step 4889900: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3804.6 M-cycles
step 4890100: timed with pme grid 60 60 60, coulomb cutoff 1.350: 4386.0 M-cycles
step 4890300: timed with pme grid 64 64 64, coulomb cutoff 1.266: 4930.7 M-cycles
step 4890500: timed with pme grid 72 72 72, coulomb cutoff 1.200: 4243.6 M-cycles
              optimal pme grid 72 72 72, coulomb cutoff 1.200
  PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:1,PME:1,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:2,PME:2,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3,PP:3,PME:3
PME tasks will do all aspects on the GPU
Will do PME sum in reciprocal space for electrostatic interactions.
step 9975100: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3882.0 M-cycles
step 9975300: timed with pme grid 60 60 60, coulomb cutoff 1.350: 4269.5 M-cycles
step 9975500: timed with pme grid 52 52 52, coulomb cutoff 1.558: 7345.5 M-cycles
step 9975700: timed with pme grid 56 56 56, coulomb cutoff 1.447: 3737.7 M-cycles
step 9975900: timed with pme grid 64 64 64, coulomb cutoff 1.266: 3896.2 M-cycles
step 9976100: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3512.4 M-cycles
step 9976300: timed with pme grid 56 56 56, coulomb cutoff 1.447: 3871.3 M-cycles
step 9976500: timed with pme grid 64 64 64, coulomb cutoff 1.266: 3566.4 M-cycles
step 9976700: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3386.8 M-cycles
step 9976900: timed with pme grid 56 56 56, coulomb cutoff 1.447: 3864.3 M-cycles
step 9977100: timed with pme grid 64 64 64, coulomb cutoff 1.266: 3583.7 M-cycles
step 9977300: timed with pme grid 72 72 72, coulomb cutoff 1.200: 3538.1 M-cycles
              optimal pme grid 72 72 72, coulomb cutoff 1.200
 PME GPU mesh              1    3   15025501   95917.664     575521.615  36.4
 Breakdown of PME mesh activities
 Wait PME GPU gather       1    3   15025501   78961.413     473781.347  30.0
 Reduce GPU PME F          1    3   15025501    2149.764      12898.935   0.8
 Launch PME GPU ops.       1    3  120204021     951.779       5710.831   0.4

So GROMACS appears to “see” the GPUs, and it think it is running on them. However, if I run nvidia-smi on the node running my job, I see “No running processes found”. Additionally, top reveals roughly 100% CPU for each of my replicas. And finally, dcgmi shows:

$ dcgmi dmon -e 100,203,252                                                                                                   
#Entity   SMCLK        GPUTL             FBUSD                                                                                                               ID                                                                                                                                                           GPU 3     1980         0                 63494                                                                                                               GPU 2     1980         N/A               48950                                                                                                               GPU 1     1980         N/A               32088                                                                                                               
GPU 0     1980         N/A               9329                                                                                                                
GPU 3     1980         0                 63494                                                                                                               
GPU 2     1980         N/A               48950                                                                                                               
GPU 1     1980         N/A               32088                                                                                                               
GPU 0     1980         N/A               9329

This all makes me think the GPUs are not getting used, which is causing the slow run times. Has anyone come across issues like this?

Please let me know if I can provide any follow-up details, or if you believe I have misinterpreted the logs above.

Thank you all so much!
Jackson

2

You are using an enormous amount of MPI ranks, I guess one per core. You should use as many ranks as there are replicas. I don’t know if that is the only performance issue though.

3

Thank you for your insight. My understanding was that I was indeed requesting 64 MPI ranks, one for each of my replicas. My SLURM directives are:

#SBATCH -o 07REMD_50-100ns/output.o%j
#SBATCH -e 07REMD_50-100ns/error.e%j
#SBATCH --gres=gpu:h200-1g.18gb:4
#SBATCH --partition=styx  # Queue (partition) name
#SBATCH --ntasks=64           # Total number of mpi tasks requested
#SBATCH --cpus-per-task=2
#SBATCH -t 80:00:00         # Run time (hh:mm:ss)

Do my --cpus-per-task double the number of MPI ranks? Or does the long list of PP:0, PME:0, … indicate a large number of ranks? I think I got a few log files mixed up, but for my 64-replica simulations I’m seeing:

Mapping of GPU IDs to the 128 GPU tasks in the 64 ranks on this node:

And the list that follows (PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,PME:0,PP:0,...) does indeed contain 128 elements.

Would it be better to keep --cpus-per-task=1?

Thank you again, and please let me know if I can add any further detail.

4

I realize now that the PP PME assignment line probably shows the assignment for all replicas. Then I don’t know what the issue is. Can you attach the timing tables that appear at the end of a log file?