MDrun: Simulation speed jumping a factor of 10 with a gpu executable

[zaidi1]

GROMACS version: 2025.3
GROMACS modification: No

My simulations on a GPU partition in our HPC cluster are behaving erratically. I see a sudden drop in my simulation speed by a factor of 10, where separate runs fluctuate between 800 ns/day and 20 ns/day.
Interestingly, the times used in the compute kernel remain the same and account for only a fraction of the total time. I noticed that in the slow jobs, it was the “REST” of the workload that took all the time, literally, and all other components had the same wall time. I am pasting snippets of my log file below from different runs.

A. Slow run:

gmx --quiet mdrun -ntmpi 1 -ntomp 36 -v -deffnm ramp -pin on -pinstride 0 -nb gpu -bonded gpu -update gpu

B. Fast run:

gmx --quiet mdrun -ntmpi 1 -ntomp 36 -v -deffnm ramp -pin on -pinstride 0 -nb gpu -bonded gpu -update gpu

My built gmx executable is as follows:

GROMACS version: 2025.3
Precision: mixed
Memory model: 64 bit
MPI library: thread_mpi
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support: CUDA
NBNxM GPU setup: super-cluster 2x2x2 / cluster 8 (cluster-pair splitting on)
SIMD instructions: ARM_SVE
CPU FFT library: fftw-3.3.10
GPU FFT library: cuFFT
Multi-GPU FFT: none
TNG support: enabled
Hwloc support: disabled
Tracing support: disabled
C compiler: /usr/bin/cc GNU 11.5.0
C compiler flags: -fexcess-precision=fast -funroll-all-loops -march=armv8.2-a+sve -msve-vector-bits=128 -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler: /usr/bin/c++ GNU 11.5.0
C++ compiler flags: -fexcess-precision=fast -funroll-all-loops -march=armv8.2-a+sve -msve-vector-bits=128 -Wno-missing-field-initializers -Wno-cast-function-type-strict SHELL:-fopenmp -O3 -DNDEBUG
BLAS library: Internal
LAPACK library: Internal
CUDA compiler: /share/apps/eb3/software/aarch64/CUDA/13.2.0/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2026 NVIDIA Corporation;Built on Mon_Mar_02_09:52:54_PM_PST_2026;Cuda compilation tools, release 13.2, V13.2.51;Build cuda_13.2.r13.2/compiler.37434383_0
CUDA compiler flags: -O3 -DNDEBUG
CUDA driver: 13.0
CUDA runtime: 13.20

I have no clue what is going on in the background. I just need a hint on where to look for the solution. Any idea, or have any users ever faced this problem? I checked with other GMX builds from the 2024 version; the results are the same.

[zaidi1]

Or can anybody help me understand what the “Rest” in the real-time log section means? So that I can resolve the problem. I look forward to your guidance @hess @jalemkul

[hess]

Rest should only contain minor stuff that is not worth timing separately. My guess is that mdrun is waiting for something on of these “small” tasks. This is difficult to diagnose.

Can you reproduce this with much shorter simulations? If you can, maybe you can run interactive with the -v option and see if mdrun is continuously slow or hangs for a while somewhere (maybe at the end of the simulation)?

Another thing which might tell something is when state_prev.cpt was written with respect to the start and end time of the simulation and at which MD step. Use gmx dump -cp to extract the step number.

[zaidi1]

Dear Dr. Hess, thank you for your comments.

The data from a smaller run (125 ps) is as follows:

Mdrun start: 08:44:58 2026

prev cpt written on step 103200 at 09:00:00 2026

Final checkpoint written on step 125000 at 09:05:23 2026

Simulation ends at 09:05:51 2026

I also recorded the time of the xtc update in an extension of the above run:

2026-08-24 11:19:52.717440921 +0200 573228
2026-08-24 11:26:19.539988942 +0200 620944
2026-08-24 11:33:12.082706327 +0200 668656

Here, 200 ps took over 14 minutes, giving ~ 20 ns/day simulation speed.

The system is just chamrmm36 decane in a 125 nm^3 cubic box with 25 Nitrogen molecules and a long-range PME solver for Coulomb and dispersion forces

It seems that the problem is not at the end. Is this problem related to CPU-GPU or CPU-CPU synchronization? I am also trying with different numbers of ranks and threads. What could these small tasks be?

[hess]

Is this using a node exclusively or sharing resources with other jobs? If there are other jobs they might interfere.

If you can run interactively, you can run in a debugger and interrupt now and then to find out where the “Rest” time is coming from.

[zaidi1]

Dear Dr. Hess,

I am sorry that I missed mentioning that all a job run exclusively on a single gpu node without sharing.

Sure, I will run interactively and try to debugg this situation.

[hess]

Great! Please report back.

You might need to build RelWithDebInfo to obtain sufficient information.