GROMACS version: 2024
GROMACS modification: Yes, with plumed
We have found a troubling beaviour when running PT-WTE simulations: despite using identical system configurations, force fields, and biasing parameters, the exchange probabilities between replicas differ dramatically between the two GROMACS versions. As we used the same plumed version, this seems to be an issue with Gromacs. We have posted an issue to the plumed repository but we believe reporting it here can also be useful.
Exchange Probabilities (Biasing Phase)
GROMACS 2023:
Average exchange probability: 0.070 ± 0.010
Range: 0.05 - 0.09
All 30 inter-replica exchanges show consistently low acceptance rates
GROMACS 2024.3:
Average exchange probability: 0.314 ± 0.044
Range: 0.22 - 0.38
All 30 inter-replica exchanges show consistently high acceptance rates
All the files needed to reproduce the simulations are available in this Zenodo repository: https://doi.org/10.5281/zenodo.19203432.
The outputs of the 2023 and 2024 versions are also there, together with a jupyter notebook with more analysis and plots.
We believe that this issue is related to this one.
Are you using GPUs and is update running on GPU? In that case I think that Plumed will operate on outdated coordinates and result will be incorrect. How much incorrect depends on when the coordinates are transfered from GPU to CPU which can depend on the GROMACS version. This issue is not present in the Plumed GROMACS interface introduced in the 2025 release.
Yes, we are running on one GPU. I am not sure what you mean by update or how to check it. I enclose some parts of the output concerning the hardware at the end of the message.
When you say that this issue is not present in the new Plumed GROMACS interface, do you mean the version that comes with GROMACS without patching, as described here? But according to that, replica exchange is not yet available. And the other option, patching with plumed 2.10 in principle only works with version 2025.0. Should we try patching newer 2025 subversions? Or even 2026.1?
GROMACS: gmx mdrun, version 2024.2-plumed_2.9.2
Executable: /prod/cuda-12.4.1/gcc-13.3.0/openmpi-5.0.3/gromacs/2024.2/bin/gmx_mpi
Data prefix: /prod/cuda-12.4.1/gcc-13.3.0/openmpi-5.0.3/gromacs/2024.2
Working dir: /data/ucsqab/obarcenas/polyq-htt-helicity/md_folders/4_wt_original/WTE_plumed/bias_0
Process ID: 39642
Command line:
gmx_mpi mdrun -deffnm md -multidir bias_0 bias_1 bias_2 bias_3 bias_4 bias_5 bias_6 bias_7 bias_8 bias_9 bias_10 bias_11 bias_12 bias_13 bias_14 bias_15 bias_16 bias_17 bias_18 bias_19 bias_20 bias_21 bias_22 bias_23 bias_24 bias_25 bias_26 bias_27 bias_28 bias_29 bias_30 -replex 2000 -plumed plumed_PTWTE.dat
GROMACS version: 2024.2-plumed_2.9.2
Precision: mixed
Memory model: 64 bit
MPI library: MPI
MPI library version: Open MPI v4.1.8, package: Open MPI gomollon@login2 Distribution, ident: 4.1.8, repo rev: v4.1.8, Feb 04, 2025
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support: CUDA
NBNxM GPU setup: super-cluster 2x2x2 / cluster 8
SIMD instructions: AVX_512
CPU FFT library: fftw-3.3.8-sse2-avx-avx2-avx2_128-avx512
GPU FFT library: cuFFT
Multi-GPU FFT: none
RDTSCP usage: enabled
TNG support: enabled
Hwloc support: disabled
Tracing support: disabled
C compiler: /prod/gcc-13.3.0/gcc/bin/gcc GNU 13.3.0
C compiler flags: -fexcess-precision=fast -funroll-all-loops -march=skylake-avx512 -Wno-missing-field-initializers -pthread -O3 -DNDEBUG
C++ compiler: /prod/gcc-13.3.0/gcc/bin/g++ GNU 13.3.0
C++ compiler flags: -fexcess-precision=fast -funroll-all-loops -march=skylake-avx512 -Wno-missing-field-initializers -pthread -Wno-cast-function-type-strict SHELL:-fopenmp -O3 -DNDEBUG
BLAS library: Internal
LAPACK library: Internal
CUDA compiler: /prod/cuda-12.4.1/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2024 NVIDIA Corporation;Built on Thu_Mar_28_02:18:24_PDT_2024;Cuda compilation tools, release 12.4, V12.4.131;Build cuda_12.4.r12.4/compiler.34097967_0
CUDA compiler flags:-std=c++17;--generate-code=arch=compute_50,code=sm_50;--generate-code=arch=compute_52,code=sm_52;--generate-code=arch=compute_60,code=sm_60;--generate-code=arch=compute_61,code=sm_61;--generate-code=arch=compute_70,code=sm_70;--generate-code=arch=compute_75,code=sm_75;--generate-code=arch=compute_80,code=sm_80;--generate-code=arch=compute_86,code=sm_86;--generate-code=arch=compute_89,code=sm_89;--generate-code=arch=compute_90,code=sm_90;-Wno-deprecated-gpu-targets;--generate-code=arch=compute_53,code=sm_53;--generate-code=arch=compute_80,code=sm_80;-use_fast_math;-Xptxas;-warn-double-usage;-Xptxas;-Werror;-D_FORCE_INLINES;-Xcompiler;-fopenmp;-fexcess-precision=fast -funroll-all-loops -march=skylake-avx512 -Wno-missing-field-initializers -pthread -Wno-cast-function-type-strict SHELL:-fopenmp -O3 -DNDEBUG
CUDA driver: 12.60
CUDA runtime: 12.40
Running on 1 node with total 31 cores, 31 processing units, 1 compatible GPU
Hardware detected on host gpu24 (the node of MPI rank 0):
CPU info:
Vendor: AMD
Brand: AMD EPYC 9334 32-Core Processor
Family: 25 Model: 17 Stepping: 1
Features: aes amd apic avx avx2 avx512f avx512cd avx512bw avx512vl avx512bf16 avx512secondFMA clfsh cmov cx8 cx16 f16c fma htt lahf misalignsse mmx msr nonstop_tsc pcid pclmuldq pdpe1gb popcnt pse rdrnd rdtscp sha sse2 sse3 sse4a sse4.1 sse4.2 ssse3 x2apic
Hardware topology: Basic
Packages, cores, and logical processors:
[indices refer to OS logical processors]
Package 1: [ 32] [ 33] [ 34] [ 35] [ 36] [ 37] [ 38] [ 39] [ 40] [ 41] [ 42] [ 43] [ 44] [ 45] [ 46] [ 47] [ 48] [ 49] [ 50] [ 51] [ 52] [ 53] [ 54] [ 55] [ 56] [ 57] [ 58] [ 59] [ 60] [ 61] [ 62]
CPU limit set by OS: -1 Recommended max number of threads: 31
GPU info:
Number of GPUs detected: 1
#0: NVIDIA NVIDIA H100 PCIe, compute cap.: 9.0, ECC: yes, stat: compatible
