Cuda error gromacs with plumed

User discussions
1

GROMACS version: 2022.3
GROMACS modification: No

Dear all,
in running a metadynamics simulation with gromacs 2022.3 and plumed 2.8 on a workstation with rtx 3090ti (CUDA 11.8) and intel xeon dual core (2 x 20 cores), I get the following error:

Program: gmx mdrun, version 2022.3-plumed_2.8.1
Source file: src/gromacs/gpu_utils/devicebuffer.cuh (line 91)
Function: freeDeviceBuffer<float2*>(float2**)::<lambda()>
MPI rank: 4 (out of 8)

Assertion failed:
Condition: stat == cudaSuccess
Source file: src/gromacs/gpu_utils/devicebuffer.cuh (line 91)
Function: freeDeviceBuffer<float2*>(float2**)::<lambda()>
Freeing of the device buffer failed. CUDA error #1 (cudaErrorInvalidValue):
invalid argument.

For more information and tips for troubleshooting, please check the GROMACS
website at Common Errors — GROMACS webpage https://www.gromacs.org documentation

MPI rank: 5 (out of 8)

Assertion failed:
Condition: stat == cudaSuccess
Freeing of the device buffer failed. CUDA error #1 (cudaErrorInvalidValue):

The simulation runs without error if I use gromacs compiled with MPI, but I tried to use a non MPI version of gromacs to get better performance and I got the error above.
The simulation was started with the following command:

gmx mdrun -deffnm MES_6_complex_metad -plumed plumed.dat -bonded gpu -nb gpu -pme gpu -ntmpi 8 -ntomp 5 -npme 1 -gputasks 00000000

Compilation was done with g++ 11.3 on ubuntu 22.04.

The version of non MPI gromacs works fine without plumed.

Thanks in advance
Stefano

2

Hello,

I’m facing exactly the same problem.
Did you say that the MPI version runs with plumed, right? Could you please print the cmake command you used?

Best,

3

Hello,
configuration was made with

cmake … -DGMX_BUILD_OWN_FFTW=ON -DGMX_GPU=CUDA -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda -DCMAKE_INSTALL_PREFIX=/home/stefano/gromacs2022_mpi -DGMX_SIMD=AVX2_256 -DGMX_MPI=on

after adding to PATH the path to mpi executables (I have tried both with OpenMPI or MPICH).

Best

5

Hi, I solved it downgrading cuda (v. 10.2) and nvidia-driver (v. 440).
Then, I’ve installed gromacs 2019.6.
Both plumed (v. 2.8.1) and gromacs were compiled without mpi.

Unfortunately, this was only way I’ve found.
I hope they will fix this problem soon. If you do so, please let me know.

Best,

6

Hi,
thanks for the feedback, I’m glad you solved; I posted the issue to PLUMED forum as well, so far I did not get response.

In the meantime I turned to Gromacs 2021 patched with Colvars module, as I have to run metadynamics, which is implementd in the module.
For the cent my opinion is worth, I have noticed that the colvars patched version of gromacs, beside being smoothier to install (nothing different from installing gromacs per se), runs with exactly the same performance of “native” gromacs, while I have always noticed a loss of performance with PLUMED (maybe this is my blame and is due to non optimal compilation of plumed).

All the best