How to use 2 GPUs at the same time for one simulation

User discussions
1

GROMACS version:2022.4
GROMACS modification: No
Hello, I would like to know how to use 2 GPUs at the same time for a simulation. I have a system with 2 RTX 4090s. But it seems like I can only use one gpu to run a simulation. I have tried adding -gpu_id 01 but still it does not work. I am using the Gromacs on Ubuntu.

2

You need to use multiple MPI ranks. I assume you have not compiled with an MPI library, so you need to use the -ntmpi option. There are two sensible ways of dividing the work. One is running PME on the GPU, where only rank can is used for PME:
mdrun -ntmpi N -npme 1 -pme gpu
Try N=2, 3 and 4 and see what it fastest.
If you have a fast CPU, you can also run PME on the CPU:
mdrun -ntmpi N
Try N=2 and 4 and see what it fastest.

3

Hi,

I suggest to try run with GPU direct communication
export GMX_ENABLE_DIRECT_GPU_COMM=1
combined with full GPU offload
-pme gpu -npme 1 -update gpu -bonded gpu
combined with oversubscription (as Berk mentions above).

E.g. something like

export GMX_ENABLE_DIRECT_GPU_COMM=1
gmx mdrun -ntmpi 8 -ntomp 4 -pin on -v -nb gpu -pme gpu -npme 1-update gpu -bonded gpu -noconfout -nsteps 100000 -resetstep 90000 -nstlist 300 -gpu_id 01

This can allow much higher performance than default settings.

If you get an error message regarding GPU update not being supported, try setting “constraints = h-bonds” in your .mdp file, as discussed at Getting good performance from mdrun — GROMACS 2022.4 documentation