# How to use 2 GPUs at the same time for one simulation

**URL:** <https://gromacs.bioexcel.eu/t/how-to-use-2-gpus-at-the-same-time-for-one-simulation/5481>\
**Category:** User discussions\
**Tags:** gpu\
**Created:** [December 25, 2022, 10:13pm UTC](https://gromacs.bioexcel.eu/t/how-to-use-2-gpus-at-the-same-time-for-one-simulation/5481 "2022-12-25T22:13:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mdsim](https://avatars.discourse-cdn.com/v4/letter/m/a587f6/32.png) [@mdsim](https://gromacs.bioexcel.eu/u/mdsim)\
**Post date:** [December 25, 2022, 10:13pm UTC](https://gromacs.bioexcel.eu/t/how-to-use-2-gpus-at-the-same-time-for-one-simulation/5481/1 "2022-12-25T22:13:28Z")

</div>

GROMACS version:2022.4  
GROMACS modification: No  
Hello, I would like to know how to use 2 GPUs at the same time for a simulation. I have a system with 2 RTX 4090s. But it seems like I can only use one gpu to run a simulation. I have tried adding -gpu\_id 01 but still it does not work. I am using the Gromacs on Ubuntu.

---

<div class="post-metadata">

**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\
**Post date:** [January 10, 2023, 2:18pm UTC](https://gromacs.bioexcel.eu/t/how-to-use-2-gpus-at-the-same-time-for-one-simulation/5481/2 "2023-01-10T14:18:35Z")

</div>

You need to use multiple MPI ranks. I assume you have not compiled with an MPI library, so you need to use the -ntmpi option. There are two sensible ways of dividing the work. One is running PME on the GPU, where only rank can is used for PME:  
mdrun -ntmpi N -npme 1 -pme gpu  
Try N=2, 3 and 4 and see what it fastest.  
If you have a fast CPU, you can also run PME on the CPU:  
mdrun -ntmpi N  
Try N=2 and 4 and see what it fastest.

---

<div class="post-metadata">

**Author:** ![alang](https://avatars.discourse-cdn.com/v4/letter/a/22d042/32.png) [@alang](https://gromacs.bioexcel.eu/u/alang)\
**Post date:** [January 13, 2023, 3:16pm UTC](https://gromacs.bioexcel.eu/t/how-to-use-2-gpus-at-the-same-time-for-one-simulation/5481/3 "2023-01-13T15:16:57Z")

</div>

Hi,

I suggest to try run with GPU direct communication  
export GMX\_ENABLE\_DIRECT\_GPU\_COMM=1  
combined with full GPU offload  
-pme gpu -npme 1 -update gpu -bonded gpu  
combined with oversubscription (as Berk mentions above).

E.g. something like

export GMX\_ENABLE\_DIRECT\_GPU\_COMM=1  
gmx mdrun -ntmpi 8 -ntomp 4 -pin on -v -nb gpu -pme gpu -npme 1-update gpu -bonded gpu -noconfout -nsteps 100000 -resetstep 90000 -nstlist 300 -gpu\_id 01

This can allow much higher performance than default settings.

If you get an error message regarding GPU update not being supported, try setting “constraints = h-bonds” in your .mdp file, as discussed at [Getting good performance from mdrun — GROMACS 2022.4 documentation](https://manual.gromacs.org/documentation/current/user-guide/mdrun-performance.html#run-setup)