# Error with GPU direct comms enabled

**URL:** <https://gromacs.bioexcel.eu/t/error-with-gpu-direct-comms-enabled/4077>\
**Category:** User discussions\
**Created:** [April 26, 2022, 10:40am UTC](https://gromacs.bioexcel.eu/t/error-with-gpu-direct-comms-enabled/4077 "2022-04-26T10:40:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sfarr-epcc](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@sfarr-epcc](https://gromacs.bioexcel.eu/u/sfarr-epcc)\
**Post date:** [April 26, 2022, 10:40am UTC](https://gromacs.bioexcel.eu/t/error-with-gpu-direct-comms-enabled/4077/1 "2022-04-26T10:40:37Z")

</div>

GROMACS version: 2022  
GROMACS modification: No  
I have been doing some benchmarking of GROMACS on JUWELS Booster. For a specific benchmark system:

benchRIB from [A free GROMACS benchmark set | Max Planck Institute for Multidisciplinary Sciences](https://www.mpinat.mpg.de/grubmueller/bench)

When I use 16 GPUs with GMX\_ENABLE\_DIRECT\_GPU\_COMM=true the simulation fails with error:

Fatal error:  
Step 600: The total potential energy is nan, which is not finite.

If I run without setting the GMX\_ENABLE\_DIRECT\_GPU\_COMM flag then the simulation completes successfully.

---

<div class="post-metadata">

**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\
**Post date:** [April 26, 2022, 2:57pm UTC](https://gromacs.bioexcel.eu/t/error-with-gpu-direct-comms-enabled/4077/2 "2022-04-26T14:57:37Z")

</div>

Can you post your command line for mdrun?  
Tagging @pszilard on this one.

---

<div class="post-metadata">

**Author:** ![sfarr-epcc](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@sfarr-epcc](https://gromacs.bioexcel.eu/u/sfarr-epcc)\
**Post date:** [April 26, 2022, 3:39pm UTC](https://gromacs.bioexcel.eu/t/error-with-gpu-direct-comms-enabled/4077/3 "2022-04-26T15:39:20Z")

</div>

This is the run command:  
srun gmx\_mpi mdrun -s benchRIB.tpr -nsteps 10000 -v -ntomp 12 -nb gpu -pme gpu -bonded gpu -npme 1 -notunepme

With slurm settings:  
#SBATCH --nodes=4  
#SBATCH --ntasks=16  
#SBATCH --hint=nomultithread  
#SBATCH --ntasks-per-node=4  
#SBATCH --cpus-per-task=12  
#SBATCH --partition=booster  
#SBATCH --gres=gpu:4

---

<div class="post-metadata">

**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\
**Post date:** [April 27, 2022, 9:58am UTC](https://gromacs.bioexcel.eu/t/error-with-gpu-direct-comms-enabled/4077/4 "2022-04-27T09:58:15Z")

</div>

That’s definitely not expected, please file an issue on [https://gitlab.com/gromacs/gromacs/-/issues](https://gitlab.com/gromacs/gromacs/-/issues) preferably with your input, job script, and log file, please.