# Low performance and Thread-MPI error in multi GPU usage

**URL:** <https://gromacs.bioexcel.eu/t/low-performance-and-thread-mpi-error-in-multi-gpu-usage/6362>\
**Category:** User discussions\
**Tags:** mdrun\
**Created:** [May 10, 2023, 6:08am UTC](https://gromacs.bioexcel.eu/t/low-performance-and-thread-mpi-error-in-multi-gpu-usage/6362 "2023-05-10T06:08:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![psj1303](https://avatars.discourse-cdn.com/v4/letter/p/5f9b8f/32.png) [@psj1303](https://gromacs.bioexcel.eu/u/psj1303)\
**Post date:** [May 10, 2023, 6:08am UTC](https://gromacs.bioexcel.eu/t/low-performance-and-thread-mpi-error-in-multi-gpu-usage/6362/1 "2023-05-10T06:08:08Z")

</div>

GROMACS version: 2023.01  
GROMACS modification: No

I installed GROMACS 2023.01 using the cmake command below.

cmake … -DGMX\_USE\_RDTSCP=ON -DGMX\_SIMD=AVX2\_256 -DGMX\_BUILD\_MDRUN\_ONLY=ON -DREGRESSIONTEST\_DOWNLOAD=ON -DGMX\_BUILD\_OWN\_FFTW=ON -DGMX\_MPI=ON -DGMX\_THREAD\_MPI=ON -DGMX\_GPU=CUDA -DCMAKE\_C\_COMPILER=gcc-9 -DCMAKE\_CXX\_COMPILER=g+±9

But, When I execute mdrun like below,  
I got the error, THREAD\_MPI is not complied during installation.  
mpirun -np 2 gmx mdrun -v -deffnm ${mini\_prefix} -ntmpi 2 -npme 1 -gputasks 01

When I use another command to execute mdrun, the process run normally but the performance is more slower than when I use one GPU (77 ns/day → 32 ns/day).  
mpirun -np 2 gmx mdrun -v -deffnm ${mini\_prefix} -gpu\_id 01

I want to execute mdrun with -ntmpi, -npme setting to improve multi-GPU performance, but I can’t do it due to THREAD\_MPI error.  
Is there any issue for my cmake build command?

---

<div class="post-metadata">

**Author:** ![al42and](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/al42and/32/1393_2.png) [@al42and](https://gromacs.bioexcel.eu/u/al42and)\
**Post date:** [May 12, 2023, 6:11pm UTC](https://gromacs.bioexcel.eu/t/low-performance-and-thread-mpi-error-in-multi-gpu-usage/6362/2 "2023-05-12T18:11:45Z")

</div>

Hi!

It looks like you are mixing up “real” MPI and threadMPI.

- threadMPI is GROMACS internal “pseudo-MPI” implementation; you launch it like `gmx mdrun -ntmpi NRANKS ...`. It can only work on a single machine.

- libMPI is an external, proper MPI implementation (OpenMPI, MPICH, etc.), able to scale across multiple compute nodes. With it, you launch GROMACS like `mpirun -np NRANKS gmx_mpi mdrun ...`. If you want to use multiple GPUs efficiently in this case, you need GPU-aware (CUDA-aware) MPI installation (if you just `apt install`ed it, it probably is not).

You cannot mix the two.

If you run on a single machine, don’t use `mpirun`. Build GROMACS with `-DGMX_MPI=OFF -DGMX_THREAD_MPI=ON`, and use ` gmx mdrun -ntmpi 2 -npme 1 ...`.

It would also be helpful to set the `GMX_ENABLE_DIRECT_GPU_COMM=1` environment variable; see [https://manual.gromacs.org/current/user-guide/mdrun-performance.html#assigning-tasks-to-gpus](https://manual.gromacs.org/current/user-guide/mdrun-performance.html#assigning-tasks-to-gpus)

---

<div class="post-metadata">

**Author:** ![psj1303](https://avatars.discourse-cdn.com/v4/letter/p/5f9b8f/32.png) [@psj1303](https://gromacs.bioexcel.eu/u/psj1303)\
**Post date:** [May 16, 2023, 12:46am UTC](https://gromacs.bioexcel.eu/t/low-performance-and-thread-mpi-error-in-multi-gpu-usage/6362/3 "2023-05-16T00:46:59Z")

</div>

I got it. I I understand your explanation.  
I must have been confusing the two concepts because I’m still a beginner at GROMACS.

Thank you for your kind explanation.  
I’ll try to rebuild GROMACS as you said.

Thank you, again.