# Error when running REUS on multiple nodes

**URL:** <https://gromacs.bioexcel.eu/t/error-when-running-reus-on-multiple-nodes/6519>\
**Category:** User discussions\
**Tags:** mdrun\
**Created:** [May 30, 2023, 4:45pm UTC](https://gromacs.bioexcel.eu/t/error-when-running-reus-on-multiple-nodes/6519 "2023-05-30T16:45:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jsaulhf](https://avatars.discourse-cdn.com/v4/letter/j/e19adc/32.png) [@jsaulhf](https://gromacs.bioexcel.eu/u/jsaulhf)\
**Post date:** [May 30, 2023, 4:45pm UTC](https://gromacs.bioexcel.eu/t/error-when-running-reus-on-multiple-nodes/6519/1 "2023-05-30T16:45:17Z")

</div>

GROMACS version: gromacs-2020.4+plumed-2.6.2.sh  
GROMACS modification: Yes  
I’m running a replica-exchange umbrella sampling simulation with 24 replicas in 48 nodes. Every node has 128 physical cores and 256 logical cores.

The command to run the job is:

srun gmx\_mpi mdrun -v -multidir 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 -deffnm md\_0\_1 -replex 1000 -plumed …/cv.dat

and gives a performance of 60 ps/h, when I modify

If I set --ntasks=128x48=6144 the job runs and I get a performance of 60 ps/h. If instead I set ntasks=256x48=12288 to use all the logical cores, my job is killed and I get the following message:

- srun gmx\_mpi mdrun -v -multidir 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 -deffnm md\_0\_1 -replex 1000 -plumed …/cv.dat  
kvsprovider[8844]: Timeout: Not all clients called pmi\_init(): init=1024 left=11264 round=1  
kvsprovider[8844]: Timeout: Not all clients called pmi\_init(): init=2048 left=10240 round=2  
kvsprovider[8844]: Timeout: Not all clients called pmi\_init(): init=2998 left=9290 round=3  
kvsprovider[8844]: Timeout: Not all clients called pmi\_init(): init=3969 left=8319 round=4  
kvsprovider[8844]: Timeout: Not all clients called pmi\_init(): init=4943 left=7345 round=5  
kvsprovider[8844]: Timeout: Not all clients called pmi\_init(): init=5926 left=6362 round=6  
kvsprovider[8844]: Timeout: Not all clients called pmi\_init(): init=6912 left=5376 round=7  
kvsprovider[8844]: Timeout: Not all clients called pmi\_init(): init=7936 left=4352 round=8  
srun: error: timeout waiting for task launch, started 8704 of 12288 tasks  
srun: launch/slurm: launch\_p\_step\_launch: StepId=12006718.0 aborted before step completely launched.  
srun: Job step aborted: Waiting up to 6 seconds for job step to finish.  
readFromPMIClient: lost connection to the PMI client  
srun: error: task 9216 launch failed: Unspecified error  
srun: error: task 9217 launch failed: Unspecified error

Do you have any idea why this occurs?

---

<div class="post-metadata">

**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\
**Post date:** [June 26, 2023, 4:30pm UTC](https://gromacs.bioexcel.eu/t/error-when-running-reus-on-multiple-nodes/6519/2 "2023-06-26T16:30:46Z")

</div>

You are using a plumed-modified version of GROMACS not an upstream official one. Can you please try running with an unmodified GROMACS and report back if you still see the failures (and you might see performance improvement too).

Cheers,  
Szilárd