# Gmx\_mpi, PME and AWH

**URL:** <https://gromacs.bioexcel.eu/t/gmx-mpi-pme-and-awh/3798>\
**Category:** User discussions\
**Created:** [March 8, 2022, 9:37am UTC](https://gromacs.bioexcel.eu/t/gmx-mpi-pme-and-awh/3798 "2022-03-08T09:37:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![WillPPK](https://avatars.discourse-cdn.com/v4/letter/w/3bc359/32.png) [@WillPPK](https://gromacs.bioexcel.eu/u/WillPPK)\
**Post date:** [March 8, 2022, 9:37am UTC](https://gromacs.bioexcel.eu/t/gmx-mpi-pme-and-awh/3798/1 "2022-03-08T09:37:23Z")

</div>

GROMACS version: 2021.4  
GROMACS modification: No

I tried to conduct AWH simulation with gmx\_mpi on slurm and it didn’t load and I got this error:  
I got this…

```auto
Program: gmx mdrun, version 2021.5
Source file: src/gromacs/mdrunutility/multisim.cpp (line 67)
Function: std::unique_ptr<gmx_multisim_t> buildMultiSimulation(MPI_Comm, gmx::ArrayRef<const std::__cxx11::basic_string<char> >)

Feature not implemented:
Multi-simulations are only supported when GROMACS has been configured with a
proper external MPI library.

For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors

```

This was solved by Yuxuan Zhuang with:

```auto
module load gromacs/2021.4 gromacs=gmx_mpi

```

Then I struggle with -pme gpu setting, as the submission script is:  
!/bin/bash

```auto
Submit to the tcb partition
#SBATCH -p tcb

The name of the job in the queue
#SBATCH -J awh_k

wall-clock time requested for this job
#SBATCH -t 24:00:00

Number of nodes and number of MPI processes per node
#SBATCH -N 4 --ntasks-per-node=4

Request GPU node(s) and two GPUs (per node)
#SBATCH -C gpu --gres=gpu:2

Output file names for stdout and stderr
#SBATCH -e "./job-%j.err"

#SBATCH -o "./slurm-%j.out"

The actual script starts here
echo $SLURM_ARRAY_TASK_ID

module unload gromacs
module unload openmpi

module load gromacs/2021.4 gromacs=gmx_mpi
module load openmpi/4.0

cd ${SLURM_SUBMIT_DIR}

mpirun -np 16 gmx_mpi mdrun -deffnm awh -cpi awh.cpt -multidir awh_1 awh_2 awh_3 awh_4 -nsteps -1 -px awh_pullx -pf awh_pullf -update gpu -v -pme gpu

```

The error message was:

```auto
Program: gmx mdrun, version 2021.4
Source file: src/gromacs/ewald/pme.cpp (line 894)
Function: gmx_pme_t* gmx_pme_init(const t_commrec, const NumPmeDomains&, const t_inputrec, gmx_bool, gmx_bool, gmx_bool, real, real, int, PmeRunMode, PmeGpu, const DeviceContext, const DeviceStream, const PmeGpuProgram, const gmx::MDLogger&)
MPI rank: 6 (out of 16)

Feature not implemented:
PME GPU does not support PME decomposition.

For more information and tips for troubleshooting, please check the GROMACS
website at http://www.gromacs.org/Documentation/Errors

```

The diagnosis of the error is:

Quote : Paul Bauer : For the error, you can’t have multiple pme ranks when offloading PME work to the GPU

Solution : Provided by Magnus Lundborg:

“mpirun -np 4” if you’re running 4 awh walkers.

---

<div class="post-metadata">

**Author:** ![pbauer](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pbauer/32/13_2.png) [@pbauer](https://gromacs.bioexcel.eu/u/pbauer)\
**Post date:** [March 8, 2022, 9:44am UTC](https://gromacs.bioexcel.eu/t/gmx-mpi-pme-and-awh/3798/2 "2022-03-08T09:44:00Z")

</div>

The expand on my answer concerning the PME GPU error:

To make sure PME can run on the GPU, use `-npme 1 -pme gpu`

---

<div class="post-metadata">

**Author:** ![Joe](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/joe/32/1322_2.png) [@Joe](https://gromacs.bioexcel.eu/u/Joe)\
**Post date:** [March 8, 2022, 9:51am UTC](https://gromacs.bioexcel.eu/t/gmx-mpi-pme-and-awh/3798/3 "2022-03-08T09:51:26Z")

</div>

I think this should get an issue in gitlab, as this is indeed confusing. If a user specifies `-pme gpu` without specifying the number of PME ranks, there should not be an error, but the choice `-npme 1` should be selected.

---

<div class="post-metadata">

**Author:** ![WillPPK](https://avatars.discourse-cdn.com/v4/letter/w/3bc359/32.png) [@WillPPK](https://gromacs.bioexcel.eu/u/WillPPK)\
**Post date:** [March 8, 2022, 10:03am UTC](https://gromacs.bioexcel.eu/t/gmx-mpi-pme-and-awh/3798/4 "2022-03-08T10:03:06Z")

</div>

The issue is now brought up and copy-paste to Gitlab for future user:

> **[gmx\_mpi, PME and AWH (#4426) · Issues · GROMACS / GROMACS](https://gitlab.com/gromacs/gromacs/-/issues/4426)**
>
> GROMACS version: 2021.4 GROMACS modification: No I tried to conduct AWH simulation with gmx\_mpi on slurm and it didn't load and I got this error: I...