# Alchemical calculations on GPU

**URL:** <https://gromacs.bioexcel.eu/t/alchemical-calculations-on-gpu/7413>\
**Category:** User discussions\
**Tags:** mdrun, gpu, free-energy\
**Created:** [October 16, 2023, 9:07pm UTC](https://gromacs.bioexcel.eu/t/alchemical-calculations-on-gpu/7413 "2023-10-16T21:07:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![xywang181](https://avatars.discourse-cdn.com/v4/letter/x/b19c9b/32.png) [@xywang181](https://gromacs.bioexcel.eu/u/xywang181)\
**Post date:** [October 16, 2023, 9:07pm UTC](https://gromacs.bioexcel.eu/t/alchemical-calculations-on-gpu/7413/1 "2023-10-16T21:07:50Z")

</div>

GROMACS version: 2023  
GROMACS modification: Yes/No

Hello Gromacs users,  
I have a general question on how to implement alchemical calculations on a GPU cluster. Our GPU cluster has 8 GPUs per node (A100) with 128 cores (2 nodes of AMD epyc). I want to run 8 replicas at different temperatures for one window in the alchemical transformation. Here is my command to run the simulations:

mpirun -np 8 gmx\_mpi mdrun -v -deffnm prd1 -ntomp 16 -nb gpu -pme gpu -bonded gpu -gputasks 01234567 -update cpu -multidir 298 303 310 317 324 331 338 345 -nex 1000 -replex 1000

Then I got an error:

Inconsistency in user input: There were 8 GPU tasks assigned on node gpu1, but 16 GPU tasks were identified, and these must match. Reconsider your GPU task assignment, number of ranks, or your use of the -nb, -pme, and -npme options.

All I want is to partition the 8 replicas so that each replica runs with 16 cores of CPU and 1 core of GPU. Can I get some suggestions from the Gromacs community?

---

<div class="post-metadata">

**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\
**Post date:** [October 17, 2023, 6:17am UTC](https://gromacs.bioexcel.eu/t/alchemical-calculations-on-gpu/7413/2 "2023-10-17T06:17:26Z")

</div>

The manual is incorrect here. You also need to specify the PME tasks, so 0011223344556677. But I think it should also work without specifying the -gputasks option.

---

<div class="post-metadata">

**Author:** ![xywang181](https://avatars.discourse-cdn.com/v4/letter/x/b19c9b/32.png) [@xywang181](https://gromacs.bioexcel.eu/u/xywang181)\
**Post date:** [October 17, 2023, 3:09pm UTC](https://gromacs.bioexcel.eu/t/alchemical-calculations-on-gpu/7413/3 "2023-10-17T15:09:54Z")

</div>

Thanks for the information. Then I got the error message like this:

Fatal error:  
The -dd or -npme option request a parallel simulation, but gmx mdrun was not started through mpirun/mpiexec or only one rank was requested through mpirun/mpiexec

Any clue on this issue?

---

<div class="post-metadata">

**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\
**Post date:** [October 18, 2023, 8:25am UTC](https://gromacs.bioexcel.eu/t/alchemical-calculations-on-gpu/7413/4 "2023-10-18T08:25:26Z")

</div>

Is that both with and without the -gputasks option? I think it should at least work without specifying this option.

---

<div class="post-metadata">

**Author:** ![xywang181](https://avatars.discourse-cdn.com/v4/letter/x/b19c9b/32.png) [@xywang181](https://gromacs.bioexcel.eu/u/xywang181)\
**Post date:** [October 18, 2023, 3:32pm UTC](https://gromacs.bioexcel.eu/t/alchemical-calculations-on-gpu/7413/5 "2023-10-18T15:32:01Z")

</div>

I have not tried without -gputasks. But I got it work out with the following command:

```auto
mpirun -np 8 gmx_mpi mdrun -v -deffnm prd1 -dlb yes -resethway -pin on \
       -ntomp 16 -npme -1 -gputasks 0011223344556677 -tunepme \
       -nb gpu -pme gpu -bonded gpu \
       -multidir 298 303 310 317 324 331 338 345 -nex 1000 -replex 1000

```

Correct me if I am wrong, but I think it is important to include `-npme -1` and `-tunepme` which let gromacs to determine the core partition by itself.

---

<div class="post-metadata">

**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\
**Post date:** [October 18, 2023, 4:29pm UTC](https://gromacs.bioexcel.eu/t/alchemical-calculations-on-gpu/7413/6 "2023-10-18T16:29:56Z")

</div>

Ah, it could be that you indeed need to specify -npme when using -gputasks. But then gromacs should complain when not specifying it.

I still think that you should be able to run without -gputasks.