Alchemical calculations on GPU

User discussions
, ,
1

GROMACS version: 2023
GROMACS modification: Yes/No

Hello Gromacs users,
I have a general question on how to implement alchemical calculations on a GPU cluster. Our GPU cluster has 8 GPUs per node (A100) with 128 cores (2 nodes of AMD epyc). I want to run 8 replicas at different temperatures for one window in the alchemical transformation. Here is my command to run the simulations:

mpirun -np 8 gmx_mpi mdrun -v -deffnm prd1 -ntomp 16 -nb gpu -pme gpu -bonded gpu -gputasks 01234567 -update cpu -multidir 298 303 310 317 324 331 338 345 -nex 1000 -replex 1000

Then I got an error:

Inconsistency in user input: There were 8 GPU tasks assigned on node gpu1, but 16 GPU tasks were identified, and these must match. Reconsider your GPU task assignment, number of ranks, or your use of the -nb, -pme, and -npme options.

All I want is to partition the 8 replicas so that each replica runs with 16 cores of CPU and 1 core of GPU. Can I get some suggestions from the Gromacs community?

2

The manual is incorrect here. You also need to specify the PME tasks, so 0011223344556677. But I think it should also work without specifying the -gputasks option.

3

Thanks for the information. Then I got the error message like this:

Fatal error:
The -dd or -npme option request a parallel simulation, but gmx mdrun was not started through mpirun/mpiexec or only one rank was requested through mpirun/mpiexec

Any clue on this issue?

4

Is that both with and without the -gputasks option? I think it should at least work without specifying this option.

5

I have not tried without -gputasks. But I got it work out with the following command:

mpirun -np 8 gmx_mpi mdrun -v -deffnm prd1 -dlb yes -resethway -pin on \
       -ntomp 16 -npme -1 -gputasks 0011223344556677 -tunepme \
       -nb gpu -pme gpu -bonded gpu \
       -multidir 298 303 310 317 324 331 338 345 -nex 1000 -replex 1000

Correct me if I am wrong, but I think it is important to include -npme -1 and -tunepme which let gromacs to determine the core partition by itself.

6

Ah, it could be that you indeed need to specify -npme when using -gputasks. But then gromacs should complain when not specifying it.

I still think that you should be able to run without -gputasks.