Simulations with multiple GPUs

User discussions
1

GROMACS version: 2020.2
GROMACS modification: No

Good morning everyone,

I want to perform MD simulations using multuple GPUs on a computing cluster. I want to use a single node formed by 32 CPU and 4 GPUs. Specifically, I want to use all the CPUs and two GPUs. I wrote the following SLURM job:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=2
#SBATCH --time=00:02:00
#SBATCH --mem=50000
#SBATCH --account=
#SBATCH --partition
#SBATCH --job-name=s3
#SBATCH --cpus-per-task=32
#SBATCH --gres=gpu:2

module load profile/…
module load autoload gromacs/2020.2

export GMX_GPU_DD_COMMS=true
export GMX_GPU_PME_PP_COMMS=true
export GMX_FORCE_UPDATE_DEFAULT_GPU=true

gmx grompp -f md.mdp -c equi2.gro -t equi2.cpt -p comp-gromacs.top -o md.tpr -maxwarn 1
gmx_thread_mpi mdrun -deffnm md -cpi md.cpt -maxh 0.03 -ntomp 8 -v -ntmpi 4 -nb gpu -pme gpu -pin off -npme 1

When I launch it, it terminate immediately with a segmentation fault. Any advice on how to fix? Did I wrote the SLURM job correctly?

Thanks in advance

Mattia

2

First and foremost, use a more recent GROMACS, preferably 2022 (or at least 2021).

Also note that you’re requesting only 2 GPUs above.

3

Hi,

Also, you’ve requested 2 tasks and 32 OMP threads via SLURM. This conflicts with mdrun options in which you’ve set up 4 thread MPI and 8 OpenMP threads.

4

Good catch, if you are using thread-MPI and only intra-node parallelization, use --nnodes=1 --ntasks-per-node=1 and especially if your SLURM sets cgruops limits make sure to request the correct --cpus-per-task and use -pin on (especially if you’re not using all hardware thread) so thread affinities are set by mdrun.