GROMACS version:2021

GROMACS modification: No

Hi everyone,

I did two setups on CHARMM-GUI, one containing a protein+water (S1) with 70,700 atoms and the other containing a protein+water+ligand (S2) with 68,200 atoms. Thus, the systems differ by 2500 atoms. Parameters in the *.mdp file were the same for both systems. When I run the simulations using 4 GPUs, the performances are about 80 ns/day and 105 ns/day for S1 and S2, respectively. The difference in performance looks big to me due to the relatively small difference in atoms number in both simulations. The outputs for both cases are as follows:

S1:

Changing nstlist from 20 to 100, rlist from 1.224 to 1.346

##
Initializing Domain Decomposition on 4 ranks

Dynamic load balancing: on

Using update groups, nr 24087, average size 2.9 atoms, max. radius 0.139 nm

Minimum cell size due to atom displacement: 0.657 nm

Initial maximum distances in bonded interactions:

two-body bonded interactions: 0.420 nm, LJ-14, atoms 109 116

multi-body bonded interactions: 0.489 nm, CMAP Dih., atoms 732 753

Minimum cell size due to bonded interactions: 0.537 nm

Scaling the initial minimum size with 1/0.8 (option -dds) = 1.25

Using 1 separate PME ranks

Optimizing the DD grid for 3 cells with a minimum initial size of 0.821 nm

The maximum allowed number of cells is: X 10 Y 10 Z 10

Domain decomposition grid 3 x 1 x 1, separate PME ranks 1

PME domain decomposition: 1 x 1 x 1

Interleaving PP and PME ranks

This rank does only particle-particle work.

Domain decomposition rank 0, coordinates 0 0 0

The maximum number of communication pulses is: X 1

The minimum size for domain decomposition cells is 1.624 nm

The requested allowed shrink of DD cells (option -dds) is: 0.80

The allowed shrink of domain decomposition cells is: X 0.55

The maximum allowed distance for atom groups involved in interactions is:

non-bonded interactions 1.624 nm

two-body bonded interactions (-rdd) 1.624 nm

multi-body bonded interactions (-rdd) 1.624 nm

On host tp2 4 GPUs selected for this run.

S2:

Changing nstlist from 20 to 100, rlist from 1.224 to 1.345

##
Initializing Domain Decomposition on 4 ranks

Dynamic load balancing: on

Minimum cell size due to atom displacement: 0.750 nm

Initial maximum distances in bonded interactions:

two-body bonded interactions: 0.433 nm, LJ-14, atoms 1660 1667

multi-body bonded interactions: 0.484 nm, CMAP Dih., atoms 227 236

Minimum cell size due to bonded interactions: 0.532 nm

Maximum distance for 5 constraints, at 120 deg. angles, all-trans: 0.222 nm

Estimated maximum distance required for P-LINCS: 0.222 nm

Scaling the initial minimum size with 1/0.8 (option -dds) = 1.25

Using 1 separate PME ranks

Optimizing the DD grid for 3 cells with a minimum initial size of 0.938 nm

The maximum allowed number of cells is: X 9 Y 9 Z 9

Domain decomposition grid 3 x 1 x 1, separate PME ranks 1

PME domain decomposition: 1 x 1 x 1

Interleaving PP and PME ranks

This rank does only particle-particle work.

Domain decomposition rank 0, coordinates 0 0 0

The maximum number of communication pulses is: X 1

The minimum size for domain decomposition cells is 1.345 nm

The requested allowed shrink of DD cells (option -dds) is: 0.80

The allowed shrink of domain decomposition cells is: X 0.46

The maximum allowed distance for atoms involved in interactions is:

non-bonded interactions 1.345 nm

two-body bonded interactions (-rdd) 1.345 nm

multi-body bonded interactions (-rdd) 1.345 nm

atoms separated by up to 5 constraints (-rcon) 1.345 nm

On host tp2 4 GPUs selected for this run.

Is there any recommendation on how to improve the performance of S1 by using some flags with mdrun?

I already tried “… mdrun -rdd 1.345 -rcon 1.345 …” but I haven’t seen any improvement. Thanks.