Man thanks for your help ! ! !
Question 1
Will reading your post I already thought it might be the Verlet buffer estimation that could be the cost. You can avoid that part of the code by setting verlet-buffer-tolerance=-1 and setting nstlist and rlist manually.
As your suggestion, we modified the mdp file with two different setting.
First setting:(rlist = 1.0 )
; Neighborsearching
cutoff-scheme = Verlet
nstlist = 20 ; 20 steps, largely irrelevant with Verlet scheme
rlist = 1.0 ; short-range neighborlist cutoff (in nm)
verlet-buffer-tolerance = -1
rcoulomb = 1.0 ; short-range electrostatic cutoff (in nm)
rvdw = 1.0 ; short-range van der Waals cutoff (in nm)
The tpr file will be generated in a short time (~15 min). Howerer, we get the following error during the MD running using GPU (with gmx mdrun -v -deffnm nvt1 -gpu_id 0 -ntmpi 1 -ntomp 64 -nb gpu -pme gpu -bonded auto -update gpu):
Program: gmx mdrun, version 2022.5
Source file: src/gromacs/gpu_utils/devicebuffer.cuh (line 93)
Function: auto freeDeviceBuffer(float **)::(anonymous class)::operator()() const
Assertion failed:
Condition: stat == cudaSuccess
Freeing of the device buffer failed. CUDA error #700
(cudaErrorIllegalAddress): an illegal memory access was encountered.
If we running the MD simulation with only CPU, the MD simulation will be running successfully, but only one CPU core can be used during md running process. The occupancy of gmx mdrun during MD running obtained from perf is shown as follow:
Second setting: (rlist = 1.2 )
; Neighborsearching
cutoff-scheme = Verlet
nstlist = 20 ; 20 steps, largely irrelevant with Verlet scheme
rlist = 1.2 ; short-range neighborlist cutoff (in nm)
verlet-buffer-tolerance = -1
rcoulomb = 1.0 ; short-range electrostatic cutoff (in nm)
rvdw = 1.0 ; short-range van der Waals cutoff (in nm)
The tpr file will be generate in a short time (17 min). Howerer, we could not judge if the MD simulation run successfully (The CPU occupancy is 100, the GPU occupancy rate keeps 0% during md running process. In addition, the corresponding log files are not updated in ten hours.). The occupancy of gmx mdrun during MD running obtained from perf is shown as follow:
Samples: 450K of event 'cycles', Event count (approx.): 323821671531
Overhead Command Shared Object Symbol
99.58% gmx libgromacs.so.7.0.0 [.] Nbnxm::sort_atoms
0.15% gmx [kernel.kallsyms] [k] change_pte_range
0.03% gmx [kernel.kallsyms] [k] _raw_spin_lock_irqsave
0.03% gmx [kernel.kallsyms] [k] timekeeping_advance
0.02% gmx [kernel.kallsyms] [k] _raw_spin_lock
0.01% gmx [kernel.kallsyms] [k] account_user_time
0.01% gmx [kernel.kallsyms] [k] __run_hrtimer
Question 2
How many atoms do you have summed over all moleculetypes? So how many atoms would there be in the system if you would have exactly one molecule per type?
Our system has 13676501 atoms in total, including 1822 different moleculer types with different number (328976 atoms in total), 4436329 water moleculers and 38538 ions. If we have exactly one molecule per typeIf each molecule, the total number of atoms is 61119.
Question 3
And could you report the number of atom types for the Verlet buffer calculation? You can get this by running:
gmx mdrun -debug 1
or
gmx grompp -debug 1
Then there will be a file called gmx.debug and the number is on the line with:
energy drift atom types:
As your suggestion, we tried the command : gmx grompp -debug 1 (The value of verlet-buffer-tolerance in mdp file was setting to 0.005), the ouput is :
energy drift atom types: 48124
Question 4
One more question: does each atom have a different charge (apart from symmetric ones as in your first post)? If so, than that will lead to an extremely large number of different atom types in the Verlet buffer calculation. Rounding the charges in that calculation would help then.
As you can see, each atom have a different charge. The charges of all small molecules are RESP charges calculated by Gaussian.
; modyfied by wangah 20230817
[ atomtypes ]
; name at.num mass charge ptype sigma epsilon
OW 8 16.00 0.0000 A 3.15061e-01 6.36386e-01
HW 1 1.008 0.0000 A 0.00000e+00 0.00000e+00
Cl 17 35.45 0.0000 A 4.40104e-01 4.18400e-01
Na 11 22.99 0.0000 A 3.32840e-01 1.15897e-02
K 19 39.10 0.0000 A 4.73602e-01 1.37235e-03
c1 c1 0.00000 0.00000 A 3.39967e-01 8.78640e-01 ; 1.91 0.2100
c2 c2 0.00000 0.00000 A 3.39967e-01 3.59824e-01 ; 1.91 0.0860
c3 c3 0.00000 0.00000 A 3.39967e-01 4.57730e-01 ; 1.91 0.1094
ca ca 0.00000 0.00000 A 3.39967e-01 3.59824e-01 ; 1.91 0.0860
c c 0.00000 0.00000 A 3.39967e-01 3.59824e-01 ; 1.91 0.0860
...
...
...
