Running density guided simulations are very slow due to it switching off domain decomposition (guessing from the log message). The simulations are running at <10ns per day which is a slow down of roughly 15 fold (typical speed of >150ns per day). This is running on a A100 GPU.
log message:
“When checking whether update groups are usable:
Domain decomposition is not active, so there is no need for update groups”
What are the typical speeds people are getting for density guided simulations? According to the gromacs team if you use a nst of 100 there shouldn’t be too much of a slow down. The parameters I’ve used are from the bioexcel webinar.
What to change to get the promised simulations speed?
Is the DD switch off normal?
Secondary issue is the density-guided-simulation-group does not read the index file, so you cannot use any unique index groups. Is this also a normal setting?
Thanks for posting this on the forum. The domain-decomposition issue is indeed strange and should not happen. There is nothing in density-guided simulations themselves that forbids domain-decomposition and no communication between the density-guided simulation module and the routines that set up domain decomposition; I know it sounds a bit like “did you try turning it on and off again”, but could you try your exact setup with density-guided-simulation-active = no ?
Other issues that could drastically reduce performance is a large map, say 512^3 voxels. In this case, cropping out all “empty” space can help drastically increase performance. A usual other suspect is an excessive spreading-width, but from your parameters it looks reasonable to me.
The fitting works with arbitrary index groups if you hand them to grompp via the -n flag. Something that is not well described in the manual, if I have time I might add a sentence about this.
I’ve systematically gone through a lot of the setting. As a quick summary.
The biggest change to the calculation is similarity-measure. Inner-product seems to give almost normal speed, whilst cross-product is much much slower (factor of 100 fold). (would be good to include the speed penalties in the manual)
However inner-product doesn’t actually do anything, the max energy recorded in the edr file is about 1.5 kJ/mol compared to the 6x10^6 for cross-product.
Changing to “Protein-H” has a reasonable effect as would be expected from the reduced number of atoms.
Trimming the density from 29nm2 to 12nm2 had a marginal effect but barely noticeable.
Interestingly, the index file isn’t actually used by the density fitting code. As far as I can tell it remakes the index file internally and ignores the one supplied via the -n flag.
I’m using a mixed protein/DNA system. if you supply an index group called protein_DNA it is ignored.
the protein_DNA is used as tc-grp, so flags an error is missing from the index file.
error if missing: one from tc-grp and one from density code.
error if present: only one from density code.
However, if you use the standard gromacs index group names that are created by default e.g. Protein, DNA etc it works fine. This happens on gmx 2021 and 2022.
Current work around is using the default “non-Water” group though that includes ions so not ideal.
That cross-correlation is 100-fold slower is very surprising - while it is a bit more intricate, the calculation routine is not that much more involved and usually spreading atoms on the grid is the largest contributor to compute time which is identical for both.
That inner product does not anything is most likely related to the fact that it usually takes some more time to scale up the force constant to have a noticable effect.
Could you share your run input files? I feel it’s easier at this point if I have a look at them rather than running a lengthy back and forth.
Sorry I cannot share any of the input files as they are all confidential.
I’ve attached the inner product and cross product energy. The inner product is clearly not doing anything (identical starting frames and reference density).
The speed of the two is actually comparable, seems like one of the nodes I was running on was playing up.
Seems like only a few nanoseconds of sim are all that are required (at least in my cases). So speed is less of an issue.
So I guess the only issues are index groups and difference between inner and cross.
