GROMACS version: 2022.3
GROMACS modification: Yes
In his excellent tutorial about virtual sites, @jalemkul uses the -nb cpu option when running his minimization and production simulation. I was wondering whether and why this is necessary? Mainly because it slows my simulation down 3-4 fold.
He does make some mention of the -nb cpu flag when discussing the calculation of interaction energies in a -rerun, but I am not sure whether that’s relevant to calculating the forces in the actual simulation for a virtual site.
The documentation makes mention of the use of the GPU in regard to virtual sites
-update …Defaults to “auto,” which currently always uses the CPU… Update and constraints on a GPU is currently not supported with …, virtual sites,…
But that is for “update and constraints”, not necessarily non-bonded interactions.
Otherwise I cannot locate documentation mentioning the use of -nb cpu with virtual sites.
The use of -nb cpu is left over from a much older GROMACS version that actually used the group cutoff scheme and thus required the use of CPU. At one point, virtual sites weren’t supported on GPU, so I left it in place. If you’re using 2022.3, it may work on GPU (note that I wrote the tutorial for 2018, so if you’re using a different version, your mileage may vary).
I just wanted to follow up with what my experience with virtual sites has been.
I did a little experiment in which I found that, if I do use virtual sites without setting -nb, my results were much closer to those of my control (in which I didn’t even have virtual sites and also left the -nb flag out) than to a third case in which I did have virtual sites and -nb cpu. In the -nb cpu case, my system is “better behaved”. While there were no crashes or anything, I’m left wondering whether virtual sites with non-bonded on the gpu is doing what I was expecting it to do. I know that this is not definitive proof one way or the other and, as you said, my mileage may vary, but for my particular case, I am going with -nb cpu.
This is all done with version 2022.3 so may be this is not the case in the current version, but I would caution anyone using virtual sites to at least check to see whether you get similar results with -nb cpu.
There should never be any significant differences between running part of the calculation on the CPU or GPU. Integration always happens on the CPU when virtual sites are present, but any part of the force calculation can be offloaded to the GPU. I don’t see how virtual sites would be “special” in any way, except for forcing the integration to be on the CPU.