I am simulating a highly dense system containing carbohydrates and amino acids. I am performing a multistep equilibration protocol with gradually decreasing force constants on the heavy atoms. The restrained NVT equilibration steps are each 1 ns long.
The problem occurs when I switch from NVT to NPT equilibration. The run eventually crashes with the following error:
step 4764300, will finish Thu Aug 20 14:52:23 2026terminate called after throwing an instance of 'gmx::InternalError'
what(): Freeing of the device buffer failed. CUDA error #700 (cudaErrorIllegalAddress): an illegal memory access was encountered.
gro_gpu_equi.sh: line 4: 46452 Aborted (core dumped) gmx mdrun -v -deffnm equilibration/step4.${ii} -ntomp 32 -ntmpi 1 -pin on -nb auto -bonded auto -pme auto -update auto
I have tried several approaches to identify the cause:
Changing the barostat and thermostat used during NPT equilibration.
Switching the nonbonded and PME calculations completely to the CPU.
Increasing the LINCS order.
Switching gromacs versions (2025.1 and 2026.0) and GPU cards (V100, A100, and RTX3080)
The thermodynamic properties appear to behave normally. The pressure, potential energy, total energy, and box volume converge smoothly, and I do not observe any obvious spikes immediately before the crash.
There also does not seem to be a specific simulation step at which the crash occurs. However, it typically happens after approximately 1 ns of NPT equilibration.
One additional detail that may be relevant is that the system has periodic connections along one of the axes.
I would appreciate any suggestions on what could cause this behavior or what additional diagnostics I should perform. Could this be related to periodic connectivity, topology, box dimensions/deformation, or perhaps an issue with the GPU implementation?
I have attached the mdp files that I am using for NVT (step 4.5) and NPT (step 4.7) runs.
We try to avoid segfaults and CUDA errors, but some kernels do not have out of bounds checks for performance reasons (we should fix this). Do you get any LINCS warnings before the crash?
Your mdp settings are very standard. But a periodic molecule can cause issues. How much does the box size along z change and in which direction?
Have you really ran with nonbonded calculations on CPU? Then the GPU will not be used att all and you can not get a CUDA error.
No, I am not getting any LINCS warning. Regarding the box size, it changes the maximum along Y by about 5 A and along Z by about 2-3 A. The periodic connections are along X.
I ran the calculations on the CPU. In that case, it just says “Segmentation Fault (core dumped).” I am not sure if this is due to periodic connections; even if I remove the periodic connections along X, the same error occurs.
Sorry, but I am not sure about what the core file is. Are you referring to the out/log file? If it is something different, then I am not sure about how to generate the core file.
If you do not have limits set, are so called “core dump” file is generated when an applicaton segfaults. Such files have names starting with “core” and contain the state of the application at crash time. You might need to do
There are limits. Both “ulimit -Sc and ulimit -Hc” are set to 0. Is there a way I can reset them through the job script? Based on what I see, they can be controlled through the Slurm configuration.
I get the same CUDA crash with gmx 2025.4 doing protein-peptide complex simulations. For these simulations I use the mass-repartition-factor = 3 and dt = 0.004 ps time step parameters. The CUDA crash occurs always when the peptide dissociate-associate from-to the protein. And it happens with different peptides, so the crass is not specific for a particular system. However, when I continue the simulation from a check point file, most of the time the dissociation-association does not happens and/or not always crashes. I should also add that when I don’t use mass repartition and 4 fs step parameters the crash does not take place. For these simulation I use RTX 5070 Ti with Nvidia 610.43.03 and CUDA 13.3
This could be a completely unrelated issue. If something goes wrong in CUDA code this is a likely error message. This sounds more like an instability in the interactions between certain groups of atoms.
Yes, I sort of agree, but this only happens with mass repartition and 4f time step, so I think this could be, by not being a coder, due to coding issue/bug as well.