GROMACS version: 2022.3
GROMACS modification: No
Hello, GROMACS users!
I have encountered problems with resuming my REMD simulation after a power outage. I was lucky that all cpt files had the same step and the simulation resumed. However, after approximately 300,000-500,000 steps, it seems to hang: the step does not change, the GPUs are not busy, although nvidia-smi shows that gmx processes are consuming memory. However, the performance is 0. The task hangs for half a day until I stop it with Ctrl-C. There is no information in the log files, and no information on the screen about hanging. I have been simulating the same system for quite a long time, this is the fifth run. The other four runs completed without errors, and this is the first time I have encountered such an issue.
I’ve tried to resume the simulation again with command:
mpirun -np 64 gmx_mpi mdrun -multidir {1…32} -s md3.tpr -deffnm md3 -gpu_id 01 -nb gpu -v -cpi -replex 1000 -noappend
and with starting a new simulation using previous cpt:
mpirun -np 32 gmx_mpi mdrun -multidir {1…32} -s md5.tpr -deffnm md3_2 -gpu_id 01 -nb gpu -v -replex 1000
I’ve changed gromacs ( I have two programs installed with different dependencies) utility. I’ve re-installed OpenMPI. I’ve rebooted the computer. I’ve tried to enable GMX_ENABLE_DIRECT_GPU_COMM. I’ve tried to change number of cpu cores and replex steps number.
In all the cases the simulation stucks after about 400 000 steps.
For now, I have no idea what to do.
I will be grateful for any help.
Best regards,
Anna