Calculation stopped but job not finished

User discussions
1

GROMACS version: 2023.2
GROMACS modification: No

Hi,

I had launched 12 independent NPT runs for 1 us, and I noticed that all the runs got stucked at 502.5 ns without any error.

Because there were no error, the calculation jobs itsels did not finished, lead to quite a waste of time and money…

Has anyone experienced the same problem

Thanks in advance
Kotaro Tanaka

2

It is very difficult to say anything specific without more information. Did you run the 12 simulations at the same time, e.g., with the -multidir option? Otherwise it would be improbable that all simulations stopped at exactly the same time.

Did you get any output to the terminal explaining why they stopped? Have you checked the end of the log files? Did you run out of disk space? Were they run on your own local computer or on a remote machine with a queueing system?

3

Thank you for your reply

Did you run the 12 simulations at the same time, e.g., with the -multidir option?

Almost at the same time. Not with the -multidir option. I ran the 12 jobs independently with a shell script.

Did you get any output to the terminal explaining why they stopped?

No.

Have you checked the end of the log files?

Yes, but no error/warning or related message

Did you run out of disk space?

No.

Were they run on your own local computer or on a remote machine with a queueing system?

A remote machine with a queueing system (Fujitsu TCS)

step7_production.mdp (1.07 KB)

4

Since the jobs ran for 500 ns I don’t think the mdp parameters are the cause. With the available information I can’t say for sure whether the problem was caused by GROMACS or some temporary problems with the hardware/software stack of the server. But we’re of course interested if others have experienced the same thing. It’s not impossible that it is related to PBC wrapping hangs after a system explodes (#4766) · Issues · GROMACS / GROMACS · GitLab, but as I said above, if they had already run for 500 ns I don’t think that’s the problem.

5

Thank you for the comments.

I’ll just continue the calculations from where they stopped for now.

Thanks

6

I got same problem, in newest version. Each time it gets stucked at different steps above 500 ns, I tried multiple times but same problems continue

7

Thanks for reporting it. Still, without more information it is very difficult for us to say whether the problem is with GROMACS or with the local installation/system setup.

When you say “I got the same problem”. Do you mean just that the calculation stopped prematurely? Or also that you do not get any information in the console output or in the GROMACS log file? Are you also running multiple simulations in parallel?

Is it system dependent for you? Or does it happen with all systems above 500 ns? If you’ve got a TPR of a system we can try to see if we can reproduce it.

8

Yeah, Calculation got frozen after 500 ns, I did not run multiple simulations, I tried reinstalling Ubuntu as well gromacs multiple times , however, for each try sometimes it goes to 650 nm then gets stucked and sometimes less than that, No error nothing as warning. For system less than 500 ns it worked fine, but not for more than that.

9

Do you have virtual sites in your systems? If so, I just realized that your crashes might be related to Simulation segfaults with Gromacs 2022 and onwards, fine with 2021 (#4879) · Issues · GROMACS / GROMACS · GitLab. This will be fixed in 2023.3, which will be released in a few weeks.