# Another MPI rank encountered an exception

**URL:** <https://gromacs.bioexcel.eu/t/another-mpi-rank-encountered-an-exception/1238>\
**Category:** User discussions\
**Created:** [December 9, 2020, 9:54pm UTC](https://gromacs.bioexcel.eu/t/another-mpi-rank-encountered-an-exception/1238 "2020-12-09T21:54:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dkotzamp](https://avatars.discourse-cdn.com/v4/letter/d/bbce88/32.png) [@dkotzamp](https://gromacs.bioexcel.eu/u/dkotzamp)\
**Post date:** [December 9, 2020, 9:54pm UTC](https://gromacs.bioexcel.eu/t/another-mpi-rank-encountered-an-exception/1238/1 "2020-12-09T21:54:45Z")

</div>

GROMACS version: 2020.2  
GROMACS modification: Yes/No  
Command line: srun gmx\_mpi mdrun -maxh 24 -deffnm XXXX -cpi XXXX.cpt -append  
and the above command line fails and returns me the following errors:

Program: gmx mdrun, version 2020.2  
Source file: src/gromacs/mdrunutility/handlerestart.cpp (line 681)  
Function: std::tuple\<gmx::StartingBehavior, std::unique\_ptr\<t\_fileio, gmx::functor\_wrapper\<t\_fileio, gmx::closeLogF  
ile\> \> \> gmx::handleRestart(bool, MPI\_Comm, const gmx\_multisim\_t\*, gmx::AppendingBehavior, int, t\_filenm\*)  
MPI rank: 123 (out of 160)

Communication (parallel processing) problem:

* * *

Program: gmx mdrun, version 2020.2  
Source file: src/gromacs/mdrunutility/handlerestart.cpp (line 681)  
Function: std::tuple\<gmx::StartingBehavior, std::unique\_ptr\<t\_fileio, gmx::functor\_wrapper\<t\_fileio, gmx::closeLogF  
ile\> \> \> gmx::handleRestart(bool, MPI\_Comm, const gmx\_multisim\_t\*, gmx::AppendingBehavior, int, t\_filenm\*)  
MPI rank: 43 (out of 160)

Communication (parallel processing) problem:

* * *

* * *

Program: gmx mdrun, version 2020.2  
Source file: src/gromacs/mdrunutility/handlerestart.cpp (line 681)  
Function: std::tuple\<gmx::StartingBehavior, std::unique\_ptr\<t\_fileio, gmx::functor\_wrapper\<t\_fileio, gmx::closeLogF  
ile\> \> \> gmx::handleRestart(bool, MPI\_Comm, const gmx\_multisim\_t\*, gmx::AppendingBehavior, int, t\_filenm\*)  
MPI rank: 126 (out of 160)

…etc

application called MPI\_Abort(MPI\_COMM\_WORLD, 1) - process 9  
slurmstepd: error: \*\*\* STEP 1058046.0 ON node033 CANCELLED AT 2020-12-09T23:24:43 \*\*\*  
srun: error: node146: tasks 40-59: Killed  
srun: Terminating job step 1058046.0  
srun: error: node033: tasks 0-19: Killed  
srun: error: node271: tasks 120-139: Killed  
srun: error: node160: tasks 60-79: Killed  
srun: error: node145: tasks 20-39: Killed  
srun: error: node272: tasks 140-159: Killed  
srun: error: node191: tasks 100-119: Killed

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![lijinhui131](https://avatars.discourse-cdn.com/v4/letter/l/dfb087/32.png) [@lijinhui131](https://gromacs.bioexcel.eu/u/lijinhui131)\
**Post date:** [April 5, 2021, 5:43pm UTC](https://gromacs.bioexcel.eu/t/another-mpi-rank-encountered-an-exception/1238/2 "2021-04-05T17:43:41Z")

</div>

Me too, I also encountered this problem. Have you got the answer? Thank you so much in advance.

---

<div class="post-metadata">

**Author:** ![ricalessandri](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@ricalessandri](https://gromacs.bioexcel.eu/u/ricalessandri)\
**Post date:** [June 19, 2021, 3:47pm UTC](https://gromacs.bioexcel.eu/t/another-mpi-rank-encountered-an-exception/1238/3 "2021-06-19T15:47:25Z")

</div>

In my case, I was getting this error with GROMACS/2020.3 when re-starting from a checkpoint file. Using a different checkpoint file (run.cpt instead of run\_prev.cpt in my case) solved it. Maybe this helps.

---

<div class="post-metadata">

**Author:** ![sofki](https://avatars.discourse-cdn.com/v4/letter/s/839c29/32.png) [@sofki](https://gromacs.bioexcel.eu/u/sofki)\
**Post date:** [October 12, 2022, 12:33pm UTC](https://gromacs.bioexcel.eu/t/another-mpi-rank-encountered-an-exception/1238/4 "2022-10-12T12:33:53Z")

</div>

I also have this problem and using a different -cpi file.cpt option didn’t help. Is there any explanation why this problem occurs and/or a global solution for fixing it? Thanks!

---

<div class="post-metadata">

**Author:** ![sofki](https://avatars.discourse-cdn.com/v4/letter/s/839c29/32.png) [@sofki](https://gromacs.bioexcel.eu/u/sofki)\
**Post date:** [November 2, 2022, 12:20pm UTC](https://gromacs.bioexcel.eu/t/another-mpi-rank-encountered-an-exception/1238/5 "2022-11-02T12:20:50Z")

</div>

Well, the actual error is printed a few lines upper:  
Inconsistency in user input:  
Some output files listed in the checkpoint file umbrella94.cpt are not present  
or not named as the output files by the current program:)  
Expected output files that are present:  
umbrella94.log  
umbrella94.xtc  
umbrella94.edr

Expected output files that are not present or named differently:  
umbrella94\_pullx.xvg  
umbrella94\_pullf.xvg

So, the problem was either that you included the -cpi option but not the -append or that you did include -append but some files were named differently (my case here). In order to fix it, you should either fiox the naming of the files, or include -noappend.