# MPI error when using ensemble-averaged distance restraints and set GMX\_DISRE\_ENSEMBLE\_SIZE **URL:** \ **Category:** User discussions\ **Tags:** restraints\ **Created:** [January 14, 2026, 2:51pm UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107 "2026-01-14T14:51:36Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![bene304](https://avatars.discourse-cdn.com/v4/letter/b/9de053/32.png) [@bene304](https://gromacs.bioexcel.eu/u/bene304)\ **Post date:** [January 14, 2026, 2:51pm UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/1 "2026-01-14T14:51:37Z")
GROMACS version: 2021.4 GROMACS modification: No Dear Gromacs community, I would like to run an ensemble-averaged restrained MD simulation using NMR distance restraints and 10 replicas for ensemble-averaging. All 10 systems are in subdirectories md01 - md10 and contain the same number of atoms and topology. The distance restraints file (same format as in the manual, starts with index 0 and with distances in nm) is referenced in the topology file and in the mdp file, I enabled them using define = -DDISRES\_NOE disre = ensemble disre\_weighting = equal disre\_mixed = no disre\_fc = 200 disre\_tau = 0 nstdisreout = 5000 I generate a tpr file in each md\* subdirectory using gmx grompp, set the ensemble size with the environment variable `export GMX_DISRE_ENSEMBLE_SIZE="10"` and then start the simulation with ```auto mpirun -np 10 gmx_mpi mdrun -ntomp $OMP_NUM_THREADS -pin on -pinstride 1 -deffnm simulationXY -multidir md0* md1* ``` My problem is that if I set the environment variable `GMX_DISRE_ENSEMBLE_SIZE`, I always get the following MPI error directly at the beginning of the simulation: ```auto [cgpu01-003:03913] *** An error occurred in MPI_Bcast [cgpu01-003:03913] *** reported by process [2687500289,0] [cgpu01-003:03913] *** on communicator MPI_COMM_WORLD [cgpu01-003:03913] *** MPI_ERR_COMM: invalid communicator [cgpu01-003:03913] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, [cgpu01-003:03913] *** and potentially your MPI job) [cgpu01-003:03895] 9 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal [cgpu01-003:03895] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages ``` In general, running Gromacs in parallel using MPI works fine for me, also using multiple replicas set with the -multidir option. Without setting the environment variable `GMX_DISRE_ENSEMBLE_SIZE`, also everything works fine without errors (distance restraints are also recognised), but then also no ensemble-averaging is applied, I think. I already tried out using different nodes (with and without GPU) and other (newer) Gromacs versions, but I always get the same error and cannot explain, what I’m doing wrong. Thank you for your help! ---
**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\ **Post date:** [January 14, 2026, 8:12pm UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/2 "2026-01-14T20:12:16Z")
This works for my test system in the 2025 version. Could you try version 2025? ---
**Author:** ![bene304](https://avatars.discourse-cdn.com/v4/letter/b/9de053/32.png) [@bene304](https://gromacs.bioexcel.eu/u/bene304)\ **Post date:** [January 15, 2026, 5:20pm UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/3 "2026-01-15T17:20:34Z")
Hi, I just tried it also with Gromacs 2025.4, but unfortunately, I still get the same error. ---
**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\ **Post date:** [January 16, 2026, 7:28am UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/4 "2026-01-16T07:28:29Z")
What are the last lines you see in the log file? ---
**Author:** ![bene304](https://avatars.discourse-cdn.com/v4/letter/b/9de053/32.png) [@bene304](https://gromacs.bioexcel.eu/u/bene304)\ **Post date:** [January 16, 2026, 11:02am UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/5 "2026-01-16T11:02:33Z")
The list of input parameters ```auto ... nnpot: active = false modelfile = model.pt input-group = System model-input1 = model-input2 = model-input3 = model-input4 = grpopts: nrdf: 14492.8 198225 ref-t: 300 300 tau-t: 0.1 0.1 annealing: No No annealing-npoints: 0 0 acc: 0 0 0 nfreeze: N N N energygrp-flags[0]: 0 ``` so directly before the distance restraints should be recognised (the following message does NOT anymore appear in the log file): ```auto Initializing the distance restraints Found GMX_DISRE_ENSEMBLE_SIZE set to 10 systems per ensemble Multi-checking the number of systems per ensemble ... OK Our ensemble consists of systems: 0 1 2 3 4 5 6 7 8 9 There are 436 distance restraints involving 1308 atom pairs Multi-checking the number of distance restraints ... OK ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ ... ``` If I don’t set the environment variable `GMX_DISRE_ENSEMBLE_SIZE`, only ```auto Initializing the distance restraints There are 436 distance restraints involving 1308 atom pairs ++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++ ... ``` is written to the log file but the remaining lines are missing. ---
**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\ **Post date:** [January 21, 2026, 7:37am UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/6 "2026-01-21T07:37:51Z")
Can you easily share you tpr file, so I can try to reproduce this? ---
**Author:** ![bene304](https://avatars.discourse-cdn.com/v4/letter/b/9de053/32.png) [@bene304](https://gromacs.bioexcel.eu/u/bene304)\ **Post date:** [January 22, 2026, 5:26pm UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/7 "2026-01-22T17:26:47Z")
Yes, thanks for your help. Strangely, tpr files are no allowed/authorised file types for uploading to the forum via the upload button, so I changed the extension to txt. I hope, I didn’t break any forum rules by doing so. [npt\_1\_posre\_fc\_1000\_disre\_fc\_200\_orire\_fc\_0.1\_change\_extension\_to\_tpr.txt](https://gromacs.bioexcel.eu/uploads/short-url/x0VN8WN1B4gRdoSFtRSjJN7pgwa.txt) (4.5 MB) ---
**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\ **Post date:** [January 23, 2026, 11:40am UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/8 "2026-01-23T11:40:15Z")
I don’t get an MPI error, but a GROMACS error: `Multi-checking the number of fit atoms for orientation restraining ... ` `the number of fit atoms for orientation restraining is not equal for all subsystems ` `subsystem 0: 711` `subsystem 1: 0` `subsystem 2: 0` `subsystem 3: 0` `------------------------------------------------------- ` `Program: gmx mdrun, version 2027.0-dev-20260120-96b77326ef ` `Source file: src/gromacs/mdrunutility/multisim.cpp (line 310) ` `MPI rank: 0 (out of 4) ` `Fatal error: ` `The 4 subsystems are not compatible` I dont’t understand how you get an MPI error. But to fix this I had to add a few lines, listed below. Could you try this? ```auto --- a/src/gromacs/listed_forces/orires.cpp +++ b/src/gromacs/listed_forces/orires.cpp @@ -286,6 +286,10 @@ t_oriresdata::t_oriresdata(FILE* fplog, refCoord -= com; } } + else + { + referenceCoordinates_.resize(fitMasses_.size(), { 0.0_real, 0.0_real, 0.0_real }); + } const size_t numFitAtoms = referenceCoordinates_.size(); xTmp_.resize(numFitAtoms); ``` ---
**Author:** ![bene304](https://avatars.discourse-cdn.com/v4/letter/b/9de053/32.png) [@bene304](https://gromacs.bioexcel.eu/u/bene304)\ **Post date:** [January 26, 2026, 5:28pm UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/9 "2026-01-26T17:28:11Z")
Hi, did you also set the environment variable `GMX_DISRE_ENSEMBLE_SIZE`? I only get the MPI error when I set this environment variable, regardless of whether I only use distance restraints or (in addition to to distance restraints) also apply position and orientation restraints. If I do not set the environment variable, I don’t get an MPI error and both, Gromacs 2021.4 and the modified version of Gromacs 2025.4 (with the fix you suggested) also don’t show any error related to orientation restraints or any other error, but run normal. The unmodified Gromacs 2025.4 version however shows the same error as for you. Here is a tpr file for comparison where only distance restraints are applied: [p38a\_md\_disres\_fc\_1000\_change\_extension\_to\_tpr.txt](https://gromacs.bioexcel.eu/uploads/short-url/twxSdqABcVGrTcKizco9SSWnNGL.txt) (4.0 MB) ---
**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\ **Post date:** [January 27, 2026, 8:48am UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/10 "2026-01-27T08:48:28Z")
I found out that I had trouble passing the env.var. Now I managed and mdrun runs fine for me. I only tried the main branch, which is equal to release-2026 for this functionality. I now also tried release-2025 and there I get your error. So changing to 2026 will solve your issue (and you still need the orires fix for orientation restraints). ---
**Author:** ![bene304](https://avatars.discourse-cdn.com/v4/letter/b/9de053/32.png) [@bene304](https://gromacs.bioexcel.eu/u/bene304)\ **Post date:** [January 27, 2026, 5:30pm UTC](https://gromacs.bioexcel.eu/t/mpi-error-when-using-ensemble-averaged-distance-restraints-and-set-gmx-disre-ensemble-size/13107/11 "2026-01-27T17:30:17Z")
Thank you very much! Now it’s working :)