for both molecules, and then used a script to compute the distances over time.
I accounted for PBC effects with -pbc whole, -pbc nojump and, finally, -pbc mol.
Visualizing the simulation box, it seems the molecules are far away, but they are actually close because of PBC effects (see figure). However, the calculated distances are bigger than expected, because the calculation method doesn’t take into account PBC effects (I think).
Do you have any ideas on how to compute the smallest distance between these two solutes?
Where did you use these flags? Because I do not see them in gmx traj, not in my GROMACS 2024 at least.
If you used the flag -pbc nojump in preparing the trajectory (like with gmx trjconv), then you are letting the molecules diffuse around and leave the box. As such, I think you are following the “original” two molecules of solute that you have inside the box, and in principle they will diffuse everywhere, while what you are trying to compute is the minimum distance between a solute and another. With your code you are following the two original molecules and disregarding the “PBC copies” that will enter the box. You can try to visualize the trajectory after you modify it with your flags to be sure about this.
If you want the distance wrt to PBCs, then or you implement the PBCs in your code (i.e. rescale x y and z components for the corresponding length of the box, but this is tedious especially if you are coupling the volume), or as @MagnusL suggested you use one of the built in tools like gmx distance, gmx pairdist, or gmx mindist. The default behaviour of gmx distance has
-[no]rmpbc (yes)
Make molecules whole for each frame
-[no]pbc (yes)
Use periodic boundary conditions for distance calculation
which means that you already consider PBCs and whole molecules, so you do not need post-processing of the trajectory. I am not familiar with these tools, but I think you can work with the centre of masses by specifying -selrpos mol_com and -seltype mol_com or better with the select flag, something like -select "com of group_1 plus com of group_2" where group_1 and group_2 are the two groups defining your molecules of interest in the index file.