Appropriate box size for protein that changes conformation

GROMACS version: 2023.3
GROMACS modification: No
I run a molecular dynamics simulation for protein-protein complex in dodecahedron box. The trajectory was managed using gmx trjconv command with -pbc mol -center -ur compact options. Centering was done by one of the protein chain in order to prevent jumps. Trajectory was visualized and rmsd was calculated.
The conformation of the complex changes significantly during simulation and the protein complex extends, which causes large rmsd values ​​(1.0-2.0 nm). Could it be necessary to increase the size of the box in such a case and how can I determine that the box is not large enough for the new conformation of the complex?

In this case, the end-to-end distance may be a more useful metric than the RMSD. Check your maximum end-to-end distance and set the box size to be 2.0 nm larger than that.

As a general rule of thumb, you want at least 2.0 nm distance between PBC images, so at least a 1.0 nm distance between the edge of the protein and the box.

Thank you for your feedback. I have a cutoff of 1.2 nm for protein-protein interactions according to mdp parameters. Should I keep the minimum distance between PBC images at more than 1.2 nm, or is 2.0 nm between them necessary?

I don’t know about the 2.0 nm being strictly necessary, but it’s the rule of thumb to be sure that the images across the PBC won’t affect each other. From my understanding, just being above the cutoff is not enough since elements further than the cutoff will still interact via long-range calculation (PME for example)