I have a feeling that there is some bug in handling of the pull distances.
I have written about it here Problem with "Distance between pull groups" and PBC and submitted an issue here https://gitlab.com/gromacs/gromacs/-/issues/3613 but apparently the developers were not able to look into it yet.