Heterogeneous mpirun with unequal number of GPUs

GROMACS version: 2024.3
GROMACS modification: Yes - Plumed
I just got my two desktops communicating well enough to run mpi gromacs-plumed hrex. However the two machines are not the same and I can’t figure out how to get full use of resources.

Both machines are running Ubuntu 22. One has an I7 cpu with 8 P cores and 4 E cores. The other has a threadripper 1920x with 12 cores. The bigger problem is that one machine has one RTX 30 series and one RTX 40 series while the second machine has 4 RTX 40 series.

I have 24 replicas and if I give a minimal mdrun command without specifying gpu ids or tasks, only one gpu gets used on the machine issuing the command.

If I specify GPU ids, I get an error since the second machine has no mapping to GPU ID 2 or greater. I apparently can’t change the GPU IDs on the remote machine so that I can set -gpu_id to 0,1,2,3,4,5.

How do I get mdrun to recognize and use all available GPUs?

Also, since I need the CPU physical cores rather than threads to get good performance, I don’t know how I get the 2 GPUs on the remote machine to just do 1/3 of the work since each machine has the same number of cpu cores and mpirun -np 24 is setting up 12 processes on each machine.

As a final note, I tried running on one machine by oversubscribing the threadripper to 24 processes with the 4 GPUs but the performance was really poor.