GPU-binding with CRAY-MPICH causes crash?

User discussions
1

GROMACS version: 2025.3; with CUDA and HeFFTe
GROMACS modification: No
Hello folks,

I’ve been trying to get a large, fairly anisotropic, polymer system up and running on a Cray machine with 4x Nvidia GPUs per node and noticed something strange. When using slurm for the PMI and setting –gpus-per-task=1 -ntasks-per-node=4, I run into the following error:

(GTL DEBUG: 70) cuIpcOpenMemHandle: invalid argument, CUDA_ERROR_INVALID_VALUE, line no 360

Followed by the following MPICH errors (the output below is the result of a test run that had 10 replicates of the system and mdrun was executed with the -multidir flag, hence the large rank number)

MPICH ERROR [Rank 68] [job id 45339999.5] [Tue Nov 18 07:00:50 2025] [nid003229] - Abort(742510594) (rank 68 in comm 0): Fatal error in PMPI_Recv: Invalid count, error stack:
PMPI_Recv(177)…: MPI_Recv(buf=0x7fdc6ce00000, count=3171330, MPI_FLOAT, src=7, tag=0, comm=0x84000005, status=0x1) failed
MPIR_Wait_impl(41)…:
MPID_Progress_wait(201)…:
MPIDI_Progress_test(105)…:
MPIDI_SHMI_progress(118)…:
MPIDI_POSIX_progress(412)…:
MPIDI_CRAY_Common_lmt_ctrl_send_rts_cb(64):
MPIDI_CRAY_Common_lmt_handle_recv(44)…:
MPIDI_CRAY_Common_lmt_import_mem(218)…:
(unknown)(): Invalid count

If I don’t use the –gpus-per-task flag, but instead use –gpus-per-node=4; the simulations run (albeit slower than I would like, but I suspect that is to do with not being able to use the -update gpu flag owing to a non-consecutive constraint).

I will note that I am using MPICH_GPU_SUPPORT_ENABLED=1 to make sure the MPI is GPU aware.

Wasn’t sure if this should be a bug report or if anyone else has run into anything like this on their own systems before.

-Mick

2

Hi,

This looks like a problem in the Cray stack, e.g.,

MPICH_GPU_IPC_ENABLED=0 is claimed to solve the problem, but at a high cost: a pure intra-node run is ~3x slower when I set this env. variable on a four-GPU Cray node (albeit, we don’t have the issue you observe).

Using gpus-per-node=4 is likely better. A wrapper script to set CUDA_VISIBLE_DEVICES=${SLURM_LOCALID} per-rank can give some control over GPU assignment.

slower than I would like, but I suspect that is to do with not being able to use the -update gpu flag owing to a non-consecutive constraint

Two more potential issues:

  • If you are using MPS, even if GROMACS uses one rank per GPU, there could still be big performance impact with some versions NVIDIA driver (we observed with 565.57.01 / 12.7)
  • Cray MPICH does not always announce itself as GPU-aware. You might need to set GMX_FORCE_GPU_AWARE_MPI=1 to make sure GROMACS known that it can use direct GPU communications (although, you crash suggest that things work as-it)
3

Ahh the Cray stack! Thanks for finding that. Right after I saw your response I realized GROMACS was the only program I was running with the –gpus-per-task flag.

And thanks for the tip on the NVIDIA driver, is there a list of problematic drivers floating around that I could use to avoid making a poor choice when using GROMACS with HeFFTe (MPS)?

4

Not that I know of. Probably easier to run the tests yourself.

First, measure the four-rank, single-node performance with MPS. Then, kill MPS with killall nvidia-cuda-mps-control nvidia-cuda-mps-server and run the same benchmark with no MPS; something like srun ./bind_cuda_device.sh gmx_mpi mdrun, where the script is:

#!/bin/bash

NGPU=4
if [ "x${PMIX_RANK}" == "x" ]; then
    RANK_LOCAL=${SLURM_LOCALID:-0}
else
    RANK_LOCAL=${PMIX_RANK}
fi
RANK=$((${RANK_LOCAL} % (${NGPU})))

export CUDA_VISIBLE_DEVICES=${RANK} 
# exec numactl --cpunodebind ${RANK} $*
exec $*