GROMACS version: 2023.2
GROMACS modification: No
Command line:
Hi,
I compiled that version of gromacs with gcc10.2 (mpi off). cuda version is 11.60. I am trying to run in a single node with 2 x AMD CPU’s, 32 cores total, and 2 RTX8000’s. When I run the following command without using the environment variable GMX_ENABLE_DIRECT_GPU_COMM=1, it runs fine.
gmx mdrun -s topol.tpr -pin on -ntomp 8 -ntmpi 4 -pme gpu -npme 1 -gpu_id 01 -nb gpu -bonded gpu -update gpu
But when I add the above environment command I get the following messages:
Reading file topol.tpr, VERSION 2023.2 (single precision)
On host n0015 2 GPUs selected for this run.
Mapping of GPU IDs to the 4 GPU tasks in the 4 ranks on this node:
PP:0,PP:0,PP:1,PME:1
PP tasks will do (non-perturbed) short-ranged and most bonded interactions on the GPU
PP task will update and constrain coordinates on the GPU
PME tasks will do all aspects on the GPU
GPU direct communication will be used between MPI ranks.
Using 4 MPI threads
Using 8 OpenMP threads per tMPI thread
starting mdrun ‘Title in water’
50000 steps, 100.0 ps.
Program: gmx mdrun, version 2023.2
Program: gmx mdrun, version 2023.2
Source file: src/gromacs/gpu_utils/device_stream.cu (line 100)
Function: DeviceStream::synchronize() const::<lambda()>
MPI rank: 0 (out of 4)
Assertion failed:
Condition: stat == cudaSuccess
Source file: src/gromacs/gpu_utils/device_stream.cu (line 100)
cudaStreamSynchronize failed. CUDA error #700 (cudaErrorIllegalAddress): an
illegal memory access was encountered.
The error message seems too terse to pinpoint the source of the problem. Does anyone has an idea what the problem could be?
Thanks
Alfredo