gmx_mpi mdrun cuda error #700

GROMACS version: 2023
GROMACS modification: No
Hi there all,

I have installed GROMACS 2023 with CUDA 12.1 on Centos 7.9 HPC system.

When I try to call these steps below, I get error about CUDA:

$ gmx_mpi grompp -f nvt.mdp -c em.gro -r em.gro -p sol2.top -o nvt.tpr

$ gmx_mpi mdrun -deffnm nvt

Error message is below:

Program: gmx mdrun, version 2023
Source file: src/gromacs/gpu_utils/cudautils.cuh (line 190)

Fatal error:
Unexpected cudaStreamQuery failure. CUDA error #700 (cudaErrorIllegalAddress):
an illegal memory access was encountered.

For more information and tips for troubleshooting, please check the GROMACS
website at Common Errors — GROMACS webpage https://www.gromacs.org documentation

Can anybody help about this error?

Thanks,

Best regards,