Fatal error: Unexpected cudaStreamQuery failure. CUDA error #700 (cudaErrorIllegalAddress)

User discussions
,
1

GROMACS version:2022.3
GROMACS modification: No
Here post your question

I am getting the following error when running gmx mdrun (for equilibrating a lipid+water system) using the GPU. How can I circumvent this?

                  :-) GROMACS - gmx mdrun, 2022.3 (-:

Executable: /usr/local/GROMACS/GROMACS-2022.3-CUDA/bin/gmx
Data prefix: /usr/local/GROMACS/GROMACS-2022.3-CUDA
Working dir: /scratch/psarngadha/30726
Command line:
gmx mdrun -nt 8 -nb gpu -bonded gpu -pme gpu -v -deffnm sm_re_op

Reading file sm_re_op.tpr, VERSION 2022.3 (single precision)
Changing nstlist from 10 to 100, rlist from 1.2 to 1.243

1 GPU selected for this run.
Mapping of GPU IDs to the 2 GPU tasks in the 1 rank on this node:
PP:0,PME:0
PP tasks will do (non-perturbed) short-ranged and most bonded interactions on the GPU
PP task will update and constrain coordinates on the CPU
PME tasks will do all aspects on the GPU
Using 1 MPI thread

Non-default thread affinity set, disabling internal thread affinity

Using 8 OpenMP threads

starting mdrun ‘Thylakoid membrane 303K’
1000000 steps, 1000.0 ps.

step 0
step 100, will finish Thu Aug 17 16:49:15 2023
Step 119, time 0.119 (ps) LINCS WARNING
relative constraint deviation after LINCS:
rms 0.006157, max 1.660899 (between atoms 81075 and 81077)
bonds that rotated more than 30 degrees:
atom 1 atom 2 angle previous, current, constraint length
81075 81077 90.0 0.1111 0.2956 0.1111
81078 81080 90.0 0.1111 0.1957 0.1111
Wrote pdb files with previous and current coordinates

Step 120, time 0.12 (ps) LINCS WARNING
relative constraint deviation after LINCS:
rms 0.011258, max 3.341298 (between atoms 81075 and 81077)
bonds that rotated more than 30 degrees:
atom 1 atom 2 angle previous, current, constraint length
24208 24210 87.8 0.1111 0.1112 0.1111
81075 81077 90.0 0.2956 0.4823 0.1111
81078 81080 77.1 0.1957 0.1110 0.1111
Wrote pdb files with previous and current coordinates

Step 121, time 0.121 (ps) LINCS WARNING
relative constraint deviation after LINCS:
rms 0.128369, max 38.073635 (between atoms 81075 and 81077)
bonds that rotated more than 30 degrees:
atom 1 atom 2 angle previous, current, constraint length
81075 81076 90.0 0.1120 0.2170 0.1111
81075 81077 90.0 0.4823 4.3411 0.1111
81078 81080 90.0 0.1110 0.2202 0.1111
Wrote pdb files with previous and current coordinates

Step 122, time 0.122 (ps) LINCS WARNING
relative constraint deviation after LINCS:
rms 1.022135, max 231.142273 (between atoms 81075 and 81076)
bonds that rotated more than 30 degrees:
atom 1 atom 2 angle previous, current, constraint length
24208 24209 38.3 0.1112 0.1059 0.1111
24208 24210 90.0 0.1111 0.2719 0.1111
81075 81076 104.3 0.2170 25.7910 0.1111
81075 81077 173.6 4.3410 21.9376 0.1111
81078 81080 90.0 0.2202 0.1132 0.1111
Wrote pdb files with previous and current coordinates

Step 123, time 0.123 (ps) LINCS WARNING
relative constraint deviation after LINCS:
rms 4.000144, max 748.196838 (between atoms 123716 and 123717)
bonds that rotated more than 30 degrees:
atom 1 atom 2 angle previous, current, constraint length
24208 24209 90.0 0.1059 0.1573 0.1111
24208 24210 90.0 0.2719 0.2040 0.1111
81078 81080 90.0 0.1132 0.2214 0.1111
117462 117464 90.0 0.1111 0.9011 0.1111
123716 123717 100.4 0.1090 81.6625 0.1090
Wrote pdb files with previous and current coordinates

Program: gmx mdrun, version 2022.3
Source file: src/gromacs/gpu_utils/cudautils.cuh (line 190)

Fatal error:
Unexpected cudaStreamQuery failure. CUDA error #700 (cudaErrorIllegalAddress):
an illegal memory access was encountered.

2

The LINCS warnings after just about 100 steps suggests that perhaps your simulation is unstable. Have you made sure that the systems is well equilibrated? Does it run on CPU only?

3

It is showing segmentation fault when I use CPU. I am looking into the input structure to see if there are any problems . I will try to equilibrate the system agian.