Segrmentation faul or processor bus error

GROMACS version: 2023
GROMACS modification: Yes/No
Here post your question

Hello gromacs users,

I installed gromacs on ubuntu 22.10 using cuda 12.1 and Nvidia driver version: 530.30.02 by command:

cmake … -DGMX_GPU=CUDA -DGMX_PREFER_STATIC_LIBS=ON -DGMXAPI=OFF -DGMX_BUILD_OWN_FFTW=ON -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda -DCMAKE_INSTALL_PREFIX=/home/xxx/progs/gromacs_2023 -DGMX_DEFAULT_SUFFIX=OFF -DGMX_BINARY_SUFFIX=_2023 -DGMX_LIBS_SUFFIX=_2023

My machine configuration is Intel I9 13900KF with one Gigabyte RTX 4090 and 64 GB DDR5.
The installation finished without any errors, however when I run my simulation using command:
gmx mdrun -v -bonded gpu -pme gpu -nb gpu -s topol.tpr

I always get two different errors:
Segmentation error (memory image flushed to disk) or processor bus error (memory image flushed to disk)

How do you think, why these errors could arise?

Best regards
Yel21

That should not happen.

Does this happen during initialization or after some/many MD steps?
What are the last few lines in your md.log file?

Hi hess,

This happen during initialization, actually I build gromacs 2022.2 with same flags for cmake and simulation started without any errors.

Below my md.log for simulation using gromacs 2023

                  :-) GROMACS - gmx mdrun, 2023 (-:

Copyright 1991-2023 The GROMACS Authors.
GROMACS is free software; you can redistribute it and/or modify it
under the terms of the GNU Lesser General Public License
as published by the Free Software Foundation; either version 2.1
of the License, or (at your option) any later version.

                     Current GROMACS contributors:
   Mark Abraham           Andrey Alekseenko           Cathrine Bergh      
  Christian Blau            Eliane Briand             Mahesh Doijade      
Stefan Fleischmann           Vytas Gapsys              Gaurav Garg        
  Sergey Gorelov         Gilles Gouaillardet            Alan Gray         
 M. Eric Irrgang         Farzaneh Jalalypour            Joe Jordan        
Christoph Junghans        Prashanth Kanduri          Sebastian Keller     
 Carsten Kutzner           Justin A. Lemkul          Magnus Lundborg      
   Pascal Merz              Vedran Miletic            Dmitry Morozov      
   Szilard Pall             Roland Schulz             Michael Shirts      
 Alexey Shvetsov            Balint Soproni         David van der Spoel    
  Philip Turner             Carsten Uphoff           Alessandra Villa     

Sebastian Wingbermuehle Artem Zhmurov

                     Previous GROMACS contributors:
    Emile Apol             Rossen Apostolov           James Barnett       

Herman J.C. Berendsen Par Bjelkmar Viacheslav Bolnykh
Kevin Boyd Aldert van Buuren Carlo Camilloni
Rudi van Drunen Anton Feenstra Oliver Fleetwood
Gerrit Groenhof Bert de Groot Anca Hamuraru
Vincent Hindriksen Victor Holanda Aleksei Iupinov
Dimitrios Karkoulis Peter Kasson Sebastian Kehl
Jiri Kraus Per Larsson Viveca Lindahl
Erik Marklund Pieter Meulenhoff Teemu Murtola
Sander Pronk Alfons Sijbers Peter Tieleman
Jon Vincent Teemu Virolainen Christian Wennberg
Maarten Wolf

              Coordinated by the GROMACS project leaders:
                Paul Bauer, Berk Hess, and Erik Lindahl

GROMACS: gmx mdrun, version 2023
Executable: /home/xxx/progs/gromacs_2023/bin/gmx_2023
Data prefix: /home/xxx/progs/gromacs_2023
Working dir: /media/xxx/20287bc1-0b40-4b78-b2a8-658638afd0f0/work/run/bp/test
Process ID: 512347
Command line:
gmx_2023 mdrun -v -deffnm def800 -bonded gpu -pme gpu -nb gpu -s topol.tpr

GROMACS version: 2023
Precision: mixed
Memory model: 64 bit
MPI library: thread_mpi
OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support: CUDA
NB cluster size: 8
SIMD instructions: AVX2_256
CPU FFT library: fftw-3.3.8-sse2-avx-avx2-avx2_128
GPU FFT library: cuFFT
Multi-GPU FFT: none
RDTSCP usage: enabled
TNG support: enabled
Hwloc support: disabled
Tracing support: disabled
C compiler: /usr/bin/cc GNU 11.3.0
C compiler flags: -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler: /usr/bin/c++ GNU 11.3.0
C++ compiler flags: -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp -O3 -DNDEBUG
BLAS library:
LAPACK library:
CUDA compiler: /usr/local/cuda/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2023 NVIDIA Corporation;Built on Tue_Feb__7_19:32:13_PST_2023;Cuda compilation tools, release 12.1, V12.1.66;Build cuda_12.1.r12.1/compiler.32415258_0
CUDA compiler flags:-std=c++17;–generate-code=arch=compute_50,code=sm_50;–generate-code=arch=compute_52,code=sm_52;–generate-code=arch=compute_60,code=sm_60;–generate-code=arch=compute_61,code=sm_61;–generate-code=arch=compute_70,code=sm_70;–generate-code=arch=compute_75,code=sm_75;–generate-code=arch=compute_80,code=sm_80;–generate-code=arch=compute_86,code=sm_86;–generate-code=arch=compute_89,code=sm_89;–generate-code=arch=compute_90,code=sm_90;-Wno-deprecated-gpu-targets;–generate-code=arch=compute_53,code=sm_53;–generate-code=arch=compute_80,code=sm_80;-use_fast_math;-Xptxas;-warn-double-usage;-Xptxas;-Werror;-D_FORCE_INLINES;-fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp -O3 -DNDEBUG
CUDA driver: 12.10
CUDA runtime: 12.10

Running on 1 node with total 24 cores, 32 processing units, 1 compatible GPU
Hardware detected on host zeus:
CPU info:
Vendor: Intel
Brand: 13th Gen Intel(R) Core™ i9-13900KF
Family: 6 Model: 183 Stepping: 1
Features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt intel lahf mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sha sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
Hardware topology: Basic
Packages, cores, and logical processors:
[indices refer to OS logical processors]
Package 0: [ 0 1] [ 2 3] [ 4 5] [ 6 7] [ 8 9] [ 10 11] [ 12 13] [ 14 15] [ 16] [ 17] [ 18] [ 19] [ 20] [ 21] [ 22] [ 23] [ 24] [ 25] [ 26] [ 27] [ 28] [ 29] [ 30] [ 31]
CPU limit set by OS: -1 Recommended max number of threads: 32
GPU info:
Number of GPUs detected: 1
#0: NVIDIA NVIDIA GeForce RTX 4090, compute cap.: 8.9, ECC: no, stat: compatible

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
M. J. Abraham, T. Murtola, R. Schulz, S. Páll, J. C. Smith, B. Hess, E.
Lindahl
GROMACS: High performance molecular simulations through multi-level
parallelism from laptops to supercomputers
SoftwareX 1 (2015) pp. 19-25
-------- -------- — Thank You — -------- --------

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
S. Páll, M. J. Abraham, C. Kutzner, B. Hess, E. Lindahl
Tackling Exascale Software Challenges in Molecular Dynamics Simulations with
GROMACS
In S. Markidis & E. Laure (Eds.), Solving Software Challenges for Exascale 8759 (2015) pp. 3-27
-------- -------- — Thank You — -------- --------

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
S. Pronk, S. Páll, R. Schulz, P. Larsson, P. Bjelkmar, R. Apostolov, M. R.
Shirts, J. C. Smith, P. M. Kasson, D. van der Spoel, B. Hess, and E. Lindahl
GROMACS 4.5: a high-throughput and highly parallel open source molecular
simulation toolkit
Bioinformatics 29 (2013) pp. 845-54
-------- -------- — Thank You — -------- --------

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
B. Hess and C. Kutzner and D. van der Spoel and E. Lindahl
GROMACS 4: Algorithms for highly efficient, load-balanced, and scalable
molecular simulation
J. Chem. Theory Comput. 4 (2008) pp. 435-447
-------- -------- — Thank You — -------- --------

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
D. van der Spoel, E. Lindahl, B. Hess, G. Groenhof, A. E. Mark and H. J. C.
Berendsen
GROMACS: Fast, Flexible and Free
J. Comp. Chem. 26 (2005) pp. 1701-1719
-------- -------- — Thank You — -------- --------

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
E. Lindahl and B. Hess and D. van der Spoel
GROMACS 3.0: A package for molecular simulation and trajectory analysis
J. Mol. Mod. 7 (2001) pp. 306-317
-------- -------- — Thank You — -------- --------

++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++
H. J. C. Berendsen, D. van der Spoel and R. van Drunen
GROMACS: A message-passing parallel molecular dynamics implementation
Comp. Phys. Comm. 91 (1995) pp. 43-56
-------- -------- — Thank You — -------- --------

++++ PLEASE CITE THE DOI FOR THIS VERSION OF GROMACS ++++

-------- -------- — Thank You — -------- --------

Input Parameters:
integrator = md
tinit = 0
dt = 0.002
nsteps = 50000000
init-step = 0
simulation-part = 1
mts = false
comm-mode = Linear
nstcomm = 10
bd-fric = 0
ld-seed = -1166540811
emtol = 10
emstep = 0.01
niter = 20
fcstep = 0
nstcgsteep = 1000
nbfgscorr = 10
rtpi = 0.05
nstxout = 0
nstvout = 0
nstfout = 0
nstlog = 0
nstcalcenergy = 100
nstenergy = 10000
nstxout-compressed = 1000
compressed-x-precision = 1000
cutoff-scheme = Verlet
nstlist = 10
pbc = xyz
periodic-molecules = false
verlet-buffer-tolerance = 0.005
rlist = 0.9
coulombtype = PME
coulomb-modifier = Potential-shift
rcoulomb-switch = 0
rcoulomb = 0.9
epsilon-r = 1
epsilon-rf = inf
vdw-type = Cut-off
vdw-modifier = Potential-shift
rvdw-switch = 0
rvdw = 0.9
DispCorr = Ener
table-extension = 1
fourierspacing = 0.12
fourier-nx = 56
fourier-ny = 56
fourier-nz = 56
pme-order = 4
ewald-rtol = 1e-05
ewald-rtol-lj = 0.001
lj-pme-comb-rule = Geometric
ewald-geometry = 3d
epsilon-surface = 0
ensemble-temperature-setting = constant
ensemble-temperature = 800
tcoupl = V-rescale
nsttcouple = 10
nh-chain-length = 0
print-nose-hoover-chain-variables = false
pcoupl = Parrinello-Rahman
pcoupltype = Isotropic
nstpcouple = 10
tau-p = 5
compressibility (3x3):
compressibility[ 0]={ 4.50000e-05, 0.00000e+00, 0.00000e+00}
compressibility[ 1]={ 0.00000e+00, 4.50000e-05, 0.00000e+00}
compressibility[ 2]={ 0.00000e+00, 0.00000e+00, 4.50000e-05}
ref-p (3x3):
ref-p[ 0]={ 1.00000e+00, 0.00000e+00, 0.00000e+00}
ref-p[ 1]={ 0.00000e+00, 1.00000e+00, 0.00000e+00}
ref-p[ 2]={ 0.00000e+00, 0.00000e+00, 1.00000e+00}
refcoord-scaling = No
posres-com (3):
posres-com[0]= 0.00000e+00
posres-com[1]= 0.00000e+00
posres-com[2]= 0.00000e+00
posres-comB (3):
posres-comB[0]= 0.00000e+00
posres-comB[1]= 0.00000e+00
posres-comB[2]= 0.00000e+00
QMMM = false
qm-opts:
ngQM = 0
constraint-algorithm = Lincs
continuation = false
Shake-SOR = false
shake-tol = 0.0001
lincs-order = 4
lincs-iter = 1
lincs-warnangle = 30
nwall = 0
wall-type = 9-3
wall-r-linpot = -1
wall-atomtype[0] = -1
wall-atomtype[1] = -1
wall-density[0] = 0
wall-density[1] = 0
wall-ewald-zfac = 3
pull = false
awh = false
rotation = false
interactiveMD = false
disre = No
disre-weighting = Conservative
disre-mixed = false
dr-fc = 1000
dr-tau = 0
nstdisreout = 100
orire-fc = 0
orire-tau = 0
nstorireout = 100
free-energy = no
cos-acceleration = 0
deform (3x3):
deform[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}
deform[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}
deform[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}
simulated-tempering = false
swapcoords = no
userint1 = 0
userint2 = 0
userint3 = 0
userint4 = 0
userreal1 = 0
userreal2 = 0
userreal3 = 0
userreal4 = 0
applied-forces:
electric-field:
x:
E0 = 0
omega = 0
t0 = 0
sigma = 0
y:
E0 = 0
omega = 0
t0 = 0
sigma = 0
z:
E0 = 0
omega = 0
t0 = 0
sigma = 0
density-guided-simulation:
active = false
group = protein
similarity-measure = inner-product
atom-spreading-weight = unity
force-constant = 1e+09
gaussian-transform-spreading-width = 0.2
gaussian-transform-spreading-range-in-multiples-of-width = 4
reference-density-filename = reference.mrc
nst = 1
normalize-densities = true
adaptive-force-scaling = false
adaptive-force-scaling-time-constant = 4
shift-vector =
transformation-matrix =
qmmm-cp2k:
active = false
qmgroup = System
qmmethod = PBE
qmfilenames =
qmcharge = 0
qmmultiplicity = 1
grpopts:
nrdf: 47625
ref-t: 800
tau-t: 1
annealing: No
annealing-npoints: 0
acc: 0 0 0
nfreeze: N N N
energygrp-flags[ 0]: 0

Could you test what happens if you run cpu only (-nb cpu)?

The same error happens:

gmx_2023 mdrun -v -deffnm BPdef800 -nb cpu -s topol.tpr
:-) GROMACS - gmx mdrun, 2023 (-:

Executable: /home/xxx/progs/gromacs_2023/bin/gmx_2023
Data prefix: /home/xxx/progs/gromacs_2023
Working dir: /home/xxx/work/run
Command line:
gmx_2023 mdrun -v -deffnm BPdef800 -nb cpu -s topol.tpr

Back Off! I just backed up BPdef800.log to ./#BPdef800.log.18#
Reading file topol.tpr, VERSION 2022 (single precision)
Note: file tpx version 127, software tpx version 129
Ошибка шины (образ памяти сброшен на диск)

Sorry, for Russian, this mean “Bus error (memory image dumped to disk)”.

This looks like the issue could be happening during reading of the tpr file. Can you share the tpr file somewhere?