Loading gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5 Loading requirement: gcc/11.3.0 cuda/11.8.0-gcc11.3.0 ucx/1.14.1-gcc11.3.0-cuda11.8.0 openmpi/4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1 ========= COMPUTE-SANITIZER ========= COMPUTE-SANITIZER ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Saved host backtrace up to driver entry point at error ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= :-) GROMACS - gmx mdrun, 2023.1 (-: Executable: /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi Data prefix: /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5 Working dir: ************************************************** Command line: gmx_mpi mdrun -ntomp 10 -v -deffnm step6.9_equilibration -npme 1 -pme gpu -update gpu -nb gpu -bonded gpu -resethway Back Off! I just backed up step6.9_equilibration.log to ./#step6.9_equilibration.log.16# Compiled SIMD: AVX2_256, but for this host/run AVX_512 might be better (see log). Reading file step6.9_equilibration.tpr, VERSION 2023.1 (single precision) GMX_ENABLE_DIRECT_GPU_COMM environment variable detected, enabling direct GPU communication using GPU-aware MPI. Changing nstlist from 20 to 100, rlist from 1.212 to 1.329 On host cx098 1 GPU selected for this run. Mapping of GPU IDs to the 1 GPU task in the 1 rank on this node: PP:0 PP tasks will do (non-perturbed) short-ranged and most bonded interactions on the GPU PP task will update and constrain coordinates on the GPU PME tasks will do all aspects on the GPU GPU direct communication will be used between MPI ranks. Using 2 MPI processes Non-default thread affinity set, disabling internal thread affinity Using 10 OpenMP threads per MPI process Back Off! I just backed up step6.9_equilibration.xtc to ./#step6.9_equilibration.xtc.16# Back Off! I just backed up step6.9_equilibration.trr to ./#step6.9_equilibration.trr.16# Back Off! I just backed up step6.9_equilibration.edr to ./#step6.9_equilibration.edr.16# starting mdrun 'Title' 50000 steps, 100.0 ps. step 0 step 100, will finish Fri Jun 16 15:36:41 2023 step 200, will finish Fri Jun 16 15:35:10 2023 step 300, will finish Fri Jun 16 15:34:41 2023 step 400, will finish Fri Jun 16 15:34:27 2023 step 600: timed with pme grid 108 108 120, coulomb cutoff 1.200: 39515.8 M-cycles step 800: timed with pme grid 100 100 108, coulomb cutoff 1.302: 47480.8 M-cycles step 1000: timed with pme grid 100 100 112, coulomb cutoff 1.277: 45450.8 M-cycles step 1200: timed with pme grid 104 104 112, coulomb cutoff 1.255: 43806.9 M-cycles step 1400: timed with pme grid 104 104 120, coulomb cutoff 1.228: 41730.3 M-cycles step 1600: timed with pme grid 108 108 120, coulomb cutoff 1.200: 39635.1 M-cycles optimal pme grid 108 108 120, coulomb cutoff 1.200