Loading gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5 Loading requirement: gcc/11.3.0 cuda/11.8.0-gcc11.3.0 ucx/1.14.1-gcc11.3.0-cuda11.8.0 openmpi/4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1 ========= COMPUTE-SANITIZER ========= COMPUTE-SANITIZER ========= COMPUTE-SANITIZER ========= COMPUTE-SANITIZER ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:mca_pml_ucx_open [0x642f] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_pml_ucx.so ========= Host Frame:mca_base_framework_components_open [0x5a4ff] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:mca_pml_base_open [0xda017] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:mca_base_framework_open [0x64451] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libopen-pal.so.40 ========= Host Frame:ompi_mpi_init [0xe7424] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit CUDA_ERROR_INVALID_CONTEXT (error 201) due to "invalid device context" on CUDA API call to cuCtxGetDevice. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x2a99d2] ========= in /lib64/libcuda.so.1 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/cuda/base/cuda_iface.c:22:uct_cuda_base_query_devices_common [0x5ef5] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/ucx/libuct_cuda.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/uct/base/uct_md.c:128:uct_md_query_tl_resources [0x12666] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libuct.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1401:ucp_add_component_resources [0x21b09] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:1579:ucp_fill_resources [0x22bbc] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame:/home/x09527a/src/ucx-1.14.1/contrib/../src/ucp/core/ucp_context.c:2011:ucp_init_version [0x23f5d] ========= in /home/x09527a/apps/ucx1.14.1-gcc11.3.0-cuda11.8.0/lib/libucp.so.0 ========= Host Frame: [0x9a8b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/openmpi/mca_osc_ucx.so ========= Host Frame:ompi_osc_base_find_available [0xd6646] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:ompi_mpi_init [0xe758b] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:PMPI_Init_thread [0x7de37] ========= in /home/x09527a/apps/openmpi4.1.5-gcc11.3.0-cuda11.8.0-ucx1.14.1/lib/libmpi.so.40 ========= Host Frame:gmx::init(int*, char***) [0x2eb485] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::initForCommandLine(int*, char***) [0x6c1669] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x56bc] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= :-) GROMACS - gmx mdrun, 2023.1 (-: Executable: /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi Data prefix: /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5 Working dir: ************************************************ Command line: gmx_mpi mdrun -ntomp 10 -v -deffnm step6.9_equilibration -npme 1 -pme gpu -update gpu -nb gpu -bonded gpu -resethway Back Off! I just backed up step6.9_equilibration.log to ./#step6.9_equilibration.log.18# Compiled SIMD: AVX2_256, but for this host/run AVX_512 might be better (see log). Reading file step6.9_equilibration.tpr, VERSION 2023.1 (single precision) GMX_ENABLE_DIRECT_GPU_COMM environment variable detected, enabling direct GPU communication using GPU-aware MPI. Changing nstlist from 20 to 100, rlist from 1.212 to 1.329 On host cx105 2 GPUs selected for this run. Mapping of GPU IDs to the 2 GPU tasks in the 2 ranks on this node: PP:0,PP:1 PP tasks will do (non-perturbed) short-ranged and most bonded interactions on the GPU PP task will update and constrain coordinates on the GPU PME tasks will do all aspects on the GPU GPU direct communication will be used between MPI ranks. Using 4 MPI processes Non-default thread affinity set, disabling internal thread affinity Using 10 OpenMP threads per MPI process Back Off! I just backed up step6.9_equilibration.xtc to ./#step6.9_equilibration.xtc.18# Back Off! I just backed up step6.9_equilibration.trr to ./#step6.9_equilibration.trr.18# Back Off! I just backed up step6.9_equilibration.edr to ./#step6.9_equilibration.edr.18# starting mdrun 'Title' 50000 steps, 100.0 ps. step 0 ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,12) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,13) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,44) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,45) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,20) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,21) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,24) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,25) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,28) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,29) in block (14,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,16) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,17) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,20) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,21) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,8) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,9) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,12) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,13) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,40) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,41) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,24) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,25) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,28) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,29) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,0) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,1) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,4) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,5) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,56) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,57) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,36) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,37) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,48) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,49) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,52) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,53) in block (39,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,0) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,1) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,4) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,5) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,40) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,41) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,44) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,45) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,48) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,49) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,52) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,53) in block (11,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,61) in block (35,0,0) ========= Address 0x2b2794a01dec is out of bounds ========= and is 20 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,41) in block (5,0,0) ========= Address 0x2b2794b59de0 is out of bounds ========= and is 277,213 bytes after the nearest allocation at 0x2b2794a05000 of size 1,118,980 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,41) in block (5,0,0) ========= Address 0x2b2794a01dfc is out of bounds ========= and is 4 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,42) in block (5,0,0) ========= Address 0x2b2794b5a650 is out of bounds ========= and is 279,373 bytes after the nearest allocation at 0x2b2794a05000 of size 1,118,980 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,45) in block (5,0,0) ========= Address 0x2b2794a01da8 is out of bounds ========= and is 88 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,37) in block (35,0,0) ========= Address 0x2b2794a01c9c is out of bounds ========= and is 93 bytes after the nearest allocation at 0x2b2794a00200 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,20) in block (1,0,0) ========= Address 0x2b27954cd1ec is out of bounds ========= and is 465,153 bytes after the nearest allocation at 0x2b2794e00000 of size 6,666,476 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,21) in block (1,0,0) ========= Address 0x2b27954cc97c is out of bounds ========= and is 462,993 bytes after the nearest allocation at 0x2b2794e00000 of size 6,666,476 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,12) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,13) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,16) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,17) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,20) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,21) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,56) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,57) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,37) in block (7,0,0) ========= Address 0x2b2794a01c9c is out of bounds ========= and is 93 bytes after the nearest allocation at 0x2b2794a00200 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,52) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,53) in block (38,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,9) in block (32,0,0) ========= Address 0x2b2794a01c78 is out of bounds ========= and is 57 bytes after the nearest allocation at 0x2b2794a00200 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,13) in block (32,0,0) ========= Address 0x2b2794a01ce0 is out of bounds ========= and is 161 bytes after the nearest allocation at 0x2b2794a00200 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,41) in block (32,0,0) ========= Address 0x2b2794a01da8 is out of bounds ========= and is 88 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,45) in block (32,0,0) ========= Address 0x2b2794a01da8 is out of bounds ========= and is 88 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,37) in block (32,0,0) ========= Address 0x2b2794a01df8 is out of bounds ========= and is 8 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,33) in block (2,0,0) ========= Address 0x2b2794a01d44 is out of bounds ========= and is 188 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,8) in block (42,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,9) in block (42,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,12) in block (42,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,13) in block (42,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,20) in block (42,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,21) in block (42,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,1) in block (31,0,0) ========= Address 0x2b2794a01d34 is out of bounds ========= and is 204 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,5) in block (31,0,0) ========= Address 0x2b2794a01dfc is out of bounds ========= and is 4 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,36) in block (28,0,0) ========= Address 0x2b279558e9e4 is out of bounds ========= and is 464,412 bytes before the nearest allocation at 0x2b2795600000 of size 5,692,032 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,37) in block (28,0,0) ========= Address 0x2b279558e174 is out of bounds ========= and is 466,572 bytes before the nearest allocation at 0x2b2795600000 of size 5,692,032 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,41) in block (3,0,0) ========= Address 0x2b2794a01ccc is out of bounds ========= and is 141 bytes after the nearest allocation at 0x2b2794a00200 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,45) in block (3,0,0) ========= Address 0x2b2794a01c78 is out of bounds ========= and is 57 bytes after the nearest allocation at 0x2b2794a00200 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,8) in block (13,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,9) in block (13,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,61) in block (3,0,0) ========= Address 0x2b2794a01df4 is out of bounds ========= and is 12 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,40) in block (13,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,41) in block (13,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,44) in block (13,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,45) in block (13,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (1,0,52) in block (0,0,0) ========= Address 0x2b279558e9e4 is out of bounds ========= and is 464,412 bytes before the nearest allocation at 0x2b2795600000 of size 5,692,032 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,53) in block (0,0,0) ========= Address 0x2b279558e174 is out of bounds ========= and is 466,572 bytes before the nearest allocation at 0x2b2795600000 of size 5,692,032 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,52) in block (9,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,53) in block (9,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,32) in block (9,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,33) in block (9,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,36) in block (9,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,37) in block (9,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,16) in block (17,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,17) in block (17,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,20) in block (17,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,21) in block (17,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,40) in block (10,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,41) in block (10,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,44) in block (10,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Invalid __global__ read of size 4 bytes ========= at 0x6a0 in void pme_spline_and_spread_kernel<(int)4, (bool)1, (bool)1, (bool)1, (bool)1, (int)1, (bool)0, (ThreadsPerAtom)0>(PmeGpuCudaKernelParams) ========= by thread (0,0,45) in block (10,0,0) ========= Address 0x2b2794a01d28 is out of bounds ========= and is 216 bytes before the nearest allocation at 0x2b2794a01e00 of size 6,720 bytes ========= Saved host backtrace up to driver entry point at kernel launch time ========= Host Frame: [0x302a52] ========= in /lib64/libcuda.so.1 ========= Host Frame:__cudart819 [0x112b6bb] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:cudaLaunchKernel [0x11889c8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_spread(PmeGpu const*, GpuEventSynchronizer*, float**, gmx_parallel_3dfft**, bool, bool, float, bool, gmx::PmeCoordinateReceiverGpu*, gmx_wallcycle*) [0xf9728a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:pme_gpu_launch_spread(gmx_pme_t*, GpuEventSynchronizer*, gmx_wallcycle*, float, bool, gmx::PmeCoordinateReceiverGpu*) [0xe16e48] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xdfff34] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventSynchronize. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventSynchronize [0x116a101] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::PmeForceSenderGpu::Impl::sendFToPpGpuAwareMpi(gmx::BasicVector*, int, int, int, ompi_request_t**) [0xf8b55d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [0xe0042a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a31] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a31] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a31] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a49] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a49] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a49] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a61] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a61] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a61] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a79] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a79] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a79] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a91] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a91] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4a91] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4aad] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4aad] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::Impl::~Impl() [0xfa4aad] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::StatePropagatorDataGpu::~StatePropagatorDataGpu() [0xfa4e21] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b829] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaEventDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaEventDestroy [0x116a2a1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::PmeForceSenderGpu::Impl::~Impl() [0xf8b43a] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::PmeForceSenderGpu::~PmeForceSenderGpu() [0xf8b4d1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:std::unique_ptr >::~unique_ptr() [0xe00e47] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b7e5] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified launch failure" on CUDA API call to cudaStreamDestroy. ========= Saved host backtrace up to driver entry point at error ========= Host Frame: [0x43ec86] ========= in /lib64/libcuda.so.1 ========= Host Frame:cudaStreamDestroy [0x11680d8] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:DeviceStream::~DeviceStream() [0xf9d308] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::PmeForceSenderGpu::Impl::~Impl() [0xf8b45c] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::PmeForceSenderGpu::~PmeForceSenderGpu() [0xf8b4d1] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:std::unique_ptr >::~unique_ptr() [0xe00e47] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx_pmeonly(gmx_pme_t**, t_commrec const*, t_nrnb*, gmx_wallcycle*, gmx_walltime_accounting*, t_inputrec*, PmeRunMode, bool, gmx::DeviceStreamManager const*) [clone .cold] [0x26b7e5] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::Mdrunner::mdrunner() [0xe6d567] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:gmx::gmx_mdrun(ompi_communicator_t*, gmx_hw_info_t const&, int, char**) [0x8dc3] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::gmx_mdrun(int, char**) [0x8f0d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:gmx::CommandLineModuleManager::run(int, char**) [0x6c3f23] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/lib64/libgromacs_mpi.so.8 ========= Host Frame:main [0x571d] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= Host Frame:__libc_start_main [0x22505] ========= in /lib64/libc.so.6 ========= Host Frame: [0x5784] ========= in /home/x09527a/apps/gromacs/2023.1-gcc11.3.0-cuda11.8.0-ucx-1.14.1-mpi4.1.5/bin/gmx_mpi ========= ------------------------------------------------------- Program: gmx mdrun, version 2023.1 Source file: src/gromacs/gpu_utils/device_stream.cu (line 81) Function: DeviceStream::~DeviceStream():: MPI rank: 3 (out of 4) Assertion failed: Condition: stat == cudaSuccess Failed to release CUDA stream. CUDA error #719 (cudaErrorLaunchFailure): unspecified launch failure. For more information and tips for troubleshooting, please check the GROMACS website at http://www.gromacs.org/Documentation/Errors ------------------------------------------------------- -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- ========= Target application returned an error ========= ERROR SUMMARY: 135 errors