GROMACS version: 2023.5
GROMACS modification: first tried plain GROMACS, then modifiedgmxManageNvccConfig.cmake
I am trying to build an executable of GROMACS 2023.5 that can run on
NVIDIA GeForce RTX 5070Ti cards, but so far all of my attempts failed. If anyone managed to do so, I am interested in the settings.
Here’s what I did so far:
First used an executable built with CUDA 12.2 and GCC 12.2
When executing mdrun, I get:
WARNING: An error occurred while sanity checking device #0. An unhandled error from a previous CUDA operation was detected. CUDA error #209 (cudaErrorNoKernelImageForDevice): no kernel image is available for execution on the device.
And the simulation continues to run, but using only the CPU.
Then I tried to fix JIT compilation
Believing the reason for this is this issue: “CUDA forward-compatibility broken”
I went ahead and back-ported the suggested fix to gmxManageNvccConfig.cmake in lines 225-226:
Old:
gmx_add_nvcc_flag_if_supported(GMX_CUDA_NVCC_GENCODE_FLAGS NVCC_HAS_GENCODE_COMPUTE_53 --generate-code=arch=compute_53,code=sm_53)
gmx_add_nvcc_flag_if_supported(GMX_CUDA_NVCC_GENCODE_FLAGS NVCC_HAS_GENCODE_COMPUTE_80 --generate-code=arch=compute_80,code=sm_80)
Now:
gmx_add_nvcc_flag_if_supported(GMX_CUDA_NVCC_GENCODE_FLAGS NVCC_HAS_GENCODE_COMPUTE_53 --generate-code=arch=compute_53,code=compute_53)
gmx_add_nvcc_flag_if_supported(GMX_CUDA_NVCC_GENCODE_FLAGS NVCC_HAS_GENCODE_COMPUTE_80 --generate-code=arch=compute_80,code=compute_80)
Now after compiling I get the following error:
1 GPU selected for this run.
Mapping of GPU IDs to the 2 GPU tasks in the 1 rank on this node:
PP:0,PME:0
PP tasks will do (non-perturbed) short-ranged interactions on the GPU
PP task will update and constrain coordinates on the CPU
PME tasks will do all aspects on the GPU
Using 1 MPI thread
Using 16 OpenMP threads
starting mdrun ‘Great Red Oystrich Makes All Chemists Sane’
10000 steps, 20.0 ps.
Program: gmx mdrun, version 2023.5-dev-2025-10-14–unknown
Source file: src/gromacs/gpu_utils/cudautils.cuh (line 292)
Function: launchGpuKernel<NBAtomDataGpu, NBParamGpu, Nbnxm::gpu_plist, int>(void ()(NBAtomDataGpu, NBParamGpu, Nbnxm::gpu_plist, int), const KernelLaunchConfig&, const DeviceStream&, CommandEvent, const char*, const std::array<void*, 4>&)::<lambda()>
Assertion failed:
Condition: stat == cudaSuccess
GPU kernel (k_pruneonly) failed to launch: CUDA error #218
(cudaErrorInvalidPtx): a PTX JIT compilation failed.
Same with CUDA 12.8 btw.
Third, I included Blackwell architectures as well
To circumvent having to JIT compile for Blackwell, I tried to add cmake
code to specifically build for Blackwell as well. Using CUDA 12.8, GCC 12.2.
In gmxManageNvccConfig.cmake, adding:
gmx_add_nvcc_flag_if_supported(GMX_CUDA_NVCC_GENCODE_FLAGS NVCC_HAS_GENCODE_COMPUTE_AND_SM_100 --generate-code=arch=compute_100,code=sm_100)
gmx_add_nvcc_flag_if_supported(GMX_CUDA_NVCC_GENCODE_FLAGS NVCC_HAS_GENCODE_COMPUTE_AND_SM_120 --generate-code=arch=compute_120,code=sm_120)
Now I get the following compile-time error:
ptxas error : Value of threads per SM for entry _Z23nbnxn_kernel_prune_cudaILb1EEv13NBAtomDataGpu10NBParamGpuN5Nbnxm9gpu_plistEi is out of range. .minnctapersm will be ignored
ptxas error : Value of threads per SM for entry _Z23nbnxn_kernel_prune_cudaILb0EEv13NBAtomDataGpu10NBParamGpuN5Nbnxm9gpu_plistEi is out of range. .minnctapersm will be ignored
ptxas fatal : Ptx assembly aborted due to errors
CMake Error at libgromacs_generated_nbnxm_cuda_kernel_pruneonly.cu.o.Release.cmake:280 (message):
Error generating file
/local/ckutzne/git-gromacs-2023/build/threads/src/gromacs/CMakeFiles/libgromacs.dir/nbnxm/cuda/./libgromacs_generated_nbnxm_cuda_kernel_pruneonly.cu.o
Am I missing something?
Any help appreciated.