GROMACS-SYCL Build with CUDA Support

User discussions
1

GROMACS version:2021-sycl
GROMACS modification: No

Hi,
I’m trying to builld the sycl version of gromacs using intel-llvm compiler with cuda backend.
The cmake options are:
$ cmake …
-DGMX_GPU=SYCL
-DGMX_BUILD_OWN_FFTW=ON
-DCMAKE_C_COMPILER=clang
-DCMAKE_CXX_COMPILER=clang++
However, gmx binary failed to regconized V100:
#0: name: Tesla V100-PCIE-32GB, vendor: NVIDIA Corporation, device version: 0.0, status: incompatible (please recompile with correct GMX_OPENCL_NB_CLUSTER_SIZE of 4)
#1: name: SYCL host device, vendor: , device version: 1.2, status: incompatible
As I understand -DGMX_OPENCL_NB_CLUSTER_SIZE=4 is reserved for Intel GPUs.

If someone has successfully build SYCL version, I appreciate some of your insights.

2

The 2021-sycl version doesn’t support Nvidia GPUs. You can use that version only with Intel GPUs. Work is ongoing on the master branch to add support for all GPUs and also different SYCL compilers.

If you want to try it you need to:

Because it is still work-in-progress it isn’t optimized yet and the current performance reflects that.
As always any user is invited to help with the effort to improve GROMACS. Let us know if you are interested.

3

Thanks for the tips. I was able to build gromacs from master branch following your instructions, yet the error persists.
I listed the steps here in case I overlooked something.

[git checkout]
git clone GROMACS / GROMACS · GitLab
git checkout remotes/origin/sz_SYCL-nbnxm-local-mem-reduction

[cmake/gmxManageSYCL.cmake : before]
" “CXX” DISABLE_SYCL_CXX_FLAGS SYCL_CXX_FLAGS “-fsycl -fsycl-device-code-split=per_kernel”)

[cmake/gmxManageSYCL.cmake : after]
" “CXX” DISABLE_SYCL_CXX_FLAGS SYCL_CXX_FLAGS “-fsycl -fsycl-targets=nvptx64-nvidia-cuda-sycldevice”)
(-fsycl-device-code-split caused configure error)

[cmake]
cmake … (same options as the opening post)
make

[test commands]
gmx mdrun -noconfout -nsteps 10000 -nb gpu -s stmv.tpr -tunepme -v

Regarding performance, I strive to compare the performance of the trifecta:
gmx-sycl (xe_hp) vs. gmx-sycl (v100) vs. gmx-cuda (v100)

4

Sorry I forgot some extra steps:

  • You need the cmake option -DGMX_GPU_NB_CLUSTER_SIZE=8
  • You need the environment variable GMX_GPU_DISABLE_COMPATIBILITY_CHECK=1 when running mdrun

If you interest is a fair performance comparison you want to wait until it is properly working. It just started to barely work and we haven’t yet done the work to optimize for performance. We expect a lot of performance improvements over the next few months. Those will be included in the next GROMACS release with the first beta release scheduled for around September.

5

Thanks.

V100s were now recognized:
Number of GPUs detected: 3
#0: name: Tesla V100-PCIE-32GB, vendor: NVIDIA Corporation, device version: 0.0, status: compatible
#1: name: Tesla V100-PCIE-32GB, vendor: NVIDIA Corporation, device version: 0.0, status: compatible
#2: name: SYCL host device, vendor: , device version: 1.2, status: compatible

Unfortunately, it still crashed with the following message.
pi_die: cuda_piEnqueueEventsWaitWithBarrier not implemented

This is an unresolved issue with intel-llvm:

I guess I have to wait until the sycl implementation matures more.