Gromacs with hipSYCL throwing a symbol lookup error at runtime

User discussions
,
1

GROMACS version: 2022 (at tag v2022 / commit c07b37733d3ad84cee11a8a7d2b841a734e42ec6)
GROMACS modification: No

Hey folks,

I’ve been trying to run a few tests with Gromacs on AMD GPUs with hipSYCL/ROCm (develop branch), but I’m getting weird symbol lookup errors at runtime with adh_dodec.

$ ./gmx mdrun -tunepme no -pin on -nsteps 10000 -ntmpi 1 -ntomp 64 -noconfout -nb gpu -bonded gpu -pme gpu -v -s topol.tpr 
           :-) GROMACS - gmx mdrun, 2022-dev-20220222-c07b37733d (-:

Executable:   /home/jmaia/Repo/gromacs/build-threads/bin/gmx
Data prefix:  /home/jmaia/Repo/gromacs (source tree)
Working dir:  /home/jmaia/Repo/benchmark-gromacs/adh_dodec
Command line:
  gmx mdrun -tunepme no -pin on -nsteps 10000 -ntmpi 1 -ntomp 64 -noconfout -nb gpu -bonded gpu -pme gpu -v -s topol.tpr

/home/jmaia/Repo/gromacs/build-threads/bin/gmx: symbol lookup error: /home/jmaia/Repo/gromacs/build-threads/lib/libgromacs.so.7: undefined symbol: _ZN7hipsycl2rt16kernel_operationC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt6vectorISt10unique_ptrINS0_23backend_kernel_launcherESt14default_deleteISC_EESaISF_EERKNS0_17requirements_listE

I’ve configured hipSYCL with:

cmake .. \
        -DCMAKE_C_COMPILER=${ROCM_PATH}/llvm/bin/clang \
        -DCMAKE_CXX_COMPILER=${ROCM_PATH}/llvm/bin/clang++ \
        -DCMAKE_CXX_FLAGS="-O3 -D_GLIBCXX_USE_CXX11_ABI=0" \
        -DLLVM_DIR=${ROCM_PATH}/llvm/lib/cmake/llvm/ \
        -DCMAKE_INSTALL_PREFIX=${path_to_hipsycl_install} \
        -DROCM_PATH=$ROCM_PATH \
        -DWITH_ROCM_BACKEND=ON \
        -DBOOST_ROOT=${path_to_boost}

And Gromacs with:

export HIPSYCL_TARGETS="hip:gfx90a"
cmake .. \
        -DCMAKE_BUILD_TYPE=Release \
        -DBUILD_SHARED_LIBS=on \
        -DGMX_BUILD_OWN_FFTW=on \
        -DCMAKE_C_COMPILER=${ROCM_PATH}/llvm/bin/clang \
        -DCMAKE_CXX_COMPILER=${ROCM_PATH}/llvm/bin/clang++ \
        -DLLVM_DIR=${ROCM_PATH}/llvm/lib/cmake/llvm/ \
        -DGMX_GPU=SYCL \
        -DGMX_SYCL_HIPSYCL=ON \
        -DHIPSYCL_TARGETS=hip:gfx90a \
        -DCMAKE_PREFIX_PATH=${path_to_hipsycl_install}

Has anyone seen an error similar to this before? Not sure if this is the right venue, but I’m refraining from opening an issue on gitlab since I’m pretty sure there is something wrong on my end.

Thanks in advance!

2

Hi!

Nothing outright wrong with what you’re doing. Does hipsycl-info (should be installed as part of hipSYCL) work fine? What is your OS/ROCm version?

One suspect is _GLIBCXX_USE_CXX11_ABI, and the error message points in the same direction (not conclusively, though). First, I don’t think this flag should be necessary since hipSYCL requires a fully C++17-compliant compiler/stdlib. Second, when you are passing this flag to hipSYCL, it might also be necessary to pass it to GROMACS.

Could you try recompiling hipSYCL without this flag? If you get errors, I would suggest determining their cause and fixing them (very old Boost?) without setting the CXX11_ABI=0 flag. If you cannot avoid setting this flag for hipSYCL, try setting it for GROMACS as well.

3

Hi,

That specific symbol lookup error does not look familiar, but the following may eliminate the issue:

  • use the latest develop of hipSYCL
  • disable the CPU backend of hipSYCL: -DWITH_ACCELERATED_CPU=off
  • in some cases I also had to disable the device mangler: -DHIPSYCL_NO_DEVICE_MANGLER=ON

Side-note, do add at least the -DHIPSYCL_SYCLCC_EXTRA_ARGS='-ffast-math' to the GROMACS build (this is WIP and not fully automated yet).

Cheers,
Szilárd

4

Hey folks,

Thanks much for the suggestions.

@al42and - hipsycl-info does indeed work, and reports the right devices.
Passing _GLIBCXX_USE_CXX11_ABI to Gromacs as well did the trick! I can now get the code running on a MI250 with the bonded terms on CPUs. I added _GLIBCXX_USE_CXX11_ABI to hipSYCL because Gromacs’ cmake step choked on this error if I didn’t:

  '/home/jmaia/Repo/hipSYCL/build/install/bin/../lib/libhipSYCL_clang.so':
  '/home/jmaia/Repo/hipSYCL/build/install/bin/../lib/libhipSYCL_clang.so:
  undefined symbol:
  _ZNK4llvm10ModulePass17createPrinterPassERNS_11raw_ostreamERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE'

So I saw this workaround online and it seemed to fix that issue.

@pszilard Yeah, I’ve added both options (WITH_ACCELERATED_CPU and HIPSYCL_SYCLCC_EXTRA_ARGS) and and I’m getting 128-ish ns/day on adh-dodec with a single GCD of the MI250, which is ~40% of what the AMD HIP fork reports.

I saw that we have a checklist for stuff to be done on the AMD GPU end. I can bring this people here over at AMD so we can assess what needs to be done. I’m also not very familiar with SYCL + the GROMACS SYCL code so it might take me some time to actually be useful.

This is excellent progress, thanks much folks.

5

I can now get the code running on a MI250 with the bonded terms on CPUs.

Great!

For performance tuning, a better starting point would be hwe-release-2022 branch, which is GROMACS 2022 with SYCL-specific improvements.