Install workflow with AMD GPU support (Framework 16, Ubuntu 24.04, GPU: AMD Radeon RX 7700S)

Hello, Dear all!

I am a person only intermediately experienced with Linux installs and also fairly easily frustrated :D. After a few days of struggle, I managed to install Gromacs with AMD GPU support on my Laptop and thought others might benefit from what I’ve learnt.

Stuff:
Computer: Framework 16 Laptop (2024)
CPU: AMD Ryzen 7 7840HS
GPU: AMD Radeon RX7700S
OS: clean install of Ubuntu 24.04

GROMACS version: 2024.4
GROMACS modification: No

Install overview:
cmake, gromacs and for AMD GPU support: rocm, rocm-llvm-dev, acpp
…and dependencies

Workflow:

I recommend using Timeshift to set up system recovery points every now and then.
Adjust package versions in the code as needed.

As I messed up in the first try, I did a clean install of Ubuntu 24.04 and made sure the essentials are there by running the system updates and:

sudo apt update
sudo apt upgrade -y
sudo apt install -y build-essential

Now, install rocm using the convenience service amdgpu-install of AMD (which is just a wrapper to installing with the system package manager).
This also installs some amdgpu drivers - if secure boot is enabled in your system, you will be prompted to set a one-time password for enrolling a new “MOK” at next startup. Choose something easy like “12345678” and on next reboot select “Enroll MOK” > “Continue” > “Yes” > Enter the Password > “Reboot”. This only has to be done once.

#taken from rocm documentation
sudo apt update
sudo apt install -y "linux-headers-$(uname -r)" "linux-modules-extra-$(uname -r)"
sudo apt install -y python3-setuptools python3-wheel
cd ~/Downloads
wget https://repo.radeon.com/amdgpu-install/6.3/ubuntu/noble/amdgpu-install_6.3.60300-1_all.deb
sudo apt install -y ./amdgpu-install_6.3.60300-1_all.deb
sudo apt update
sudo amdgpu-install -y --usecase=rocm
# Add the current user to the render and video groups
sudo usermod -a -G render,video $LOGNAME

Reboot.

Check the user groups (should belong to render and video now) and the rocm install:

groups
rocminfo

rocminfo also informs about the rocm “Name” of the GPU, in my case “gfx1102”, which is needed later on.

rocm comes “bundled” with it’s own version of llvm. The current gromacs documentation recommends doing the installation of acpp and gromacs with that very same llvm of the rocm installation. So I tried. However, it took me quite some time to realize that another package is needed to be able to do that: rocm-llvm-dev. This will add llvm components essential for the further installation to the existing rocm directory.

sudo apt install -y rocm-llvm-dev

For installing acpp (AdaptiveCpp, formerly named hipSYCL/Open SYCL), we also need libboost-all-dev (for building acpp), git (to get acpp) and cmake (for building acpp and gromacs).

sudo apt install -y libboost-all-dev git cmake

Now to get, build and install AdaptiveCpp.

Building with cmake (see code) will get a lot of options such as:
-DCMAKE_INSTALL_PREFIX=/usr/local
for providing the install destination. According to acpp documentation, this is actually the default path, but I chose to explicitly set it.

As mentioned, the version of llvm coming with rocm and rocm-llvm-dev is used. Also, clang and clang++ which are shipped with rocm will be used instead of generic versions. cmake is told to look for those by providing the paths with:
-DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang
-DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++
-DLLVM_DIR=/opt/rocm/llvm/lib/cmake/llvm/

AdaptiveCpp offers a variety of backends for different systems. When building gromacs it is explicitly stated it does not support the “generic SSCP compiler” and aborts building. Therefore, this compiler is switched off. Actually, for AMD GPUs the HIP/ROCm backend is the only other option, and therefore only this remains switched on:
-DWITH_ROCM_BACKEND=ON
-DWITH_SSCP_COMPILER=OFF
-DWITH_OPENCL_BACKEND=OFF
-DWITH_LEVEL_ZERO_BACKEND=OFF
-DWITH_CUDA_BACKEND=OFF
The build process states that these options are deprecated, but right now they still do work.

The last option sets the default target device, which might help in avoiding mixing up with the on board graphics. This is where the rocm “Name” of the GPU is used:
-DDEFAULT_TARGETS=‘hip:gfx1102’

cd ~/Downloads
git clone https://github.com/AdaptiveCpp/AdaptiveCpp
cd AdaptiveCpp
mkdir build && cd build
sudo cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local \
-DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang \
-DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ \
-DLLVM_DIR=/opt/rocm/llvm/lib/cmake/llvm/ \
-DWITH_ROCM_BACKEND=ON \
-DWITH_SSCP_COMPILER=OFF \
-DWITH_OPENCL_BACKEND=OFF \
-DWITH_LEVEL_ZERO_BACKEND=OFF \
-DWITH_CUDA_BACKEND=OFF -DDEFAULT_TARGETS='hip:gfx1102'
sudo make install

Check acpp:

acpp-info

Finally, get, build and install gromacs.

The cmake options:
-DGMX_BUILD_OWN_FFTW=ON
-DREGRESSIONTEST_DOWNLOAD=ON
are the default recommendation according to gromacs documentation.

The paths to clang and clang++ in the rocm directory are again provided with:
-DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang
-DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++

cmake is instructed to set acpp for gpu support, which is an implementation of SYCL (hence the two options). Also, the rocm “Name” of the GPU is provided again:
-DGMX_GPU=SYCL
-DGMX_SYCL=ACPP
-DHIPSYCL_TARGETS=‘hip:gfx1102’

#taken from gromacs documentation
cd ~/Downloads
wget ftp://ftp.gromacs.org/gromacs/gromacs-2024.4.tar.gz
tar xfz gromacs-2024.4.tar.gz
cd gromacs-2024.4
mkdir build
cd build
sudo cmake .. -DGMX_BUILD_OWN_FFTW=ON \
-DREGRESSIONTEST_DOWNLOAD=ON \
-DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang \
-DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++ \
-DGMX_GPU=SYCL \
-DGMX_SYCL=ACPP \
-DHIPSYCL_TARGETS='hip:gfx1102'

The make commands took quite a while.

sudo make
sudo make check

With make check, one test failed, MDRunIOTests. I chose to continue with the installation.

sudo make install

Now, all that’s left is to delete the downloaded content from ~/Downloads and get started with gromacs.
Remember to source gromacs before each use:

source /usr/local/gromacs/bin/GMXRC

Hope this can help fellow AMD GPU users :)

l

Questions:

Gromacs seems to be running well, but I would like to know if the following may cause problems:

  1. make check: test #63 MDRunIOTests failed. This is very similar to Stuck at 'make': libgromacs.so.7: undefined reference to `__kmpc_...' - #9 by al42and. Therefore, I’m not too worried, but would still like to share the log: Dropbox

  2. make check, gmx mdrun: produce a warning
    WARNING: While sanity checking device #1, Dummy kernel produced invalid values
    It’s also in the log file. What to make of this?

Many thanks to the developers, contributors and the community :)
Greetings!

1 Like

Thanks for writing up a detailed step-by-step guide!

  1. test #63 MDRunIOTests failed.

Yes, ContinuationIsExact tests tend to be overly sensitive, and if it’s the only test failing, that is very likely benign.

  1. make check, gmx mdrun: produce a warning WARNING: While sanity checking device #1, Dummy kernel produced invalid values

Could you share the full log file from gmx mdrun? Specifically, the hardware section at the top, so that we know what device #1 is. acpp-info output would also be helpful. This could be some other backend (e.g., Mesa) presenting the same physical GPU, and it should be perfectly safe (a device with failed sanity check would not be used), but would be nice to figure things out anyway.

Minor notes about sudo make and sudo make check:

  • Usually, it’s better to run cmake, make, and make check without sudo (you don’t need any special permissions to compile things); only make install needs sudo when you’re installing into /usr or some other system directory. Not a big deal, though, and if you run cmake with sudo once, then your build files are owned by root and you will have to continue using sudo.
  • You can use make -j$(nproc) to run the compilation in parallel on all cores (or use a specific value, e.g., -j4, if you run out of RAM).

Hello.

Thanks a ton for writing this all out! I’ve been meaning to try out GROMACS on an AMD system that I have, but the installation procedure was all too daunting for me to attempt. Having some time over the holidays, I decided to spend an evening to try and have a good go at it, and I’m happy to report that I have a successfully running 2024.4 build on a Radeon RX 6600 system.

To get into some details, I ran into some trouble installing rocm using your instructions, but I was able to narrow it down to not using the correct distro version. The script out of the box didn’t work. I’m using Zorin 17.2, which is based on Ubuntu 22.04, hence I had to follow the instructions from the following:

Which went smoothly. To preempt another quick fix later on, I had to rename the /opt/rocm-6.3.1/ directory to /opt/rocm/ to make it work with the rest of the instructions. I also followed the post-installation instructions for good measure.

From here onwards, it was smooth sailing installing acpp and then building GROMACS after I modified the target architecture names.

For what it’s worth, I did not have the test #63 fail during make check. In fact, none of them failed. Subsequently, I ran the regressiontest suite with the all option and it ran successfully. I also managed to run a small-ish toy system simulation successfully, and it yielded a nice speed-up over my CPU-only runs.

All in all, this worked out quite successfully, and I’m sure to come back to this link many times in the foreseeable future.

Output of gmx -version:

                         :-) GROMACS - gmx, 2024.4 (-:

Executable:   /usr/local/gromacs/bin/gmx
Data prefix:  /usr/local/gromacs
Working dir:  /home/user/gromacs-2024.4/regressiontests-2024.4
Command line:
  gmx -version

GROMACS version:     2024.4
Precision:           mixed
Memory model:        64 bit
MPI library:         thread_mpi
OpenMP support:      enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support:         SYCL (hipSYCL)
NBNxM GPU setup:     super-cluster 2x2x2 / cluster 8
SIMD instructions:   AVX2_128
CPU FFT library:     fftw-3.3.8-sse2-avx-avx2-avx2_128
GPU FFT library:     VkFFT internal (1.3.1) with HIP backend
Multi-GPU FFT:       none
RDTSCP usage:        enabled
TNG support:         enabled
Hwloc support:       disabled
Tracing support:     disabled
C compiler:          /opt/rocm/llvm/bin/clang Clang 18.0.0
C compiler flags:    -mavx2 -mfma -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler:        /opt/rocm/llvm/bin/clang++ Clang 18.0.0
C++ compiler flags:  -mavx2 -mfma -Wno-reserved-identifier -Wno-missing-field-initializers -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-source-uses-openmp -Wno-c++17-extensions -Wno-documentation-unknown-command -Wno-covered-switch-default -Wno-switch-enum -Wno-switch-default -Wno-extra-semi-stmt -Wno-weak-vtables -Wno-shadow -Wno-padded -Wno-reserved-id-macro -Wno-double-promotion -Wno-exit-time-destructors -Wno-global-constructors -Wno-documentation -Wno-format-nonliteral -Wno-used-but-marked-unused -Wno-float-equal -Wno-cuda-compat -Wno-conditional-uninitialized -Wno-conversion -Wno-disabled-macro-expansion -Wno-unused-macros -Wno-unsafe-buffer-usage -Wno-unused-parameter -Wno-unused-variable -Wno-newline-eof -Wno-old-style-cast -Wno-zero-as-null-pointer-constant -Wno-unused-but-set-variable -Wno-sign-compare -Wno-unused-result -Wno-cast-function-type-strict -fopenmp=libomp -O3 -DNDEBUG
BLAS library:        Internal
LAPACK library:      Internal
hipSYCL launcher:    /usr/local/lib/cmake/AdaptiveCpp/syclcc-launcher
hipSYCL flags:       -Wno-unknown-cuda-version -Wno-unknown-attributes  --acpp-targets="hip:gfx1032"
hipSYCL GPU flags:   -ffast-math;-fgpu-inline-threshold=99999
hipSYCL targets:     hip:gfx1032
hipSYCL version:     AdaptiveCpp 24.10.0+git.7677cf6e.20241220.branch.develop

1 Like

Hello!

Sorry for the late reply, I was off for vacation.

@al42and, thanks a lot for your input!

I didn’t think of this before but apparently, “device #1” is the on board graphics chip, while “device #0” is the dedicated GPU.

This is the head of the gmx mdrun log (See bottom lines):

GROMACS:      gmx mdrun, version 2024.4
Executable:   /usr/local/gromacs/bin/gmx
Data prefix:  /usr/local/gromacs
Working dir:  /home/cyano/Test_gromacs_Framework16
Process ID:   150356
Command line:
  gmx mdrun -v -deffnm step7_500

GROMACS version:     2024.4
Precision:           mixed
Memory model:        64 bit
MPI library:         thread_mpi
OpenMP support:      enabled (GMX_OPENMP_MAX_THREADS = 128)
GPU support:         SYCL (hipSYCL)
NBNxM GPU setup:     super-cluster 2x2x2 / cluster 8
SIMD instructions:   AVX_512
CPU FFT library:     fftw-3.3.8-sse2-avx-avx2-avx2_128
GPU FFT library:     VkFFT internal (1.3.1) with HIP backend
Multi-GPU FFT:       none
RDTSCP usage:        enabled
TNG support:         enabled
Hwloc support:       disabled
Tracing support:     disabled
C compiler:          /opt/rocm/llvm/bin/clang Clang 18.0.0
C compiler flags:    -march=skylake-avx512 -Wno-missing-field-initializers -O3 -DNDEBUG
C++ compiler:        /opt/rocm/llvm/bin/clang++ Clang 18.0.0
C++ compiler flags:  -march=skylake-avx512 -Wno-reserved-identifier -Wno-missing-field-initializers -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-source-uses-openmp -Wno-c++17-extensions -Wno-documentation-unknown-command -Wno-covered-switch-default -Wno-switch-enum -Wno-switch-default -Wno-extra-semi-stmt -Wno-weak-vtables -Wno-shadow -Wno-padded -Wno-reserved-id-macro -Wno-double-promotion -Wno-exit-time-destructors -Wno-global-constructors -Wno-documentation -Wno-format-nonliteral -Wno-used-but-marked-unused -Wno-float-equal -Wno-cuda-compat -Wno-conditional-uninitialized -Wno-conversion -Wno-disabled-macro-expansion -Wno-unused-macros -Wno-unsafe-buffer-usage -Wno-unused-parameter -Wno-unused-variable -Wno-newline-eof -Wno-old-style-cast -Wno-zero-as-null-pointer-constant -Wno-unused-but-set-variable -Wno-sign-compare -Wno-unused-result -Wno-cast-function-type-strict SHELL:-fopenmp=libomp -O3 -DNDEBUG
BLAS library:        Internal
LAPACK library:      Internal
hipSYCL launcher:    /usr/local/lib/cmake/AdaptiveCpp/syclcc-launcher
hipSYCL flags:       -Wno-unknown-cuda-version -Wno-unknown-attributes  --acpp-targets="hip:gfx1102"
hipSYCL GPU flags:   -ffast-math;-fgpu-inline-threshold=99999
hipSYCL targets:     hip:gfx1102
hipSYCL version:     AdaptiveCpp 24.06.0+git.ba3fb96f.20241218.branch.develop


Running on 1 node with total 8 cores, 16 processing units, 1 compatible GPU
Hardware detected on host cyano-Laptop-16-AMD-Ryzen-7040-Series:
  CPU info:
    Vendor: AMD
    Brand:  AMD Ryzen 7 7840HS w/ Radeon 780M Graphics     
    Family: 25   Model: 116   Stepping: 1
    Features: aes amd apic avx avx2 avx512f avx512cd avx512bw avx512vl avx512bf16 avx512secondFMA clfsh cmov cx8 cx16 f16c fma htt lahf misalignsse mmx msr nonstop_tsc pclmuldq pdpe1gb popcnt pse rdrnd rdtscp sha sse2 sse3 sse4a sse4.1 sse4.2 ssse3 x2apic
  Hardware topology: Basic
    Packages, cores, and logical processors:
    [indices refer to OS logical processors]
      Package  0: [   0   1] [   2   3] [   4   5] [   6   7] [   8   9] [  10  11] [  12  13] [  14  15]
    CPU limit set by OS: -1   Recommended max number of threads: 16
  GPU info:
    Number of GPUs detected: 2
    #0: name: AMD Radeon™ RX 7700S, architecture 11.0.2, vendor: AMD, device version: gfx1102, driver version 60342131, status: compatible
    #1: N/A, status: non-functional

And interesting bits of accp-info:

=================Backend information===================
Loaded backend 0: HIP
  Found device: AMD Radeon™ RX 7700S
  Found device: AMD Radeon Graphics
Loaded backend 1: OpenMP
  Found device: AdaptiveCpp OpenMP host device

=================Device information===================
***************** Devices for backend HIP *****************
Device 0:
 General device information:
  Name: AMD Radeon™ RX 7700S
...
Device 1:
 General device information:
  Name: AMD Radeon Graphics
...

I don’t want the on board GPU to be used and according to the Gromacs docs it also cannot be used, if I recall correctly. So it seems all good and I’m not concerned about the warning :)

Thanks for pointing this out. I’m a bit surprised that didn’t come to my mind earlier, haha.

You are absolutely right. I tried it. I think I should change this in my original instructions. Also, this time after installing gromacs, MDRunIOTests succeeded. Yay!

@a.hazarika, I’m glad this could be of help! :)

Hmm, I assume the problem was in getting the right convenience wrapper “amdgpu-install” from AMD

wget https://repo.radeon.com/amdgpu-install/6.3/ubuntu/noble/amdgpu-install_6.3.60300-1_all.deb

You do need to download the version which fits your distro. But using the quick start install instructions is a good idea. It’s really just a matter of taste if you want to actually use amdgpu-install as I did or just install it use it to add the AMD repositories so you can directly call apt install amdgpu-dkms rocm as you did. Happy it worked out!

Hmm, for me there is a folder /opt/rocm-version AND a folder /opt/rocm which only contains links to the first. I assume the latter is used for providing paths in a version independent manner. Did you not get both folders after installing rocm?

It’s really just a matter of taste if you want to actually use amdgpu-install as I did or just install it use it to add the AMD repositories so you can directly call apt install amdgpu-dkms rocm as you did. Happy it worked out!

I found that the amdgpu-install script does a rudimentary string comparison of sorts (?) with /etc/os-release. It doesn’t recognize any Ubuntu-derivative distros like Pop or Zorin despite them being mostly the same under the hood, so I had to take the scenic route.

Hmm, for me there is a folder /opt/rocm- AND a folder /opt/rocm which only contains links to the first. I assume the latter is used for providing paths in a version independent manner. Did you not get both folders after installing rocm?

Nay, only the /opt/rocm-x.x.x directory. I could’ve done a symlink from /opt/rocm/ to /opt/rocm-x.x.x/ but it just skipped my mind at the time. So far I haven’t noticed any alarming signs, which puts my mind at ease.

1 Like

Can you please tell in greater detail about installing acpp, as I also have rx6600 and was successful in installing ROCm, however i get error while installing acpp:

CMake Error at /usr/share/cmake-3.22/Modules/CMakeTestCXXCompiler.cmake:62 (message):
The C++ compiler

"/opt/rocm/llvm/bin/clang++"

is not able to compile a simple test program.

It fails with the following output:

Change Dir: /home/blue_orchid/Downloads/Adaptive2/AdaptiveCpp/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/gmake -f Makefile cmTC_1618b/fast && /usr/bin/gmake  -f CMakeFiles/cmTC_1618b.dir/build.make CMakeFiles/cmTC_1618b.dir/build
gmake[1]: Entering directory '/home/blue_orchid/Downloads/Adaptive2/AdaptiveCpp/CMakeFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_1618b.dir/testCXXCompiler.cxx.o
/opt/rocm/llvm/bin/clang++    -MD -MT CMakeFiles/cmTC_1618b.dir/testCXXCompiler.cxx.o -MF CMakeFiles/cmTC_1618b.dir/testCXXCompiler.cxx.o.d -o CMakeFiles/cmTC_1618b.dir/testCXXCompiler.cxx.o -c /home/blue_orchid/Downloads/Adaptive2/AdaptiveCpp/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
Linking CXX executable cmTC_1618b
/usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_1618b.dir/link.txt --verbose=1
/opt/rocm/llvm/bin/clang++ CMakeFiles/cmTC_1618b.dir/testCXXCompiler.cxx.o -o cmTC_1618b 
ld.lld: error: unable to find library -lstdc++
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[1]: *** [CMakeFiles/cmTC_1618b.dir/build.make:100: cmTC_1618b] Error 1
gmake[1]: Leaving directory '/home/blue_orchid/Downloads/Adaptive2/AdaptiveCpp/CMakeFiles/CMakeTmp'
gmake: *** [Makefile:127: cmTC_1618b/fast] Error 2

I feel frustration coming

Hi!

Seems ld.lld from the rocm-llvm install can’t find libstdc++.
libstdc++ is not installed with rocm. It’s an additional library for (compiling?) C++ programs.

It’s likely either not installed or cannot be found. Please check if there is any libstdc++ package installed using your package manager, e.g. with apt:

apt list --installed libstdc*

For me it returns as installed: libstdc++-11-dev, libstdc++-13-dev and libstdc++6 (But I have no idea which one ld.lld wants to use).

I couldn’t find out, if libstdc++ or libstdc++-dev pakages are included in build-essential (or any other package). But to make sure: did you install build-essential?

sudo apt install -y build-essential

If this installs anything new, check again for libstdc++ using apt list as before.


If for whatever reasons, the libraries are still missing, my last idea would be to install libstdc++ packages “by hand” using your package manager. Before trying out different setups you should probably set up a system restore point (e.g. using Timeshift).
I have no idea, which version is needed so I suggest you try installing one or a combination of the same packages I have: libstdc++-11-dev, libstdc++-13-dev and libstdc++6 e.g.:

sudo apt install libstdc++-11-dev libstdc++-13-dev libstdc++6

If you’re missing a larger base package/metapackage (comparable to build-essential) I wouldn’t be too surprised if you run into another “library missing” error with different packages. But maybe… hopefully… it’s only libstdc++.

Good luck!

Yes, that’s right. ROCm compiler needs a certain version of libstdc++, but, for some reason, it does not install it (nor document it; there have had a bugreport open for quite a while)

The specific version needed depends on the distribution and ROCm version. For Ubuntu 22.04, it is actually libstdc++-12-dev that’s missing. Installing all available versions of libstdc++-*-dev won’t hurt.