Make check tests fail on amd rx 6650xt

User discussions
1

GROMACS version: gromacs-2023
GROMACS modification: No
I managed to build gromacs with hipSYCL support using the clang compiler bundled with ROCm 5.2.1 using the following
cmake .. -DGMX_GPU_FFT_LIBRARY=VKFFT -DGMX_BUILD_OWN_FFTW=ON \ -DREGRESSIONTEST_DOWNLOAD=ON \ -DCMAKE_C_COMPILER=/opt/rocm-5.2.1/llvm/bin/clang \ -DCMAKE_CXX_COMPILER=/opt/rocm-5.2.1/llvm/bin/clang++ \ -DGMX_GPU=SYCL -DGMX_SYCL_HIPSYCL=ON -DHIPSYCL_TARGETS='hip:gfx1032'

however, when running make check I get following :

0% tests passed, 87 tests failed out of 87

Label Time Summary:
GTest              =   0.44 sec*proc (81 tests)
IntegrationTest    =   0.13 sec*proc (25 tests)
MpiTest            =   0.10 sec*proc (19 tests)
QuickGpuTest       =   0.10 sec*proc (17 tests)
SlowGpuTest        =   0.87 sec*proc (18 tests)
SlowTest           =   0.07 sec*proc (13 tests)
UnitTest           =   0.24 sec*proc (43 tests)

Total Test time (real) =   1.60 sec

The following tests FAILED:
	  1 - GmxapiExternalInterfaceTests (Failed)
	  2 - GmxapiInternalInterfaceTests (Failed)
	  3 - NbLibListedForcesTests (Failed)
	  4 - NbLibSamplesTestArgon (Failed)
	  5 - NbLibSamplesTestMethaneWater (Failed)
	  6 - NbLibUtilTests (Failed)
	  7 - NbLibSetupTests (Failed)
	  8 - NbLibTprTests (Failed)
	  9 - NbLibIntegrationTests (Failed)
	 10 - NbLibIntegratorTests (Failed)
	 11 - TestUtilsUnitTests (Failed)
	 12 - TestUtilsMpiUnitTests (Failed)
	 13 - UtilityUnitTests (Failed)
	 14 - UtilityMpiUnitTests (Failed)
	 15 - GmxlibTests (Failed)
	 16 - MdlibUnitTest (Failed)
	 17 - AwhTest (Failed)
	 18 - DensityFittingAppliedForcesUnitTest (Failed)
	 19 - QMMMAppliedForcesUnitTest (Failed)
	 20 - AppliedForcesUnitTest (Failed)
	 21 - ListedForcesTest (Failed)
	 22 - NbnxmTests (Failed)
	 23 - CommandLineUnitTests (Failed)
	 24 - DomDecTests (Failed)
	 25 - DomDecMpiTests (Failed)
	 26 - EwaldUnitTests (Failed)
	 27 - FFTUnitTests (Failed)
	 28 - GpuUtilsUnitTests (Failed)
	 29 - HardwareUnitTests (Failed)
	 30 - MathUnitTests (Failed)
	 31 - MdrunUtilityUnitTests (Failed)
	 32 - MdrunUtilityMpiUnitTests (Failed)
	 33 - MDSpanTests (Failed)
	 34 - MdtypesUnitTest (Failed)
	 35 - OnlineHelpUnitTests (Failed)
	 36 - OptionsUnitTests (Failed)
	 37 - PbcutilUnitTest (Failed)
	 38 - RandomUnitTests (Failed)
	 39 - RestraintTests (Failed)
	 40 - TableUnitTests (Failed)
	 41 - TaskAssignmentUnitTests (Failed)
	 42 - GmxTimingTests (Failed)
	 43 - TopologyTest (Failed)
	 44 - PullTest (Failed)
	 45 - SimdUnitTests (Failed)
	 46 - CompatibilityHelpersTests (Failed)
	 47 - GmxAnaTest (Failed)
	 48 - GmxPreprocessTests (Failed)
	 49 - Pdb2gmx1Test (Failed)
	 50 - Pdb2gmx2Test (Failed)
	 51 - Pdb2gmx3Test (Failed)
	 52 - CorrelationsTest (Failed)
	 53 - AnalysisDataUnitTests (Failed)
	 54 - CoordinateIOTests (Failed)
	 55 - TrajectoryAnalysisUnitTests (Failed)
	 56 - EnergyAnalysisUnitTests (Failed)
	 57 - ToolUnitTests (Failed)
	 58 - FileIOTests (Failed)
	 59 - SelectionUnitTests (Failed)
	 60 - MdrunOutputTests (Failed)
	 61 - MdrunModulesTests (Failed)
	 62 - MdrunIOTests (Failed)
	 63 - MdrunTestsOneRank (Failed)
	 64 - MdrunTestsTwoRanks (Failed)
	 65 - MdrunSingleRankAlgorithmsTests (Failed)
	 66 - MdrunNonIntegratorTests (Failed)
	 67 - MdrunTpiTests (Failed)
	 68 - MdrunMpiTests (Failed)
	 69 - MdrunMultiSimTests (Failed)
	 70 - MdrunMultiSimReplexTests (Failed)
	 71 - MdrunMultiSimReplexEquivalenceTests (Failed)
	 72 - MdrunMpi1RankPmeTests (Failed)
	 73 - MdrunMpi2RankPmeTests (Failed)
	 74 - MdrunCoordinationBasicTests1Rank (Failed)
	 75 - MdrunCoordinationBasicTests2Ranks (Failed)
	 76 - MdrunCoordinationCouplingTests1Rank (Failed)
	 77 - MdrunCoordinationCouplingTests2Ranks (Failed)
	 78 - MdrunCoordinationConstraintsTests1Rank (Failed)
	 79 - MdrunCoordinationConstraintsTests2Ranks (Failed)
	 80 - MdrunFEPTests (Failed)
	 81 - MdrunPullTests (Failed)
	 82 - MdrunSimulatorComparison (Failed)
	 83 - MdrunVirtualSiteTests (Failed)
	 84 - regressiontests/complex (Failed)
	 85 - regressiontests/freeenergy (Failed)
	 86 - regressiontests/rotation (Failed)
	 87 - regressiontests/essentialdynamics (Failed)
Errors while running CTest
make[3]: *** [CMakeFiles/run-ctest-nophys.dir/build.make:71: CMakeFiles/run-ctest-nophys] Error 8
make[2]: *** [CMakeFiles/Makefile2:3246: CMakeFiles/run-ctest-nophys.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:3279: CMakeFiles/check.dir/rule] Error 2
make: *** [Makefile:628: check] Error 2


clinfo
Number of platforms                               2
  Platform Name                                   AMD Accelerated Parallel Processing
  Platform Vendor                                 Advanced Micro Devices, Inc.
  Platform Version                                OpenCL 2.1 AMD-APP (3452.0)
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd cl_amd_event_callback 
  Platform Host timer resolution                  1ns
  Platform Extensions function suffix             AMD

  Platform Name                                   Clover
  Platform Vendor                                 Mesa
  Platform Version                                OpenCL 1.1 Mesa 22.2.2 - kisak-mesa PPA
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd
  Platform Extensions function suffix             MESA

  Platform Name                                   AMD Accelerated Parallel Processing
Number of devices                                 1
  Device Name                                     gfx1032
  Device Vendor                                   Advanced Micro Devices, Inc.
  Device Vendor ID                                0x1002
  Device Version                                  OpenCL 2.0 
  Driver Version                                  3452.0 (HSA1.1,LC)
  Device OpenCL C Version                         OpenCL C 2.0 
  Device Type                                     GPU
  Device Board Name (AMD)                         (n/a)
  Device Topology (AMD)                           PCI-E, 03:00.0
  Device Profile                                  FULL_PROFILE
  Device Available                                Yes
  Compiler Available                              Yes
  Linker Available                                Yes
  Max compute units                               16
  SIMD per compute unit (AMD)                     4
  SIMD width (AMD)                                32
  SIMD instruction width (AMD)                    1
  Max clock frequency                             2765MHz
  Graphics IP (AMD)                               10.3
  Device Partition                                (core)
    Max number of sub-devices                     16
    Supported partition types                     None
    Supported affinity domains                    (n/a)
  Max work item dimensions                        3
  Max work item sizes                             1024x1024x1024
  Max work group size                             256
  Preferred work group size (AMD)                 256
  Max work group size (AMD)                       1024
  Preferred work group size multiple              32
  Wavefront width (AMD)                           32
  Preferred / native vector sizes                 
    char                                                 4 / 4       
    short                                                2 / 2       
    int                                                  1 / 1       
    long                                                 1 / 1       
    half                                                 1 / 1        (cl_khr_fp16)
    float                                                1 / 1       
    double                                               1 / 1        (cl_khr_fp64)
  Half-precision Floating-point support           (cl_khr_fp16)
    Denormals                                     No
    Infinity and NANs                             No
    Round to nearest                              No
    Round to zero                                 No
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
  Single-precision Floating-point support         (core)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  Yes
  Double-precision Floating-point support         (cl_khr_fp64)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
  Address bits                                    64, Little-Endian
  Global memory size                              8573157376 (7.984GiB)
  Global free memory (AMD)                        8372224 (7.984GiB)
  Global memory channels (AMD)                    4
  Global memory banks per channel (AMD)           4
  Global memory bank width (AMD)                  256 bytes
  Error Correction support                        No
  Max memory allocation                           7287183768 (6.787GiB)
  Unified memory for Host and Device              No
  Shared Virtual Memory (SVM) capabilities        (core)
    Coarse-grained buffer sharing                 Yes
    Fine-grained buffer sharing                   Yes
    Fine-grained system sharing                   No
    Atomics                                       No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       1024 bits (128 bytes)
  Preferred alignment for atomics                 
    SVM                                           0 bytes
    Global                                        0 bytes
    Local                                         0 bytes
  Max size for global variable                    7287183768 (6.787GiB)
  Preferred total size of global vars             8573157376 (7.984GiB)
  Global Memory cache type                        Read/Write
  Global Memory cache size                        16384 (16KiB)
  Global Memory cache line size                   64 bytes
  Image support                                   Yes
    Max number of samplers per kernel             29679
    Max size for 1D images from buffer            134217728 pixels
    Max 1D or 2D image array size                 8192 images
    Base address alignment for 2D image buffers   256 bytes
    Pitch alignment for 2D image buffers          256 pixels
    Max 2D image size                             16384x16384 pixels
    Max 3D image size                             16384x16384x8192 pixels
    Max number of read image args                 128
    Max number of write image args                8
    Max number of read/write image args           64
  Max number of pipe args                         16
  Max active pipe reservations                    16
  Max pipe packet size                            2992216472 (2.787GiB)
  Local memory type                               Local
  Local memory size                               65536 (64KiB)
  Local memory syze per CU (AMD)                  65536 (64KiB)
  Local memory banks (AMD)                        32
  Max number of constant args                     8
  Max constant buffer size                        7287183768 (6.787GiB)
  Preferred constant buffer size (AMD)            16384 (16KiB)
  Max size of kernel argument                     1024
  Queue properties (on host)                      
    Out-of-order execution                        No
    Profiling                                     Yes
  Queue properties (on device)                    
    Out-of-order execution                        Yes
    Profiling                                     Yes
    Preferred size                                262144 (256KiB)
    Max size                                      8388608 (8MiB)
  Max queues on device                            1
  Max events on device                            1024
  Prefer user sync for interop                    Yes
  Number of P2P devices (AMD)                     0
  P2P devices (AMD)                               <printDeviceInfo:147: get number of CL_DEVICE_P2P_DEVICES_AMD : error -30>
  Profiling timer resolution                      1ns
  Profiling timer offset since Epoch (AMD)        0ns (Thu Jan  1 02:00:00 1970)
  Execution capabilities                          
    Run OpenCL kernels                            Yes
    Run native kernels                            No
    Thread trace supported (AMD)                  No
    Number of async queues (AMD)                  8
    Max real-time compute queues (AMD)            8
    Max real-time compute units (AMD)             16
  printf() buffer size                            4194304 (4MiB)
  Built-in kernels                                (n/a)
  Device Extensions                               cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_depth_images cl_amd_copy_buffer_p2p cl_amd_assembly_program 

  Platform Name                                   Clover
Number of devices                                 1
  Device Name                                     NAVI23 (navi23, LLVM 14.0.6, DRM 3.48, 6.0.3-060003-generic)
  Device Vendor                                   AMD
  Device Vendor ID                                0x1002
  Device Version                                  OpenCL 1.1 Mesa 22.2.2 - kisak-mesa PPA
  Driver Version                                  22.2.2 - kisak-mesa PPA
  Device OpenCL C Version                         OpenCL C 1.1 
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
  Device Available                                Yes
  Compiler Available                              Yes
  Max compute units                               32
  Max clock frequency                             2765MHz
  Max work item dimensions                        3
  Max work item sizes                             256x256x256
  Max work group size                             256
=== CL_PROGRAM_BUILD_LOG ===
fatal error: cannot open file '/usr/lib/clc/gfx1032-amdgcn-mesa-mesa3d.bc': No such file or directory
  Preferred work group size multiple              <getWGsizes:1200: create kernel : error -46>
  Preferred / native vector sizes                 
    char                                                16 / 16      
    short                                                8 / 8       
    int                                                  4 / 4       
    long                                                 2 / 2       
    half                                                 0 / 0        (n/a)
    float                                                4 / 4       
    double                                               2 / 2        (cl_khr_fp64)
  Half-precision Floating-point support           (n/a)
  Single-precision Floating-point support         (core)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 No
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Double-precision Floating-point support         (cl_khr_fp64)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
  Address bits                                    64, Little-Endian
  Global memory size                              8589934592 (8GiB)
  Error Correction support                        No
  Max memory allocation                           2147483648 (2GiB)
  Unified memory for Host and Device              No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       32768 bits (4096 bytes)
  Global Memory cache type                        None
  Image support                                   No
  Local memory type                               Local
  Local memory size                               65536 (64KiB)
  Max number of constant args                     16
  Max constant buffer size                        67108864 (64MiB)
  Max size of kernel argument                     1024
  Queue properties                                
    Out-of-order execution                        No
    Profiling                                     Yes
  Profiling timer resolution                      0ns
  Execution capabilities                          
    Run OpenCL kernels                            Yes
    Run native kernels                            No
  Device Extensions                               cl_khr_byte_addressable_store cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_fp64 cl_khr_extended_versioning


NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  No platform
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   No platform
  clCreateContext(NULL, ...) [default]            No platform
  clCreateContext(NULL, ...) [other]              Success [AMD]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 AMD Accelerated Parallel Processing
    Device Name                                   gfx1032
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (1)
    Platform Name                                 AMD Accelerated Parallel Processing
    Device Name                                   gfx1032
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (1)
    Platform Name                                 AMD Accelerated Parallel Processing
    Device Name                                   gfx1032


neofetch
            .-/+oossssoo+/-.               icaspell@icaspell-B450M-S2H-V2 
        `:+ssssssssssssssssss+:`           ------------------------------ 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 20.04.5 LTS x86_64 
    .ossssssssssssssssssdMMMNysssso.       Host: B450M S2H V2 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 6.0.3-060003-generic 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 2 hours, 52 mins 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 1951 (dpkg), 6 (snap) 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.0.17 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 1920x1200 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   DE: Xfce 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   WM: Xfwm4 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   WM Theme: Greybird 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Theme: Greybird [GTK2/3] 
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/    Icons: elementary-xfce-darker [GTK2/ 
  +sssssssssdmydMMMMMMMMddddyssssssss+     Terminal: xfce4-terminal 
   /ssssssssssshdmNNNNmyNMMMMhssssss/      Terminal Font: Monospace 12 
    .ossssssssssssssssssdMMMNysssso.       CPU: AMD Ryzen 5 PRO 4650G with Rade 
      -+sssssssssssssssssyyyssss+-         GPU: AMD ATI 03:00.0 Device 73ef 
        `:+ssssssssssssssssss+:`           Memory: 3149MiB / 15864MiB 
            .-/+oossssoo+/-.
2

Hi!

Since you have an integrated GPU, you must compile for it, too; that’s a limitation of ROCm. Could you try rebuilding GROMACS with -DHIPSYCL_TARGETS='hip:gfx90c,gfx1032'?

If that does not help, please post the entire output of make check (not just the “The following tests FAILED” summary) and the output of rocm-smi and hipsycl-info.

3

the problem still persists even after rebuilding GROMACS with -DHIPSYCL_TARGETS='hip:gfx90c,gfx1032'
output.log (92.9 KB)

rocm-smi


======================= ROCm System Management Interface =======================
================================= Concise Info =================================
GPU  Temp   AvgPwr  SCLK    MCLK   Fan  Perf  PwrCap  VRAM%  GPU%  
0    37.0c  4.0W    700Mhz  96Mhz  0%   auto  164.0W    6%   0%    
================================================================================
============================= End of ROCm SMI Log ==============================


hipsycl-info
[hipSYCL Warning] backend_loader: Could not load backend plugin: /usr/local/lib/hipSYCL/librt-backend-hip.so
[hipSYCL Warning] /usr/local/lib/hipSYCL/librt-backend-hip.so: undefined symbol: hipEventQuery
[hipSYCL Warning] backend_loader: Could not load backend plugin: /usr/local/lib/hipSYCL/librt-backend-omp.so
[hipSYCL Warning] libomp.so: cannot open shared object file: No such file or directory
[hipSYCL Error] No CPU backend has been loaded. Terminating.
terminate called without an active exception
Aborted (core dumped)

I used the following to compile and install hipSYCL

cmake .. -DCMAKE_C_COMPILER=/opt/rocm-5.2.1/llvm/bin/clang \
         -DCMAKE_CXX_COMPILER=/opt/rocm-5.2.1/llvm/bin/clang++ \
         -DLLVM_DIR=/opt/rocm-5.2.1/llvm/lib/cmake/llvm/
make
make install
4

Okay, thanks. Looking at your hipsycl-info output, it appears your hipSYCL installation is not functional.

Could you:

  • Check whether libomp.so exists somewhere in /opt/rocm-5.2.1/ (e.g., by running find /opt/rocm-5.2.1/ -name libomp.so)? If not, installing openmp-extras-runtime5.2.1 package (if you installed ROCm) and rebuilding hipSYCL should help.
  • Check that the correct ROCm/LLVM combo is used by librt-backend-hip.so (run ldd /usr/local/lib/hipSYCL/librt-backend-hip.so and see whether libamdhip64.so and libLLVM-*.so point to the reasonable directories)?
  • Try doing export LD_LIBRARY_PATH=/opt/rocm-5.2.1/llvm/lib/:/opt/rocm-5.2.1/lib:$LD_LIBRARY_PATH, and then again run hipsycl-info in the same console.
5 
(base) icaspell@icaspell-B450M-S2H-V2:/media/icaspell/F8FCADECFCADA57A/gromacs_sycl/gromacs-2023/build$ find /opt/rocm-5.2.1/ -name libomp.so
/opt/rocm-5.2.1/llvm/lib/libomp.so
/opt/rocm-5.2.1/llvm/lib-debug/libomp.so
(base) icaspell@icaspell-B450M-S2H-V2:/media/icaspell/F8FCADECFCADA57A/gromacs_sycl/gromacs-2023/build$ ldd /usr/local/lib/hipSYCL/librt-backend-hip.so
	linux-vdso.so.1 (0x00007ffd87bec000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fe7d232d000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe7d21de000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fe7d21c3000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe7d1fd1000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe7d2554000)
(base) icaspell@icaspell-B450M-S2H-V2:/media/icaspell/F8FCADECFCADA57A/gromacs_sycl/gromacs-2023/build$ export LD_LIBRARY_PATH=/opt/rocm-5.2.1/llvm/lib/:/opt/rocm-5.2.1/lib:$LD_LIBRARY_PATH
(base) icaspell@icaspell-B450M-S2H-V2:/media/icaspell/F8FCADECFCADA57A/gromacs_sycl/gromacs-2023/build$ hipsycl-info
[hipSYCL Warning] backend_loader: Could not load backend plugin: /usr/local/lib/hipSYCL/librt-backend-hip.so
[hipSYCL Warning] /usr/local/lib/hipSYCL/librt-backend-hip.so: undefined symbol: hipEventQuery
=================Backend information===================
Loaded backend 0: OpenMP
  Found device: hipSYCL OpenMP host device

=================Device information===================
***************** Devices for backend OpenMP *****************
Device 0:
 General device information:
  Name: hipSYCL OpenMP host device
  Backend: OpenMP
  Vendor: the hipSYCL project
  Arch: <native-cpu>
  Driver version: 1.2
  Is CPU: 1
  Is GPU: 0
 Default executor information:
  Is in-order queue: 1
  Is out-of-order queue: 0
  Is task graph: 0
 Device support queries:
  images: 0
  error_correction: 0
  host_unified_memory: 1
  little_endian: 1
  global_mem_cache: 1
  global_mem_cache_read_only: 0
  global_mem_cache_read_write: 1
  emulated_local_memory: 1
  sub_group_independent_forward_progress: 0
  usm_device_allocations: 1
  usm_host_allocations: 1
  usm_atomic_host_allocations: 1
  usm_shared_allocations: 1
  usm_atomic_shared_allocations: 1
  usm_system_allocations: 1
  execution_timestamps: 1
 Device properties:
  max_compute_units: 12
  max_global_size0: 18446744073709551615
  max_global_size1: 18446744073709551615
  max_global_size2: 18446744073709551615
  max_group_size: 1024
  max_num_sub_groups: 18446744073709551615
  preferred_vector_width_char: 4
  preferred_vector_width_double: 1
  preferred_vector_width_float: 1
  preferred_vector_width_half: 2
  preferred_vector_width_int: 1
  preferred_vector_width_long: 1
  preferred_vector_width_short: 2
  native_vector_width_char: 4
  native_vector_width_double: 1
  native_vector_width_float: 1
  native_vector_width_half: 2
  native_vector_width_int: 1
  native_vector_width_long: 1
  native_vector_width_short: 2
  max_clock_speed: 0
  max_malloc_size: 18446744073709551615
  address_bits: 64
  max_read_image_args: 0
  max_write_image_args: 0
  image2d_max_width: 0
  image2d_max_height: 0
  image3d_max_width: 0
  image3d_max_height: 0
  image3d_max_depth: 0
  image_max_buffer_size: 0
  image_max_array_size: 0
  max_samplers: 0
  max_parameter_size: 18446744073709551615
  mem_base_addr_align: 8
  global_mem_cache_line_size: 64
  global_mem_cache_size: 1
  global_mem_size: 18446744073709551615
  max_constant_buffer_size: 18446744073709551615
  max_constant_args: 18446744073709551615
  local_mem_size: 18446744073709551615
  printf_buffer_size: 18446744073709551615
  partition_max_sub_devices: 0
  vendor_id: 18446744073709551615
  sub_group_sizes: 1

using the environment variable and running make check fixes the errors, the tests are running now will update when the results come

6

Well, it’s still not fully functional: hipSYCL does not detect any GPUs. So, the tests work, but only on the CPU.

Looking at your ldd output, hipSYCL is not even attempting to use HIP.

Could you try cleaning your hipSYCL build directory, and re-run cmake, this time also adding -DROCM_PATH=/opt/rocm-5.2.1/? And with the LD_LIBRARY_PATH set.

7

OK here is the output of the test just in case. Surprisingly I saw an increase in GPU activity while running the test. will rebuild hipSYCL with LD_LIBRARY_PATH and update.
test.log (32.6 KB)

8

Interesting. GROMACS is detecting a GPU, while hipsycl-info does not.

The test timeout is suspicious, but can happen, especially if you also running graphics on this device.

So, according to tests, things seem to work okay.

Using a newer ROCm version might help (we’ve seen noticeable performance improvements with 5.3.x compared to earlier versions), but not sure you want to experiment more :)

9

Yes I want to. will update to 5.3 first thing in the morning than do a clean install of both hipsycl and gromacs and update the post.

1 Like
10

If you’re updating, I think it would be better to go straight to the latest version, 5.4.3. We haven’t tested it as extensively as 5.3.3 internally, but it works just fine.

1 Like
11

Hello…
sorry for the long wait. I updated to 2023.1 and 5.4.3 and I am getting way worse performance compared to 2023 & 5.2.1. I attached the log file for a run on the same system using both of these combinations. what could be the reason for that and should I downgrade.
2023.1_&_5.4.3.log (34.8 KB)
2023_&_5.2.1.log (3.0 MB)

12

Hi!

Thanks for sharing. That’s weird, especially the huge “Rest” part in the performance counters.

Since the “new” run is much shorter than the old one, could you please run with -resetstep 5000 (feel free to adjust it if GROMACS complains that it coincides with PME tuning) to exclude the one-off GPU initialization part from the timing measurement? It should not have such a huge impact, but it will make measurements easier to compare.

Another thing to try is to leave a couple of CPU cores unoccupied for the ROCm/hipSYCL runtime to do its background work (-ntomp 10 -pin on).

13

How long should I let it run for with -resetstep 5000 and -ntomp 10 -pin on ?
I ran it with -resetstep 5000 and -ntomp 10 -pin on for ~100k step and here is the log file
step5_5.log (88.3 KB)

14

I am sorry but I have another question, can I have mdrun utilize both of my integrated and dedicated gpu ?

15

I ran it with -resetstep 5000 and -ntomp 10 -pin on for ~100k step and here is the log file

Yes, that’s long enough. Now the timings look much more reasonable, but still, it’s ~15% slower. Weird.

Do you have a chance to rebuild GROMACS with -DGMX_CYCLE_SUBCOUNTERS=ON and then run with ROCm 5.2.1 and 5.4.3, keeping the rest of the settings identical?

You can set -resetstep 5000 -nsteps 20000 mdrun options to automatically stop the simulation after 20k steps (should be enough).

Another performance tip: you’re using Nose-Hoover thermostat, which is not compatible with GPU update. If the physics of your system allows the use of V-rescale, the things can run faster if you set tcoupl = v-rescale in the MDP and add -update gpu -bonded gpu mdrun flags.

can I have mdrun utilize both of my integrated and dedicated gpu ?

That’s a good question.

As long as hipsycl-info detects both devices, you can: set -ntmpi 2 -ntpme 1, then check that the log says something like

Mapping of GPU IDs to the 2 GPU tasks in the 2 rank on this node:
  PP:0,PME:1

(there’s typically less PME work than PP work, so we use the “big” GPU for PP)

But you probably should not. Using several GPUs is different from using a single, slightly bigger GPU. With PME enabled, you can only split work in specific ways, so if one GPU is 10x slower, but you offload 25% of the computations to it, the whole performance will drop. Moreover, using multiple GPUs adds a significant overhead of coordinating/synchronizing work and exchanging data between them. There is a way to reduce this overhead with direct GPU-GPU communication, but that’s currently supported only for discreet GPUs and is aimed at HPC/supercomputing systems.

tl;dr: it’s possible, but will be slower.