Low GROMACS Performance on RTX 5090 Workstation

User discussions
1

GROMACS version: 2025.2
GROMACS modification: No

Hi everyone,

I am seeing unexpectedly low performance on RTX 5090 workstation and would appreciate any suggestions.

For the same GROMACS MD simulation (~113k atoms):

  • RTX 4080 Super (Ryzen 7 5700X, GROMACS 2024.2, CUDA 11.8, -ntomp 8): ~230 ns/day
  • RTX 5090 (Intel Core Ultra 9 285, GROMACS 2025.2, CUDA 12.9, -ntomp 24): ~190–200 ns/day

I also tested -ntmpi 2 -ntomp 12, but performance dropped further to ~48 ns/day, so using multiple MPI ranks on a single GPU appears to be much worse for this system.

The RTX 5090 build is:

  • GROMACS 2025.2 (Thread-MPI)
  • CUDA runtime/compiler 12.9
  • NVIDIA driver 570.211.01 (CUDA 12.8)
  • SIMD: AVX2_256
  • Intel Core Ultra 9 285 (24 physical cores)

The log indicates that all major workloads are already offloaded to the GPU:

  • Nonbonded interactions
  • PME (including FFTs)
  • Coordinate updates
  • Constraints

The relevant timing summary is:

On 1 MPI rank, each using 24 OpenMP threads

Activity Wall time (%)

------------------------------------------------

Force 77.5%

Wait GPU state copy 7.3%

Neighbor search 3.6%

Launch PP GPU ops. 2.2%

PME GPU mesh 2.2%

Rest 1.9%

Unlike another recent RTX 5090 report (Aug 2025) on the forum, the timing report does not show excessive “Rest” time or GPU launch overhead, so I cannot identify an obvious software bottleneck from the log.

During the simulation I consistently observe:

  • GPU utilization: 25–50% (occasionally up to ~60%)
  • GPU power: 200–250 W (well below the 575 W TGP)
  • GPU memory usage: ~1.5 GB
  • SM clock: ~2.8 GHz
  • Memory clock: 13.8 GHz
  • GPU remains in P1 throughout the run

Since the GPU maintains full clocks without thermal or power throttling, I am unsure why the RTX 5090 is still approximately 15–20% slower than the RTX 4080 Super for the same simulation.

One notable difference between the two systems is that the RTX 4080 uses GROMACS 2024.2 built with CUDA 11.8, whereas the RTX 5090 uses GROMACS 2025.2 built with CUDA 12.9. I am not sure whether this could contribute to the observed performance.

I also noticed that published benchmarks from HPC Technologies (GROMACS Benchmark ~ 分子動力学シミュレーション 性能評価 ~ - HPCテクノロジーズ) report even 500 ns/day for a single RTX 5090 using GROMACS 2025.4 (built from source with AVX-512) on an AMD Ryzen 9 9950X system. Although the hardware and benchmark system differ from mine, these results suggest that the RTX 5090 is capable of substantially higher performance than I am observing.

Has anyone observed similar behavior on an RTX 5090 or Intel Core Ultra 9 285? Are there any known issues with this platform, or are there additional settings or build options that I should investigate?

Thank you for any suggestions.

2

Hi!

“Force” reports the CPU force computation, so that’s where the bottleneck is.

You CPU has 8 P-cores and 16 E-cores. The 24 threads are getting split between big and small cores, and GROMACS cannot do any load-balancing between OpenMP threads.

Adding -bonded gpu to move the bonded interactions computation onto the GPU should reduce the load on the CPU and can (depending on the forcefield and other settings) enable fully GPU-resident simulation, which is usually a good idea. It could also be better to do -ntomp 8 -pin on to just use the 8 P-cores.

Multiple threadMPI ranks per GPU are rarely a good idea.

3

Thanks for the suggestions!

I tested -bonded gpu with -ntomp 8 on the same system and verified using taskset/ps that GROMACS was restricted to the 8 P-cores. The log confirms:

PP tasks will do (non-perturbed) short-ranged and most bonded interactions on the GPU
PP task will update and constrain coordinates on the GPU
PME tasks will do all aspects on the GPU
Using 1 MPI thread
Using 8 OpenMP threads

I ran five independent 1-ns benchmarks. The average profile is:

Force ~61.8%
Wait GPU state copy ~12.7%
PME GPU mesh ~2.6%
Average performance: 217.9 ns/day

Individual runs were:

md1: 241.854 ns/day (Force 57.4%)
md2: 198.850 ns/day (Force 62.1%)
md3: 197.374 ns/day (Force 67.6%)
md4: 221.646 ns/day (Force 62.1%)
md5: 229.650 ns/day (Force 59.9%)

So the CPU Force fraction is now around 60% rather than the ~77.5%. However, the GPU utilization remains only about 40–60%, and the overall performance is still lower than I expected for an RTX 5090. It seems the P-/E-core scheduling wasn’t the main bottleneck.

Do you have any suggestions on what else might be limiting performance?