Low GROMACS Performance on RTX 5090 Workstation

GROMACS version: 2025.2
GROMACS modification: No

Hi everyone,

I am seeing unexpectedly low performance on RTX 5090 workstation and would appreciate any suggestions.

For the same GROMACS MD simulation (~113k atoms):

  • RTX 4080 Super (Ryzen 7 5700X, GROMACS 2024.2, CUDA 11.8, -ntomp 8): ~230 ns/day
  • RTX 5090 (Intel Core Ultra 9 285, GROMACS 2025.2, CUDA 12.9, -ntomp 24): ~190–200 ns/day

I also tested -ntmpi 2 -ntomp 12, but performance dropped further to ~48 ns/day, so using multiple MPI ranks on a single GPU appears to be much worse for this system.

The RTX 5090 build is:

  • GROMACS 2025.2 (Thread-MPI)
  • CUDA runtime/compiler 12.9
  • NVIDIA driver 570.211.01 (CUDA 12.8)
  • SIMD: AVX2_256
  • Intel Core Ultra 9 285 (24 physical cores)

The log indicates that all major workloads are already offloaded to the GPU:

  • Nonbonded interactions
  • PME (including FFTs)
  • Coordinate updates
  • Constraints

The relevant timing summary is:

On 1 MPI rank, each using 24 OpenMP threads

Activity Wall time (%)

------------------------------------------------

Force 77.5%

Wait GPU state copy 7.3%

Neighbor search 3.6%

Launch PP GPU ops. 2.2%

PME GPU mesh 2.2%

Rest 1.9%

Unlike another recent RTX 5090 report (Aug 2025) on the forum, the timing report does not show excessive “Rest” time or GPU launch overhead, so I cannot identify an obvious software bottleneck from the log.

During the simulation I consistently observe:

  • GPU utilization: 25–50% (occasionally up to ~60%)
  • GPU power: 200–250 W (well below the 575 W TGP)
  • GPU memory usage: ~1.5 GB
  • SM clock: ~2.8 GHz
  • Memory clock: 13.8 GHz
  • GPU remains in P1 throughout the run

Since the GPU maintains full clocks without thermal or power throttling, I am unsure why the RTX 5090 is still approximately 15–20% slower than the RTX 4080 Super for the same simulation.

One notable difference between the two systems is that the RTX 4080 uses GROMACS 2024.2 built with CUDA 11.8, whereas the RTX 5090 uses GROMACS 2025.2 built with CUDA 12.9. I am not sure whether this could contribute to the observed performance.

I also noticed that published benchmarks from HPC Technologies (GROMACS Benchmark ~ 分子動力学シミュレーション 性能評価 ~ - HPCテクノロジーズ) report even 500 ns/day for a single RTX 5090 using GROMACS 2025.4 (built from source with AVX-512) on an AMD Ryzen 9 9950X system. Although the hardware and benchmark system differ from mine, these results suggest that the RTX 5090 is capable of substantially higher performance than I am observing.

Has anyone observed similar behavior on an RTX 5090 or Intel Core Ultra 9 285? Are there any known issues with this platform, or are there additional settings or build options that I should investigate?

Thank you for any suggestions.

Hi!

“Force” reports the CPU force computation, so that’s where the bottleneck is.

You CPU has 8 P-cores and 16 E-cores. The 24 threads are getting split between big and small cores, and GROMACS cannot do any load-balancing between OpenMP threads.

Adding -bonded gpu to move the bonded interactions computation onto the GPU should reduce the load on the CPU and can (depending on the forcefield and other settings) enable fully GPU-resident simulation, which is usually a good idea. It could also be better to do -ntomp 8 -pin on to just use the 8 P-cores.

Multiple threadMPI ranks per GPU are rarely a good idea.

Thanks for the suggestions!

I tested -bonded gpu with -ntomp 8 on the same system and verified using taskset/ps that GROMACS was restricted to the 8 P-cores. The log confirms:

PP tasks will do (non-perturbed) short-ranged and most bonded interactions on the GPU
PP task will update and constrain coordinates on the GPU
PME tasks will do all aspects on the GPU
Using 1 MPI thread
Using 8 OpenMP threads

I ran five independent 1-ns benchmarks. The average profile is:

Force ~61.8%
Wait GPU state copy ~12.7%
PME GPU mesh ~2.6%
Average performance: 217.9 ns/day

Individual runs were:

md1: 241.854 ns/day (Force 57.4%)
md2: 198.850 ns/day (Force 62.1%)
md3: 197.374 ns/day (Force 67.6%)
md4: 221.646 ns/day (Force 62.1%)
md5: 229.650 ns/day (Force 59.9%)

So the CPU Force fraction is now around 60% rather than the ~77.5%. However, the GPU utilization remains only about 40–60%, and the overall performance is still lower than I expected for an RTX 5090. It seems the P-/E-core scheduling wasn’t the main bottleneck.

Do you have any suggestions on what else might be limiting performance?

You can increase the number of OpenMP threads. I assume you can theoretically run 24 threads on your 8 physical cores, although I would not advise it. I run simulations on AWS and have this same issue.

You can watch your real-time computational load from your terminal with the command

top

and in another window run the command

nvidia-smi

This will show your usage between the two components. I usually use a g5.16xl instance (32 cores) with an A10 Nvidia GPU (24GB) with CUDA 13.0, and my memory usage is low (10% of maximum), while the GPU utility is at 80%.

Perhaps you can increase your OpenMP thread count to 20 to leave a little headroom for background tasks, but this is your bottleneck.

1 GPU selected for this run.
Mapping of GPU IDs to the 2 GPU tasks in the 1 rank on this node:
PP:0,PME:0
PP tasks will do (non-perturbed) short-ranged and most bonded interactions on the GPU
PP task will update and constrain coordinates on the CPU
PME tasks will do all aspects on the GPU
Using 1 MPI process

Non-default thread affinity set, disabling internal thread affinity

Using 32 OpenMP threads

top

Tasks: 629 total, 3 running, 626 sleeping, 0 stopped, 0 zombie
%Cpu(s): 76.6 us, 0.5 sy, 0.0 ni, 22.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 255002.4 total, 248985.8 free, 3766.4 used, 2250.2 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 248810.5 avail Mem

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND              

37490 ec2-user 20 0 8823796 1.1g 348968 R 2781 0.4 470:23.48 gmx_mpi
36796 ec2-user 20 0 9068960 1.3g 372140 R 2146 0.5 450:14.53 gmx_mpi

nvidia-smi

Wed Jul 15 18:22:52 2026
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.82.07 Driver Version: 580.82.07 CUDA Version: 13.0 |
±----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 |
| 0% 76C P0 285W / 300W | 1102MiB / 23028MiB | 100% Default |
| | | N/A |
±----------------------------------------±-----------------------±---------------------+

±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 36796 C gmx_mpi 556MiB |
| 0 N/A N/A 37490 C gmx_mpi 532MiB |
±----------------------------------------------------------------------------------------+

That being said, 220 ns/day is not too shabby. It’s definitely better than my 24 ns/day in my 1 million-atom simulations