# Low Performance due to low utilisation of GPU **URL:** \ **Category:** User discussions\ **Created:** [May 16, 2023, 2:15am UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414 "2023-05-16T02:15:40Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![sagarrathee](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@sagarrathee](https://gromacs.bioexcel.eu/u/sagarrathee)\ **Post date:** [May 16, 2023, 2:15am UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/1 "2023-05-16T02:15:40Z")
GROMACS version: 2023 GROMACS modification: No My desktop has Intel(R) Core™ i9-10900K CPU @ 3.70GHz processor and Nvidia RTX 4090 GPU. This is gromacs version installed on my system gmx --version GROMACS version: 2023 Precision: mixed Memory model: 64 bit MPI library: thread\_mpi OpenMP support: enabled (GMX\_OPENMP\_MAX\_THREADS = 128) GPU support: CUDA NB cluster size: 8 SIMD instructions: AVX2\_256 CPU FFT library: fftw-3.3.8-sse2-avx-avx2-avx2\_128 GPU FFT library: cuFFT Multi-GPU FFT: none RDTSCP usage: enabled TNG support: enabled Hwloc support: disabled Tracing support: disabled C compiler: /usr/bin/cc GNU 11.3.0 C compiler flags: -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -O3 -DNDEBUG C++ compiler: /usr/bin/c++ GNU 11.3.0 C++ compiler flags: -fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp BLAS library: LAPACK library: CUDA compiler: /usr/local/cuda-12.1/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2023 NVIDIA Corporation;Built on Tue\_Feb\_\_7\_19:32:13\_PST\_2023;Cuda compilation tools, release 12.1, V12.1.66;Build cuda\_12.1.r12.1/compiler.32415258\_0 CUDA compiler flags:-std=c++17;–generate-code=arch=compute\_50,code=sm\_50;–generate-code=arch=compute\_52,code=sm\_52;–generate-code=arch=compute\_60,code=sm\_60;–generate-code=arch=compute\_61,code=sm\_61;–generate-code=arch=compute\_70,code=sm\_70;–generate-code=arch=compute\_75,code=sm\_75;–generate-code=arch=compute\_80,code=sm\_80;–generate-code=arch=compute\_86,code=sm\_86;–generate-code=arch=compute\_89,code=sm\_89;–generate-code=arch=compute\_90,code=sm\_90;-Wno-deprecated-gpu-targets;–generate-code=arch=compute\_53,code=sm\_53;–generate-code=arch=compute\_80,code=sm\_80;-use\_fast\_math;-Xptxas;-warn-double-usage;-Xptxas;-Werror;-D\_FORCE\_INLINES;-fexcess-precision=fast -funroll-all-loops -mavx2 -mfma -Wno-missing-field-initializers -Wno-cast-function-type-strict -fopenmp CUDA driver: 12.10 CUDA runtime: 12.10 I am running a md where i offloaded everything on GPU gmx mdrun -deffnm md -nb gpu -pme gpu -bonded gpu -update gpu Still my GPU utilisation is less than 10% while my cpu utilisation ~ 100%. Here is log file results ![image](https://europe1.discourse-cdn.com/flex017/uploads/bioexcel1/original/2X/3/3e896f82fb646087a274b50684efc3f69a70a935.png) How to increase my GPU utilization?? Thanks and Regards ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [May 16, 2023, 11:24am UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/2 "2023-05-16T11:24:27Z")
Hi, As you can see from the log output, the work on the CPU side that leads to the behavior you see is: - pair search is taking 35% of the wall-time; it appears that you have nstlist=100 so not sure why is that so high. - “Rest” time 37% this is time mdrun timing does not explicitly account for, perhaps you have some special algorithm enabled? - “Force” time is 25% which is likely due to some bonded types needing the CPU kernels. We should rule out first the possibility that you are running something else on the CPU that is interfering. Can you share a complete log file? Cheers, Szilárd ---
**Author:** ![sagarrathee](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@sagarrathee](https://gromacs.bioexcel.eu/u/sagarrathee)\ **Post date:** [May 16, 2023, 11:48am UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/3 "2023-05-16T11:48:41Z")
> **[Google Drive: Sign-in](https://accounts.google.com/v3/signin/identifier?dsh=S839470236%3A1684237707468989&continue=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1Wtz2YQnEC2N65r-Kyx73QA5sZi4H1Lhl%2Fview%3Fusp%3Dsharing&followup=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1Wtz2YQnEC2N65r-Kyx73QA5sZi4H1Lhl%2Fview%3Fusp%3Dsharing&ifkv=Af_xneEV0yL9W7EwBNWNI7O3imUkjIy9QcG61oMVO_iiAJYr6VNq7gimI19o_Zcow8yFLlHIx-kRcA&osid=1&passive=1209600&service=wise&flowName=GlifWebSignIn&flowEntry=ServiceLogin)** > > Access Google Drive with a Google account (for personal use) or Google Workspace account (for business use). ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [May 16, 2023, 1:32pm UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/4 "2023-05-16T13:32:20Z")
I can’t access that file without signing in, please allow downloads without sign in or upload a file here. ---
**Author:** ![XYT](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/xyt/32/3546_2.png) [@XYT](https://gromacs.bioexcel.eu/u/XYT)\ **Post date:** [July 18, 2024, 4:27am UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/5 "2024-07-18T04:27:58Z")
Hello, I’ve encountered the same problem as you. When I was submitting calculations on my Ubuntu linux system, I realized that my gromacs were very fast before, then I accidentally upgraded the kernel by updating the CUDA and drivers, and **now after downgrading the kernel (5.15.0-71-generic)**, I found out that the gromacs have gone from 130ns/day to 3ns/day. ( **Normally, I was using CPU is 2400% and now the CPU usage is only 1200%. Moreover, my GPU usage is below 10%** ). Here is the configuration and kernel information of the server. ![1](https://europe1.discourse-cdn.com/flex017/uploads/bioexcel1/original/2X/1/13355897f8778a5a814773d45d69383b374984ab.png) The CPU model of the server is: **13th Gen Intel(R) Core™ i7-13700KF**, the GPU model is: 01:00.0 VGA compatible controller: **NVIDIA Corporation Device 2782 (rev a1)**, and the driver information of the graphics card is as follows: ![2](https://europe1.discourse-cdn.com/flex017/uploads/bioexcel1/original/2X/5/5ea10af0722c74de883e518902f6a4a1829786b0.png) The gromacs installation version is **2023.5** , here are the details: ![3](https://europe1.discourse-cdn.com/flex017/uploads/bioexcel1/original/2X/f/f72d845b1fa19ba4f14f39e504083c52df4347b2.png) I would appreciate your help in checking this out and if you need any more information, please do not hesitate to contact me and I look forward to hearing from you. ---
**Author:** ![XYT](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/xyt/32/3546_2.png) [@XYT](https://gromacs.bioexcel.eu/u/XYT)\ **Post date:** [July 18, 2024, 7:35am UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/6 "2024-07-18T07:35:16Z")
Additionally, I added the npt log file for not updating CUDA and its drivers (npt\_win0\_conf0.log) and the npt log file for updating CUDA and its drivers (npt\_win26\_conf455.log). [npt\_win0\_conf0.log](https://gromacs.bioexcel.eu/uploads/short-url/ek1ZUtdX8QnEHE7QlQTYJp3X6wO.log) (59.8 KB) [npt\_win26\_conf455.log](https://gromacs.bioexcel.eu/uploads/short-url/a4kWAWhY014ObCyzH3AozvVh2ZU.log) (61.7 KB) Looking forward to your reply! ---
**Author:** ![al42and](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/al42and/32/1393_2.png) [@al42and](https://gromacs.bioexcel.eu/u/al42and)\ **Post date:** [July 18, 2024, 2:26pm UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/7 "2024-07-18T14:26:21Z")
Hi! Do you have any other compute-intensive workload running on your machine? The following line indicates that the CPU performance is the bottleneck: ```auto Force 1 24 50001 1904.652 156221.791 81.9 ``` And your observation that “Normally, I was using CPU is 2400% and now the CPU usage is only 1200%” (while still running 24 threads) suggests that somehow all 24 threads are put onto 12 (logical) cores now. Since you ask GROMACS to use all logical cores (`-nt 24`), it expects to have them fully available. If you have another application running, you should tell GROMACS to use only some of the cores. Since your CPU has both P- and E-cores, I’d suggest `-ntmpi 1 -ntomp 8 -pin on` to limit GROMACS to using only P-cores and with one thread per physical core. ---
**Author:** ![XYT](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/xyt/32/3546_2.png) [@XYT](https://gromacs.bioexcel.eu/u/XYT)\ **Post date:** [July 19, 2024, 4:57am UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/8 "2024-07-19T04:57:16Z")
> [@al42and](#): > > application Hi al42and! Thank you for your reply. According to your suggestion, I have modified the parameters of NPT with the following commands: > gmx mdrun -v -deffnm npt\_win27\_conf467 -pin on -gpu\_id 0 -ntmpi 1 -ntomp 8 -update gpu [npt\_win27\_conf467.log](https://gromacs.bioexcel.eu/uploads/short-url/xjv4pd03GU1UvbQ7sWx98Vnd34H.log) (61.4 KB) For the same system (with 381731 atoms), the simulation time changed from **3.714 ns/day to 19.936 ns/day, and the CPU utilization was found to be 600%, and the GPU utilization was lower, about 20-40%.** ![CPU utilization is only 600% and not 800%](https://europe1.discourse-cdn.com/flex017/uploads/bioexcel1/original/2X/0/0c610c9249c95b6277c7ee22f53dc03373dd5b45.jpeg) And when I use the command: > gmx mdrun -v -deffnm npt\_win28\_conf473 -pin on -gpu\_id 0 -ntmpi 1 -ntomp 16 -update gpu [npt\_win28\_conf473.log](https://gromacs.bioexcel.eu/uploads/short-url/7t9xEf9znVlzaNTH7hTEMQyBs5C.log) (63.3 KB) The **CPU utilization was found to be 960%** and the screen appeared to have **a lot of calculations about PME**. The GPU utilization was also very low at about 30%. The simulation took **8.925 ns/day**. ![PME frequently appears on the screen](https://europe1.discourse-cdn.com/flex017/uploads/bioexcel1/original/2X/0/019c070f67ed01ba2dddd08789d6ea0d1da38db5.jpeg) In addition, I used the `(kill -STOP PID)` to suspend the application that was calculating in the background before the gromacs calculations were submitted. Thus, all of the above studies are based on the case where no other application is computing in the background. The **log files of the above 2 NPTs** are saved to the attachment. How should I speed up the calculation of groamcs to **improve GPU and CPU utilization**? ![GPU utilization below 50%](https://europe1.discourse-cdn.com/flex017/uploads/bioexcel1/original/2X/a/a59d30f4f2f072805cfad577f8447b781b53adb8.jpeg) If you know it, please provide me some suggestions, thanks for your valuable advice. ---
**Author:** ![al42and](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/al42and/32/1393_2.png) [@al42and](https://gromacs.bioexcel.eu/u/al42and)\ **Post date:** [July 25, 2024, 8:51am UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/9 "2024-07-25T08:51:04Z")
> [@XYT](#): > > And when I use the command: > > > gmx mdrun -v -deffnm npt\_win28\_conf473 -pin on -gpu\_id 0 -ntmpi 1 -ntomp 16 -update gpu > > The **CPU utilization was found to be 960%** In the `top` screenshot, you have `gmx` using 600% of the CPU, but the “load average” is around 30 (the three numbers correspond to the last 1, 5 and 15 minutes), so there’s around 30 active threads competing for the 23 (logical) cores, so there are some other processes using the CPU. A large “Rest” time (from your log) and the observation that a lower value of `ntomp` seems to give better performance also suggest that something else on your system is heavily using the CPU. > [@XYT](#): > > **a lot of calculations about PME**. That’s just status reports from PME autotuning, no need to worry. > [@XYT](#): > > The **log files of the above 2 NPTs** are saved to the attachment. How should I speed up the calculation of groamcs to **improve GPU and CPU utilization**? Offloading bonded to the GPU (`-bonded gpu`) could help move more load from CPU to GPU. You can also increase neighbor search interval (e.g., `-nstlist 200`) while keeping `-update gpu`, this will reduce “Neighbor search” time. If you do all this, there will be very little CPU work, so you can reduce `-ntomp` to 4 or 6, which will reduce interference with whatever else is using the CPU. Ideally, you could pin each load to a separate set of CPU cores (in GROMACS, you can use `-pin`, `-pinoffset`, `-pinstride` options; for other applications, you can use, for example, the `taskset` utility). ---
**Author:** ![XYT](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/xyt/32/3546_2.png) [@XYT](https://gromacs.bioexcel.eu/u/XYT)\ **Post date:** [July 26, 2024, 2:12pm UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/10 "2024-07-26T14:12:42Z")
> [@al42and](#): > > so there are some other processes using the CPU Dear al42and! Thank you very much for your patience in responding. After restarting my server, I typed in the command `htop` and found that all CPUs were being utilized, and finally realized that my server had a virus. The problem caused by the virus has now been resolved. Currently, the system of **16 W atoms** is simulated using **gromacs 2023.5 (GPU-4070Ti)** with a time scale of **150 ns/day**. Thank you for your help and being able to get my problem solved in a timely manner. ---
**Author:** ![XYT](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/xyt/32/3546_2.png) [@XYT](https://gromacs.bioexcel.eu/u/XYT)\ **Post date:** [July 26, 2024, 2:18pm UTC](https://gromacs.bioexcel.eu/t/low-performance-due-to-low-utilisation-of-gpu/6414/11 "2024-07-26T14:18:17Z")
> [@al42and](#): > > If you do all this, there will be very little CPU work, so you can reduce `-ntomp` to 4 or 6, which will reduce interference with whatever else is using the CPU. Thank you for your valuable suggestion, I am currently able to run gromacs properly using my school’s LAN, I will try the method you provided.