# GPU usage in FEP calculation **URL:** \ **Category:** User discussions\ **Created:** [June 21, 2023, 8:43am UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659 "2023-06-21T08:43:43Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [June 21, 2023, 8:43am UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/1 "2023-06-21T08:43:43Z")
GROMACS version:2023.1 GROMACS modification: No Hi, Is there a way to get more GPU utilization during FEP simulation since the GPU utilization is only 18℅ when i am running this type of simulation. ---
**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\ **Post date:** [June 21, 2023, 8:13pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/2 "2023-06-21T20:13:17Z")
That is extremely little. Is your system very small? Maybe running two or more FEP simulations simultaneously on the same GPU can help a bit. ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [June 22, 2023, 2:47am UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/3 "2023-06-22T02:47:27Z")
My system has close to 60k atoms. Submitting multiple FEP jobs slows it down further. ---
**Author:** ![hess](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/hess/32/416_2.png) [@hess](https://gromacs.bioexcel.eu/u/hess)\ **Post date:** [June 22, 2023, 1:29pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/4 "2023-06-22T13:29:27Z")
Do the two jobs together finish later than running then in sequence? ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [June 23, 2023, 6:15am UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/5 "2023-06-23T06:15:55Z")
Yes running two job’s finish later if submitted parallel then if it’s submitted in sequence. ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [June 26, 2023, 3:45pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/6 "2023-06-26T15:45:20Z")
Hi, The performance of such simulations will strongly depends on the type of hardware and inputs. Please provide a complete log file and a description of the simulation input to help identifying whether performance can be improved. Cheers, Szilárd ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [June 27, 2023, 9:44am UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/7 "2023-06-27T09:44:24Z")
My workstation has 80 cpu’s and 2 GPU (A5000). The numbering of cpu’s is not consistent which leads to difficulty in pinning the cores. Due some reason I can’t share log file or input file for the simulation ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [June 27, 2023, 3:42pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/8 "2023-06-27T15:42:54Z")
> [@Chetan](#): > > Due some reason I can’t share log file or input file for the simulation You can share a link of a log uploaded to an external service (e.g., Google drive). Without a log file it is hard to judge what is happening. > [@Chetan](#): > > The numbering of cpu’s is not consistent which leads to difficulty in pinning the cores. Not sure what you mean, how did you try to pin to cores? Did you run multiple simulations per GPU. If so, did you use `-pinoffset`? ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [June 27, 2023, 4:23pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/9 "2023-06-27T16:23:25Z")
My computer has 2 GPUs. So when i run one job with ntmpi 1 - ntomp 40 -pin on -pinoffset 0 -gpu\_id 0 command then I get 45% GPU usage. But as i increase the cpus the usage drops. Since there are 2 sockets and one socket has 40 cpu,'s.So the command with 40 cpus runs on one complete socket but increasing the number to 60 doesn’t distribute the job in consistent manner. So I tried to run another job also using similar command on gpu id 1 but then both jobs use gpu to 25%. I have tried various combinations but still the max usage i could get was 45% and it doesn’t scale up from here. ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [June 28, 2023, 4:24pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/10 "2023-06-28T16:24:46Z")
below find the link to input file and production run stats of the log file. > **[FEP\_md.log](https://drive.google.com/file/d/1NPCtqlUomzn-pdw6OIjTuHqRmLua8HQ3/view?usp=sharing)** > > Google Drive file. > **[fep\_md.mdp](https://drive.google.com/file/d/1NGpewnEKRvUZ_nQsDbsgtLHUP_04FlDL/view?usp=sharing)** > > Google Drive file. ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [July 1, 2023, 5:50am UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/11 "2023-07-01T05:50:47Z")
Using external MPI library it does use GPU’s (98 % both gpu usage) with high number -np rank but still due to load imbalance the overeall job is still slow compared to the previouse best with 1 ntmpi 40 ntomp and 1 GPU (45% usage) ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [July 3, 2023, 2:25pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/12 "2023-07-03T14:25:13Z")
This is not a _complete_ log file, please don’t select parts of it share the entire file. Based on the incomplete information, it seems that you are not using GPU resident mode (ie… `-update gpu`), but since the log file is incomplete it is not clear whether you have a compatible integrator. Otherwise, most of the CPU computational cost is in PME, so your best bet to improve performance is to allow PME balancing to shift load to the GPU and perhaps further reduce PME load with a larger PME order. If you have multiple simulations to run, I suggest to focus on maximizing the overall throughput rather than the throughput of an individual simulation by mapping 2-3 runs to a single GPU and splitting the available CPU cores across these simulations (e.g. using `-multidir`). Cheers, Szilárd ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [July 4, 2023, 1:30pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/13 "2023-07-04T13:30:09Z")
you can find the complete log file on below link > **[window\_0\_md.log](https://drive.google.com/file/d/1OF7AgPyGinhmxphK8ZOL6dczg3RQ7eGQ/view?usp=sharing)** > > Google Drive file. Also i have tried update gpu option but in this case it was not supported i will try multidir and check if it helps to do multiple simulations with same speed ---
**Author:** ![shivam1](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@shivam1](https://gromacs.bioexcel.eu/u/shivam1)\ **Post date:** [July 6, 2023, 1:58am UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/14 "2023-07-06T01:58:05Z")
@pszilard @hess @jalemkul Hi I also have the same problem. For Protein-Protein system, Gromacs utilize only max ~15-18% of GPU. > My system specifications are as below: ```auto $lscpu architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 1 Core(s) per socket: 16 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 106 Model name: Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz Stepping: 6 CPU MHz: 3300.000 CPU max MHz: 2901.0000 CPU min MHz: 800.0000 BogoMIPS: 5800.00 Virtualization: VT-x L1d cache: 48K L1i cache: 32K L2 cache: 1280K L3 cache: 24576K NUMA node0 CPU(s): 0-15 NUMA node1 CPU(s): 16-31 ``` > and GPU configuration are as below: ```auto $ lshw -C display product: TU102GL [Quadro RTX 6000/8000] vendor: NVIDIA Corporation physical id: 0 bus info: pci@0000:4b:00.0 version: a1 width: 64 bits clock: 33MHz capabilities: vga_controller bus_master cap_list rom configuration: driver=nvidia latency=0 ``` Command That I run for protein-protein production run is as below: ```auto gmx mdrun -v -noappend -deffnm pro1 -ntmpi 8 -ntomp 4 ``` using the above command it utilizes only **~18% GPU**. > $ nvidia-smi > ±--------------------------------------------------------------------------------------+ > | NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 | > |-----------------------------------------±---------------------±---------------------+ > | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | > | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | > | | | MIG M. | > |=========================================+======================+======================| > | 0 Quadro RTX 6000 Off | 00000000:4B:00.0 Off | Off | > | 42% 66C P2 93W / 260W | 302MiB / 24576MiB | **18%** Default | > | | | N/A | > ±----------------------------------------±---------------------±---------------------+ ±--------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 2357 G /usr/libexec/Xorg 38MiB | | 0 N/A N/A 3204 G /usr/bin/gnome-shell 36MiB | | 0 N/A N/A 188285 C gmx 222MiB | ±--------------------------------------------------------------------------------------+ Please have a look into above info and guide me **How can I maximize the GPU/CPU utilization?** Do I need to modify the mdrun command inputs? or Gromacs in not well configured with GPU? Thanks for your kind consideration. ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [July 14, 2023, 11:18am UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/15 "2023-07-14T11:18:23Z")
Using multidir command 6 windows gets completed in 24 to 28 hrs. So if someone has some benchmark for FEP simulations then please share. ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [July 17, 2023, 1:08pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/16 "2023-07-17T13:08:58Z")
> [@Chetan](#): > > Using multidir command 6 windows gets completed in 24 to 28 hrs. What does that mean, please use a throughput measure e.g. ns/day, otherwise it is hard to compare? What is the GPU utilization? Note that even if individual simulations get slower when you map fewwer CPU cores to them and have them share GPUs, in the end you want to maximize your total throughput, so do compute the _aggregate_ throughput rather than that of a single simulation. ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [July 18, 2023, 5:21pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/17 "2023-07-18T17:21:49Z")
For my system with 61522 atoms the total from 6 windows md run (15ns) reaches 94 ns/day Since I have 80 cores (with hyperthreading) and 2 Gpu’s (RTX A5000). I had used np 6 ,ntomp 12 and gpu\_id 01 option in multidir command. The GPU utilization was close to 40% for both GPU’s. i am sharing one file with some more details. > **[Loading Google Sheets](https://docs.google.com/spreadsheets/d/1as1FZ41MqKs_VB7lkeLZunLHBeJ36Ruv/edit?usp=sharing&ouid=111479468552927493371&rtpof=true&sd=true)** > > This Sheet is private ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [July 19, 2023, 4:35pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/18 "2023-07-19T16:35:16Z")
Without comparing this to something else, one can’t say whether this is optimal or not. Why not run short benchmark runs with 1, 2, 3, 4, 5, 8 runs per GPU (three may not even be ideal since you will be leaving some CPU cores idle)? ---
**Author:** ![Chetan](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@Chetan](https://gromacs.bioexcel.eu/u/Chetan)\ **Post date:** [July 29, 2023, 2:03pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/19 "2023-07-29T14:03:48Z")
I did run some short md runs for 1ns and found that earlier i had used 12 ranks for running 6 windows. So I tried 6 ranks for 6 windows using both Gpu’s and not defining number of thread and total throughput was 109.17 ns/day (combined performance for all 6 windows). Then I increase and decreased the number of windows and giving same number of ranks as the number of windows. The best i could get was running 8 windows with 8 ranks which gave 114.54 ns/day (combined performance of all windows). Since above this number of windows it gives Cuda memory allocation error. And both GPU usage was 40% for the 8 windows parallel run job and also it never crossed this number in other combinations also. ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [August 1, 2023, 4:40pm UTC](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659/20 "2023-08-01T16:40:40Z")
Given that you have 20 CPU cores per GPU which, it sounds like you have a lot of CPU work – perhaps a lot of perturbed atoms? If so there is not a lot more you can do because your run is CPU bound. [Next page](https://gromacs.bioexcel.eu/t/gpu-usage-in-fep-calculation/6659.md?page=2)