# 1GPU vs 4 GPU per single node; performance **URL:** \ **Category:** User discussions\ **Tags:** mdrun\ **Created:** [April 12, 2021, 7:00pm UTC](https://gromacs.bioexcel.eu/t/1gpu-vs-4-gpu-per-single-node-performance/1950 "2021-04-12T19:00:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Alex](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@Alex](https://gromacs.bioexcel.eu/u/Alex)\ **Post date:** [April 12, 2021, 7:00pm UTC](https://gromacs.bioexcel.eu/t/1gpu-vs-4-gpu-per-single-node-performance/1950/1 "2021-04-12T19:00:58Z")
GROMACS version: 2020.1 GROMACS modification: No Dear all, I run a system, around 400k atoms, on 1 node which has 40 core and and 10 compatible GPUs. When I select one GPU the performance is 31 ns/day while selecting 4 GPUs only enhance the performance to 43 ns/day, so the performance doesn’t scale up accordingly. Would you please let me know what I am using wrongly or need to consider to have a better performance? For further details, please find below the shared log files of the two simulations. Middle parts have just been deleted to make the log files smaller. > **[diff.part0010.log](https://drive.google.com/file/d/196unDMeZ4KaHzocAu2G-Kr71LhDDHvZu/view?usp=sharing)** > > Google Drive file. > **[diff.part0011.log](https://drive.google.com/file/d/1S2OJ3XO93ht6VSgRU-u1FE_iLR2a9lw1/view?usp=sharing)** > > Google Drive file. Regards, Alex ---
**Author:** ![Alex](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@Alex](https://gromacs.bioexcel.eu/u/Alex)\ **Post date:** [April 19, 2021, 5:32am UTC](https://gromacs.bioexcel.eu/t/1gpu-vs-4-gpu-per-single-node-performance/1950/2 "2021-04-19T05:32:16Z")
@pszilard Any comment would be highly appreciated! Thank you. ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [April 19, 2021, 9:07am UTC](https://gromacs.bioexcel.eu/t/1gpu-vs-4-gpu-per-single-node-performance/1950/3 "2021-04-19T09:07:46Z")
Alex, I think there is not a lot you can do to improve this. The two main reasons this does not scale is that: - you are not using the GPU-resident parallelization, that is `-update gpu`, and therefore data needs to be moved GPU-\>CPU-\>GPU every step; in addition - the machine you are running on is not ideal for strong scaling (without GPU-resident loop): it is a dense GPU machine slow interconnect; as it has only PCIe 3.0 and only 8 lanes per GPU (so max 6 GB/s), the data transfers are the main bottleneck. Try using GPU update so data can reside for tens to hundreds of steps on the GPU. If your machine only has PCIe to communicate GPU\<-\>GPU (not only CPU\<-\>GPU), your scaling will likely still be limited by communication. If it had NVLink at least across the GPUs it could scale better. You could try running 8 ranks, 2 per GPU to expose more computation and overlap more of the slow communication, but it is not guaranteed it will help. Cheers, Szilard ---
**Author:** ![Alex](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@Alex](https://gromacs.bioexcel.eu/u/Alex)\ **Post date:** [April 20, 2021, 6:08am UTC](https://gromacs.bioexcel.eu/t/1gpu-vs-4-gpu-per-single-node-performance/1950/4 "2021-04-20T06:08:11Z")
Thank you Szilard for the informative response. Indeed, I set to have the update on GPU using the “export GMX\_FORCE\_UPDATE\_DEFAULT\_GPU=true export GMX\_GPU\_DD\_COMMS=true export GMX\_GPU\_PME\_PP\_COMMS=true”; However, it switches to “update cpu” because of the below conditions pointed out in the log file. %----------- Update task on the GPU was required, by the GMX\_FORCE\_UPDATE\_DEFAULT\_GPU environment variable, but the following condition(s) were not satisfied: Domain decomposition is only supported with constraints when update groups are used. This means constraining all bonds is not supported, except for small molecules, and box sizes close to half the pair-list cutoff are not supported. Will use CPU version of update. %--------------- If I am not wrong it required to have the “constraints = h-bonds” instead of the all-bonds, however the point is that I have already have the constraints = h-bonds in my mdp file. I have a periodic molecule in the system (a slab), which I think it might causes to switch the asked “-update gpu” to “-update cpu”, don’t you? Also, I see bunch of “peer mapping resources exhausted” massages as below in the log file, I wonder how one can avoid these? GPU peer access not enabled between GPUs 0 and 9 due to unexpected return value from cudaDeviceEnablePeerAccess: peer mapping resources exhausted GPU peer access not enabled between GPUs 1 and 9 due to unexpected return value from cudaDeviceEnablePeerAccess: peer mapping resources exhausted Thank you ---
**Author:** ![pszilard](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/pszilard/32/21_2.png) [@pszilard](https://gromacs.bioexcel.eu/u/pszilard)\ **Post date:** [April 21, 2021, 6:57pm UTC](https://gromacs.bioexcel.eu/t/1gpu-vs-4-gpu-per-single-node-performance/1950/5 "2021-04-21T18:57:10Z")
> [@Alex](#): > > If I am not wrong it required to have the “constraints = h-bonds” instead of the all-bonds, however the point is that I have already have the constraints = h-bonds in my mdp file. > I have a periodic molecule in the system (a slab), which I think it might causes to switch the asked “-update gpu” to “-update cpu”, don’t you? I think you are right, but I’ll check that. If update groups are used you should see that reported in the log (i.e. "Using update groups, nr XXX, average size YYY atoms, max radius ZZZ nm). > [@Alex](#): > > Also, I see bunch of “peer mapping resources exhausted” massages as below in the log file, I wonder how one can avoid these? > > GPU peer access not enabled between GPUs 0 and 9 due to unexpected return value from cudaDeviceEnablePeerAccess: peer mapping resources exhausted > > GPU peer access not enabled between GPUs 1 and 9 due to unexpected return value from cudaDeviceEnablePeerAccess: peer mapping resources exhausted I have never seen that error, so we will have to take a look into that. Can you please file an issue on [https://gitlab.com/gromacs/gromacs/-/issues](https://gitlab.com/gromacs/gromacs/-/issues) and attach your log file? Thanks, Sz. ---
**Author:** ![mabraham](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/mabraham/32/33_2.png) [@mabraham](https://gromacs.bioexcel.eu/u/mabraham)\ **Post date:** [April 22, 2021, 10:03am UTC](https://gromacs.bioexcel.eu/t/1gpu-vs-4-gpu-per-single-node-performance/1950/6 "2021-04-22T10:03:51Z")
> Indeed, I set to have the update on GPU… Can we see the whole log file please? Knowing why update groups wasn’t chosen is probably useful to guiding you and/or making a better error message in future. ---
**Author:** ![Alex](https://avatars.discourse-cdn.com/v4/letter/a/db5fbb/32.png) [@Alex](https://gromacs.bioexcel.eu/u/Alex)\ **Post date:** [April 23, 2021, 5:15am UTC](https://gromacs.bioexcel.eu/t/1gpu-vs-4-gpu-per-single-node-performance/1950/7 "2021-04-23T05:15:42Z")
Please find the whole log file shared below: > **[diff.part0026.log](https://drive.google.com/file/d/1XD4etvO9bLO5o6DPICO7FR6621IUTuoi/view?usp=sharing)** > > Google Drive file. Thank you ---
**Author:** ![Pragin](https://avatars.discourse-cdn.com/v4/letter/p/94ad74/32.png) [@Pragin](https://gromacs.bioexcel.eu/u/Pragin)\ **Post date:** [January 20, 2023, 6:29am UTC](https://gromacs.bioexcel.eu/t/1gpu-vs-4-gpu-per-single-node-performance/1950/8 "2023-01-20T06:29:20Z")
[npt\_prod.log](https://gromacs.bioexcel.eu/uploads/short-url/wWWeiQOrbvdPfvHXLttnolXJDr0.log) (14.5 KB) Hello, I am facing similar issue. I was trying to run the simulation on an HPC system with the following command _gmx\_GIMPI mdrun -deffnm npt\_prod -nb gpu -pme gpu -npme 1 -bonded cpu -update gpu_ and found the following error in the SLURM output _Inconsistency in user input:_ _Update task on the GPU was required,_ _but the following condition(s) were not satisfied:_ _Domain decomposition is only supported with constraints when update groups are_ _used. This means constraining all bonds is not supported, except for small_ _molecules, and box sizes close to half the pair-list cutoff are not supported._ I did set the mdp option for constraints as h-bond. Is there any update on the original thread? Or could you kindly suggest any solution for this problem? Let me give a few details about my trial. I am trying to request 2 nodes with 2 NVIDIA Tesla V100 GPU cards per node, for this run. The HPC facility is running on CentOS 7. Thank you