# Optimizing Production Run with GPUs and CHARMM36 **URL:** \ **Category:** User discussions\ **Tags:** gpu, mdrun-performance, simulation-setup\ **Created:** [February 25, 2025, 2:44pm UTC](https://gromacs.bioexcel.eu/t/optimizing-production-run-with-gpus-and-charmm36/11623 "2025-02-25T14:44:28Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![b-omics](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@b-omics](https://gromacs.bioexcel.eu/u/b-omics)\ **Post date:** [February 25, 2025, 2:44pm UTC](https://gromacs.bioexcel.eu/t/optimizing-production-run-with-gpus-and-charmm36/11623/1 "2025-02-25T14:44:28Z")
GROMACS version: 2024.4 GROMACS modification: Yes/No I am simulating a system of approximately 330,000 atoms using **CHARMM36** and aiming to significantly improve the performance of my production run. Currently, I am utilizing **3 GPUs (Nvidia A100) and 64 CPUs** on a single node (my node can support up to 6 GPUs). I have also increased the **nstlist** from 10 to 40 to enhance the efficiency of the pair search. Currently I am getting 134 nsec/day but I see that the PME wait and GPU state copy is 30-35 percent. I am looking for further optimizations to increase the production rate of the **mdrun**. With -tunepme I managed to get 138 nsec. Any suggestions? Log file: gmx mdrun -ntmpi 3 -ntomp 10 -npme 3 -nb gpu -bonded gpu -pme gpu -update gpu -g gpu3\_test\_multigpu\_pme\_optimized -nsteps -1 -maxh 0.017 -resethway -notunepme -s benchmark.tpr -deffnm gpu3\_test\_multigpu\_pme\_optimized ![image](https://europe1.discourse-cdn.com/flex017/uploads/bioexcel1/original/2X/6/60b9f0479f4960090926a2ee8d95e49adfd13ad1.png) ---
**Author:** ![obZehn](https://dub1.discourse-cdn.com/flex017/user_avatar/gromacs.bioexcel.eu/obzehn/32/2629_2.png) [@obZehn](https://gromacs.bioexcel.eu/u/obZehn)\ **Post date:** [February 25, 2025, 10:01pm UTC](https://gromacs.bioexcel.eu/t/optimizing-production-run-with-gpus-and-charmm36/11623/2 "2025-02-25T22:01:35Z")
A couple of general considerations: 1. Run for a bit longer the simulation, I would stick to a couple of mins, even if I do not tune the pme (very minor point) 2. how did you increase the nstlist? Because GROMACS will fine tune it at the beginning of the simulation anyway; if it is in the mdp file then it will be overwritten. It’s probably going to use ~100 3. Have you tried with just one/two GPUs? ---
**Author:** ![b-omics](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@b-omics](https://gromacs.bioexcel.eu/u/b-omics)\ **Post date:** [February 25, 2025, 10:48pm UTC](https://gromacs.bioexcel.eu/t/optimizing-production-run-with-gpus-and-charmm36/11623/3 "2025-02-25T22:48:32Z")
1. From the gromacs wiki, it says to increase the nstlist from 10 to 20-40 for gpus. 2. yes, but with 2 gpus got like 80-90 nsec if i remember correctly ---
**Author:** ![Florian\_Leidner](https://avatars.discourse-cdn.com/v4/letter/f/dbc845/32.png) [@Florian\_Leidner](https://gromacs.bioexcel.eu/u/Florian_Leidner)\ **Post date:** [February 27, 2025, 3:05pm UTC](https://gromacs.bioexcel.eu/t/optimizing-production-run-with-gpus-and-charmm36/11623/4 "2025-02-27T15:05:54Z")
With your setup, offloading everything to the GPUs should be optimal. As a rule of thumb, you want to aim for a 3:1 ratio of NB:PME ranks. You mentioned that your nodes can support up to 6 GPUs, so I would start by requesting 4 GPUs and having one dedicated to PME (i.e. -npme 1) and 3 ranks for everything else. Note that you are limited to one GPU for PME unless you have compiled gromacs with cuFFTMp. You could try requesting more GPUs for (short-range) non-bonded computations, but this is potentially not very efficient because they may have to wait for PME to “catch up”. If you really want to get into the nitty-gritty of performance optimization, you might want to take a look at some of the nvidia blog posts: > **[Creating Faster Molecular Dynamics Simulations with GROMACS 2020 | NVIDIA...](https://developer.nvidia.com/blog/creating-faster-molecular-dynamics-simulations-with-gromacs-2020/)** > > GROMACS—one of the most widely used HPC applications— has received a major upgrade with the release of GROMACS 2020. The new version includes exciting new performance improvements resulting from a… > **[A Guide to CUDA Graphs in GROMACS 2023 | NVIDIA Technical Blog](https://developer.nvidia.com/blog/a-guide-to-cuda-graphs-in-gromacs-2023/)** > > This post describes how CUDA Graphs have been recently leveraged by GROMACS, a simulation package for biomolecular systems and one of the most highly used scientific software applications worldwide. > **[Massively Improved Multi-node NVIDIA GPU Scalability with GROMACS | NVIDIA...](https://developer.nvidia.com/blog/massively-improved-multi-node-nvidia-gpu-scalability-with-gromacs/)** > > GROMACS, a scientific software package widely used for simulating biomolecular systems, plays a crucial role in comprehending important biological processes important for disease prevention and… However, 138 ns/day already seems relatively OK with this hardware and system size.