[gmx-users] Optimising mdrun
Robert Cordina
robert.cordina at strath.ac.uk
Thu Apr 2 18:00:02 CEST 2020
Hi Kevin,
I'm working with the HPC cluster administrator and we manged to get much better performance with
mpirun -np 1 gmx_mpi mdrun -ntomp 16
which got it up to 500 ns/day. Will now be trying with
-nb gpu -pme gpu
and also with
-pin on
to see if that helps even further.
Thanks,
Robert
-----Original Message-----
From: gromacs.org_gmx-users-bounces at maillist.sys.kth.se <gromacs.org_gmx-users-bounces at maillist.sys.kth.se> On Behalf Of Kevin Boyd
Sent: 02 April 2020 04:59
To: gmx-users at gromacs.org
Subject: Re: [gmx-users] Optimising mdrun
Hi -
> This setting is using 16 MPI process and using 2 OpenMP thrads per MPI
proces
With ntomp 1 you should only be getting one OpenMP thread, not sure why that's not working. Can you post a link to a log file?
For a small system like that and a powerful GPU, you're likely going to have some inefficiency. Can you run replicates? You'd want to do something like this (though I'm not sure about the mpirun part, I usually do non-MPI gromacs):
mpirun -np 8 gmx_mpi mdrun -nt 8 -ntomp 8 -ntmpi 1 -pin on -pinoffset 0 -nb gpu -pme gpu mpirun -np 8 gmx_mpi mdrun -nt 8 -ntomp 8 -ntmpi 1 -pin on -pinoffset 8 -nb gpu -pme gpu
where the first job runs on cores 0-7 and the second job runs on cores 8-15, sharing the same GPU.
> The log states "Will do PME sum in reciprocal space for electrostatic
interactions"
Reciprocal space does not indicate whether PME was run on the CPU or GPU, it's just reporting math.
On Wed, Apr 1, 2020 at 1:03 AM Robert Cordina <robert.cordina at strath.ac.uk>
wrote:
> *Message sent from a system outside of UConn.*
>
>
> Hi,
>
> I'm trying to use GROMACS 2019.3 on two different machines, simulating
> the same system. The system is made up of 100 identical molecules,
> consisting of 61 united atoms each, in a non-aqueous system. I am
> however getting vastly different performance on the two machines, and
> I don't really understand why. The two machines' specs are as follows:
>
> Machine 1 (stand-alone machine)
>
>
> * 40-core CPU (Intel Xeon Gold 6148 2.4GHz, 3.7GHz, 20C, 10.4GT/s
> 2UPI, 27M Cache, HT (150W) DDR4-2666) - so 20 hyper-threaded cores to
> give a total of 40, although in my case only 20 cores are being used
> given the size of the system (see below)
> * 1 GPU (Nvidia Quadro RTX5000, 16GB, 4DP, VirtualLink (XX20T))
> * Memory (512GB (8x64GB) 2666MHz DDR4 LRDIMM ECC) - although in
> reality only about 10 GB are used during my simulation
> * CUDA 10.10
>
> When I simulate my system, I've found that the optimal mdrun settings
> are extremely simple
>
> mdrun -pin on
>
> This setting uses 1 MPI thread and 20 OpenMP threads, while the PP and
> PME tasks are carried out automatically on the GPU. With this I'm
> getting about 800 ns/day.
>
>
> Machine 2 (part of a HPC cluster - of which I can only use 1 node at a
> time - the below details are for a GPU node)
>
>
> * 16 core CPU (Intel Xeon E5-2600, 2.2 GHz)
> * 1 GPU (NVIDIA Tesla V100-PCIE-32GB)
> * Memory (64GB RAM)
> * CUDA 10.0
>
> I've tried reading the GROMACS documentation, repeatedly, but I really
> can't understand much about how to go about choosing the right mdrun
> settings for this machine. I've tried several, and the best
> performance I could get is with
>
> mpirun -np 16 gmx_mpi mdrun -ntomp 1
>
> Even then, I get a performance of about 45-50 ns/day, which is way
> lower than the 800 ns/day I get with the stand-alone machine. This
> setting is using 16 MPI process and using 2 OpenMP thrads per MPI
> process. Also, only PP tasks are being carried out on the GPU. The
> log states "Will do PME sum in reciprocal space for electrostatic
> interactions". Also I get a Warning "On rank 0: oversubscribing the
> available 16 logical CPU cores per node with 32 threads. This will
> cause considerable performance loss", with Note "Oversubscribing the CPU, will not pin threads".
>
>
> Does anyone have any guidance as to what I should try to get good
> performance on the GPU node of the cluster please? Any help will be
> really appreciated.
>
> Thanks,
>
> Robert
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.g
> romacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Crobert.cordina%40strath.ac.uk%7C315ea393228247714fb608d7d6ba437e%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C637213967796410028&sdata=U3Y5vSj5lZXp68PFEerHc0qVtgPDDUuR169ahdbsnag%3D&reserved=0 before posting!
>
> * Can't post? Read
> https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.g
> romacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Crobert.cordina
> %40strath.ac.uk%7C315ea393228247714fb608d7d6ba437e%7C631e0763153347eba
> 5cd0457bee5944e%7C0%7C0%7C637213967796410028&sdata=9NqaDxydhHN7Lsi
> AbgRzm4qxq0z28RvD3F0MU2Qmn8M%3D&reserved=0
>
> * For (un)subscribe requests visit
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail
> list.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Crobert.cordina%40strath.ac.uk%7C315ea393228247714fb608d7d6ba437e%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C637213967796410028&sdata=IQTo3qrzS4u6RKkGe6%2B94zHz2xSCkV4VndupszFntTg%3D&reserved=0 or send a mail to gmx-users-request at gromacs.org.
>
--
Gromacs Users mailing list
* Please search the archive at https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Crobert.cordina%40strath.ac.uk%7C315ea393228247714fb608d7d6ba437e%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C637213967796410028&sdata=U3Y5vSj5lZXp68PFEerHc0qVtgPDDUuR169ahdbsnag%3D&reserved=0 before posting!
* Can't post? Read https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Crobert.cordina%40strath.ac.uk%7C315ea393228247714fb608d7d6ba437e%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C637213967796410028&sdata=9NqaDxydhHN7LsiAbgRzm4qxq0z28RvD3F0MU2Qmn8M%3D&reserved=0
* For (un)subscribe requests visit
https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Crobert.cordina%40strath.ac.uk%7C315ea393228247714fb608d7d6ba437e%7C631e0763153347eba5cd0457bee5944e%7C0%7C0%7C637213967796415018&sdata=7jrbv1%2F%2B4ZaHRJX38zpPLhJL8Hn3erc%2BknNxTxsHVvI%3D&reserved=0 or send a mail to gmx-users-request at gromacs.org.
More information about the gromacs.org_gmx-users
mailing list