<div dir="ltr">Hi Peicho,<div><br></div><div>The general plan is that we'll be moving away from the separate PP vs. PME nodes in the future, and rather have one MPI task per node that in turns uses threads/cores with slightly different tasks (in particular to enable more advanced load balancing, multiple-time-step algorithms, avoid communication, and also make it easier to handle the cases where multiple GPUs do PME).</div><div><br></div><div>So, for the future I think it would be significantly more impact to try and help with those efforts, or at least reformulate the work to work well with that setup. </div><div><br></div><div>I would also strongly recommend to start working on *small* changes/improvements and gradually get them into the main codebase piece-by-piece while engaging in the rest of the development. There is likely going to be a lot of other work going on related to multiple-time-stepping in the near future, so the likelihood that a large separate branch that has been developed independently with a different type of multiple-time-stepping can later be merged is unfortunately close to zero - at that point the ask would be to rewrite it to be compatible and submitted as a sequence of small changes!</div><div><br></div><div>Cheers,</div><div><br></div><div>Erik</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 14, 2019 at 12:02 PM Peicho Petkov <<a href="mailto:peicho.petkov@gmail.com">peicho.petkov@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Dear Gromacs developers,</div><div><br></div><div>NCSA-Bulgaria aims at optimising Gromacs for Modular Supercomputing Architecture as a participant in the DEEP-EST project. Some of you may remember my presentation at Gromacs Workshop 2018 in Goettingen, but let me illustrate the essence again. We plan to run simulations on two modules namely in Booster-Cluster configuration, where the Booster is high-performance scalable parallel computing system (based on MIC, GPU or another type of accelerators) and the Cluster module - a Linux cluster based on relatively high-performance CPU cores. Both modules’ intraconnecting networks to be chosen in such a way to ensure good performance scalability, while module interconnecting network might be a network federation.</div><div>As you know and I’ve already discussed on the Goettingen workshop with some of you, the performance of the module interconnection network is of crucial importance. While the hardware people in DEEP-EST project are working on the module interconnecting design, we decided to try optimising PP-PME communications. We started developing asynchronous PME electrostatic algorithm. The rough idea is to use multiple-timesteplike approach in such a way that the PME nodes’ performance and communication between PP nodes and PME nodes become less limiting the overall application performance. Even the cluster-booster configuration fails in the future such an algorithm might be of interest (basically but not only) of users who run relatively big and huge parallel simulations on quite a significant number of computing nodes. For instance, such simulations will be less sensitive to the PP to PME nodes number ratio and the interconnecting network properties.</div><div><br></div><div>The main point is to organise communications in the way shown in fig. 1 (see the attached plot) particularly and only in parallel simulations with PME dedicated ranks. The price to be paid is allocating additional memory, do some more calculations and a bit lower but controllable accuracy of electrostatic interactions treatment. We decided to separate electrostatic interaction in two parts fast and slow changing ones as described in [J. Chem. Phys., Vol. 115, No. 5, 1 August 2001] and [J. Chem. Phys., Vol. 116, No. 14, 8 April 2002]. We consider modifying Verlet cutoff scheme only. To do so, we need to add a force and energy calculating kernel which calculate pure cut-off electrostatic interactions with switch function mentioned in [J. Chem. Phys., Vol. 116, No. 14, 8 April 2002] and fast-changing real space terms of PME forces calculated in a separate output buffer. Then we estimate fast-changing real space terms of PME forces for each atom and save them in a buffer to be subtracted from the corresponding forces and energies received from PME nodes in the next time step. The latter is because the forces and energies calculated in the reciprocal space have a small but non-negligible, rapidly changing component (as discussed in [J. Chem. Phys., Vol. 115, No. 5, 1 August 2001]).</div><div><br></div><div>More than a month ago we took a snapshot of the gromacs source code and started developing a proof of concept implementation of reference kernels and added necessary buffers and flags to test the idea, having in mind that all of the building blocks needed for the implementation of the algorithm are already developed in the source. As a result, we almost double the number of operations for calculating non-bonded interactions. The latter was done with the clear understanding that it was not the optimal way, but we wanted to make as fewer changes in the Gromacs source code as possible. We still have not implemented any switch function, and there is energy drift as expected. Currently, we are going to develop one reference kernel with additional arguments which will take pointers to force and energy buffers needed for slow changing force component calculations.</div><div><br></div><div>With this letter, we would like to inform you of our activities, and if you consider that implementation mentioned above could be of interest of Gromacs user community, we will be happy to start working on it with your help. Otherwise, we will appreciate your helpful comments.</div><div><br></div><div>Best regards,</div><div>Peicho, Valentin and Stoyan Markov.</div></div></div>
-- <br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Erik Lindahl <<a href="mailto:erik.lindahl@dbb.su.se" target="_blank">erik.lindahl@dbb.su.se</a>></div><div>Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm University</div><div>Science for Life Laboratory, Box 1031, 17121 Solna, Sweden</div></div></div></div></div></div></div></div></div>