<div dir="ltr">The command-line is:<br>> srun -n 512 -N 32 --cpu_bind=cores -c 16 gmx_mpi mdrun -ntomp 4 -npme 256 -nstlist 35 -nsteps 2000 -resetstep 1000 -pin off -noconfout -v -notunepme<br><br>Log file information:<br>=====================================================================<br>Initializing Domain Decomposition on 512 ranks<br>Dynamic load balancing: auto<br>Initial maximum inter charge-group distances:<br> two-body bonded interactions: 0.393 nm, LJ-14, atoms 198 204<br> multi-body bonded interactions: 0.393 nm, Proper Dih., atoms 198 204<br>Minimum cell size due to bonded interactions: 0.433 nm<br>Maximum distance for 5 constraints, at 120 deg. angles, all-trans: 0.740 nm<br>Estimated maximum distance required for P-LINCS: 0.740 nm<br>This distance will limit the DD cell size, you can override this with -rcon<br>Using 256 separate PME ranks, per user request<br>Scaling the initial minimum size with 1/0.8 (option -dds) = 1.25<br>Optimizing the DD grid for 256 cells with a minimum initial size of 0.926 nm<br>The maximum allowed number of cells is: X 16 Y 16 Z 16<br>Domain decomposition grid 16 x 4 x 4, separate PME ranks 256<br>PME domain decomposition: 16 x 16 x 1<br>Interleaving PP and PME ranks<br>This rank does only particle-particle work.<br><br>Domain decomposition rank 0, coordinates 0 0 0<br><br>The initial number of communication pulses is: X 2 Y 1 Z 1<br>The initial domain decomposition cell size is: X 0.94 nm Y 3.74 nm Z 3.74 nm<br><br>The maximum allowed distance for charge groups involved in interactions is:<br> non-bonded interactions 1.200 nm<br> two-body bonded interactions (-rdd) 1.200 nm<br> multi-body bonded interactions (-rdd) 0.935 nm<br> atoms separated by up to 5 constraints (-rcon) 0.935 nm<br><br>When dynamic load balancing gets turned on, these settings will change to:<br>The maximum number of communication pulses is: X 2 Y 2 Z 2<br>The minimum size for domain decomposition cells is 0.740 nm<br>The requested allowed shrink of DD cells (option -dds) is: 0.80<br>The allowed shrink of domain decomposition cells is: X 0.79 Y 0.20 Z 0.20<br>The maximum allowed distance for charge groups involved in interactions is:<br> non-bonded interactions 1.200 nm<br> two-body bonded interactions (-rdd) 1.200 nm<br> multi-body bonded interactions (-rdd) 0.740 nm<br> atoms separated by up to 5 constraints (-rcon) 0.740 nm<br>======================================================================<br><br><div><div class="gmail_extra"><div class="gmail_quote">On Tue, Sep 19, 2017 at 4:39 PM, Szilárd Páll <span dir="ltr"><<a href="mailto:pall.szilard@gmail.com" target="_blank">pall.szilard@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">That sounds extreme -- although not unfamiliar (the same issue come up<br>
in GPU accelerated runs).<br>
<br>
What cut-off and grid spacing? And what node-count, is this at low,<br>
medium or high parallelization?<br>
--<br>
Szilárd<br>
<br>
<br>
On Tue, Sep 19, 2017 at 6:08 PM, John Eblen <<a href="mailto:jeblen@acm.org">jeblen@acm.org</a>> wrote:<br>
> Hi<br>
><br>
> On KNL, I achieve best performance by using the maximum number of PME nodes<br>
> allowed.<br>
> So I figured removing the upper bound might allow for even better<br>
> performance.<br>
><br>
><br>
> John<br>
><br>
> On Tue, Sep 19, 2017 at 5:50 AM, Szilárd Páll <<a href="mailto:pall.szilard@gmail.com">pall.szilard@gmail.com</a>><br>
> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> Why would you want to increase the MPI rank count in PME? Is it to<br>
>> compensate for the thread scaling being worse than in the PP ranks?<br>
>><br>
>> It might be more worthwhile improving PME multi-threading rather than<br>
>> allowing higher rank count.<br>
>><br>
>> --<br>
>> Szilárd<br>
>><br>
>><br>
>> On Tue, Sep 19, 2017 at 10:01 AM, Berk Hess <<a href="mailto:hess@kth.se">hess@kth.se</a>> wrote:<br>
>> > On 2017-09-18 18:34, John Eblen wrote:<br>
>> ><br>
>> > Hi Szilárd<br>
>> ><br>
>> > These runs used 2M huge pages. I will file a redmine shortly.<br>
>> ><br>
>> > On a related topic, how difficult would it be to modify GROMACS to<br>
>> > support ><br>
>> > 50%<br>
>> > PME nodes?<br>
>> ><br>
>> > That's not so hard, but I see little benefit, since then the MPI<br>
>> > communication is not reduced much compared to all ranks doing PME.<br>
>> ><br>
>> > Berk<br>
>> ><br>
>> ><br>
>> ><br>
>> > John<br>
>> ><br>
>> > On Fri, Sep 15, 2017 at 6:37 PM, Szilárd Páll <<a href="mailto:pall.szilard@gmail.com">pall.szilard@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> Hi John,<br>
>> >><br>
>> >> Thanks for diagnosing the issue!<br>
>> >><br>
>> >> We have been aware of this behavior, but been both intentional (as we<br>
>> >> re-scan grids after the first pass at least once more); plus, it's<br>
>> >> also simply been considered a "not too big of a deal" given that in<br>
>> >> general mdrun has very low memory footprint. However, it seems that,<br>
>> >> at least on this particular machine, our assumption was wrong. What is<br>
>> >> the page sizes on Cori KNL?<br>
>> >><br>
>> >> Can you please file a redmine with your observations?<br>
>> >><br>
>> >> Thanks,<br>
>> >> --<br>
>> >> Szilárd<br>
>> >><br>
>> >><br>
>> >> On Fri, Sep 15, 2017 at 8:25 PM, John Eblen <<a href="mailto:jeblen@acm.org">jeblen@acm.org</a>> wrote:<br>
>> >> > This issue appears to not be a GROMACS problem so much as a problem<br>
>> >> > with<br>
>> >> > "huge pages" that is<br>
>> >> > triggered by PME tuning. PME tuning creates a large data structure<br>
>> >> > for<br>
>> >> > every<br>
>> >> > cutoff that it tries, which<br>
>> >> > is replicated on each PME node. These data structures are not freed<br>
>> >> > during<br>
>> >> > tuning, so memory usage<br>
>> >> > expands. Normally it is still too small to cause problems. With huge<br>
>> >> > pages,<br>
>> >> > however, I get errors from<br>
>> >> > "libhugetlbfs" and very slow runs if more than about five cutoffs are<br>
>> >> > attempted.<br>
>> >> ><br>
>> >> > Sample output on NERSC Cori KNL with 32 nodes. Input system size is<br>
>> >> > 248,101<br>
>> >> > atoms.<br>
>> >> ><br>
>> >> > step 0<br>
>> >> > step 100, remaining wall clock time: 24 s<br>
>> >> > step 140: timed with pme grid 128 128 128, coulomb cutoff 1.200:<br>
>> >> > 66.2<br>
>> >> > M-cycles<br>
>> >> > step 210: timed with pme grid 112 112 112, coulomb cutoff 1.336:<br>
>> >> > 69.6<br>
>> >> > M-cycles<br>
>> >> > step 280: timed with pme grid 100 100 100, coulomb cutoff 1.496:<br>
>> >> > 63.6<br>
>> >> > M-cycles<br>
>> >> > step 350: timed with pme grid 84 84 84, coulomb cutoff 1.781: 85.9<br>
>> >> > M-cycles<br>
>> >> > step 420: timed with pme grid 96 96 96, coulomb cutoff 1.559: 68.8<br>
>> >> > M-cycles<br>
>> >> > step 490: timed with pme grid 100 100 100, coulomb cutoff 1.496:<br>
>> >> > 68.3<br>
>> >> > M-cycles<br>
>> >> > libhugetlbfs [nid08887:140420]: WARNING: New heap segment map at<br>
>> >> > 0x10001200000 failed: Cannot allocate memory<br>
>> >> > libhugetlbfs [nid08881:97968]: WARNING: New heap segment map at<br>
>> >> > 0x10001200000 failed: Cannot allocate memory<br>
>> >> > libhugetlbfs [nid08881:97978]: WARNING: New heap segment map at<br>
>> >> > 0x10001200000 failed: Cannot allocate memory<br>
>> >> ><br>
>> >> > Szilárd, to answer to your questions: This is the verlet scheme. The<br>
>> >> > problem<br>
>> >> > happens during tuning, and<br>
>> >> > no problems occur if -notunepme is used. In fact, the best<br>
>> >> > performance<br>
>> >> > thus<br>
>> >> > far has been with 50% PME<br>
>> >> > nodes, using huge pages, and '-notunepme'.<br>
>> >> ><br>
>> >> ><br>
>> >> > John<br>
>> >> ><br>
>> >> > On Wed, Sep 13, 2017 at 6:20 AM, Szilárd Páll<br>
>> >> > <<a href="mailto:pall.szilard@gmail.com">pall.szilard@gmail.com</a>><br>
>> >> > wrote:<br>
>> >> >><br>
>> >> >> Forking the discussion as now we've learned more about the issue Åke<br>
>> >> >> is reporting and it is quiterather dissimilar.<br>
>> >> >><br>
>> >> >> On Mon, Sep 11, 2017 at 8:09 PM, John Eblen <<a href="mailto:jeblen@acm.org">jeblen@acm.org</a>> wrote:<br>
>> >> >> > Hi Szilárd<br>
>> >> >> ><br>
>> >> >> > No, I'm not using the group scheme.<br>
>> >> >><br>
>> >> >> $ grep -i 'cutoff-scheme' md.log<br>
>> >> >> cutoff-scheme = Verlet<br>
>> >> >><br>
>> >> >> > The problem seems similar because:<br>
>> >> >> ><br>
>> >> >> > 1) Deadlocks and very slow runs can be hard to distinguish.<br>
>> >> >> > 2) Since Mark mentioned it, I assume he believes PME tuning is a<br>
>> >> >> > possible<br>
>> >> >> > cause, which is also the cause in my situation.<br>
>> >> >><br>
>> >> >> Does that mean you tested with "-notunepme" and the excessive memory<br>
>> >> >> usage could not be reproduced? Did the memory usage increase only<br>
>> >> >> during the tuning or did it keep increasing after the tuning<br>
>> >> >> completed?<br>
>> >> >><br>
>> >> >> > 3) Åke may be experiencing higher-than-normal memory usage as far<br>
>> >> >> > as<br>
>> >> >> > I<br>
>> >> >> > know.<br>
>> >> >> > Not sure how you know otherwise.<br>
>> >> >> > 4) By "successful," I assume you mean the tuning had completed.<br>
>> >> >> > That<br>
>> >> >> > doesn't<br>
>> >> >> > mean, though, that the tuning could not be creating conditions<br>
>> >> >> > that<br>
>> >> >> > causes the<br>
>> >> >> > problem, like an excessively high cutoff.<br>
>> >> >><br>
>> >> >> Sure. However, it's unlikely that the tuning creates conditions<br>
>> >> >> under<br>
>> >> >> which the run proceeds after the after the initial tuning phase and<br>
>> >> >> keeps allocating memory (which is more prone to be the source of<br>
>> >> >> issues).<br>
>> >> >><br>
>> >> >> I suggest to first rule our the bug I linked and if that's not the<br>
>> >> >> culprit, we can have a closer look.<br>
>> >> >><br>
>> >> >> Cheers,<br>
>> >> >> --<br>
>> >> >> Szilárd<br>
>> >> >><br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> > John<br>
>> >> >> ><br>
>> >> >> > On Mon, Sep 11, 2017 at 1:09 PM, Szilárd Páll<br>
>> >> >> > <<a href="mailto:pall.szilard@gmail.com">pall.szilard@gmail.com</a>><br>
>> >> >> > wrote:<br>
>> >> >> >><br>
>> >> >> >> John,<br>
>> >> >> >><br>
>> >> >> >> In what way do you think your problem is similar? Åke seems to be<br>
>> >> >> >> experiencing a deadlock after successful PME tuning, much later<br>
>> >> >> >> during<br>
>> >> >> >> the run, but no excessive memory usage.<br>
>> >> >> >><br>
>> >> >> >> Do you happen to be using the group scheme with 2016.x (release<br>
>> >> >> >> code)?<br>
>> >> >> >><br>
>> >> >> >> Your issue sounds more like it could be related to the the<br>
>> >> >> >> excessive<br>
>> >> >> >> tuning bug with group scheme fixed quite a few months ago, but<br>
>> >> >> >> it's<br>
>> >> >> >> yet to be released (<a href="https://redmine.gromacs.org/issues/2200" rel="noreferrer" target="_blank">https://redmine.gromacs.org/<wbr>issues/2200</a>).<br>
>> >> >> >><br>
>> >> >> >> Cheers,<br>
>> >> >> >> --<br>
>> >> >> >> Szilárd<br>
>> >> >> >><br>
>> >> >> >><br>
>> >> >> >> On Mon, Sep 11, 2017 at 6:50 PM, John Eblen <<a href="mailto:jeblen@acm.org">jeblen@acm.org</a>><br>
>> >> >> >> wrote:<br>
>> >> >> >> > Hi<br>
>> >> >> >> ><br>
>> >> >> >> > I'm having a similar problem that is related to PME tuning.<br>
>> >> >> >> > When<br>
>> >> >> >> > it<br>
>> >> >> >> > is<br>
>> >> >> >> > enabled, GROMACS often, but not<br>
>> >> >> >> > always, slows to a crawl and uses excessive amounts of memory.<br>
>> >> >> >> > Using<br>
>> >> >> >> > "huge<br>
>> >> >> >> > pages" and setting a high<br>
>> >> >> >> > number of PME processes seems to exacerbate the problem.<br>
>> >> >> >> ><br>
>> >> >> >> > Also, occurrences of this problem seem to correlate with how<br>
>> >> >> >> > high<br>
>> >> >> >> > the<br>
>> >> >> >> > tuning<br>
>> >> >> >> > raises the cutoff value.<br>
>> >> >> >> ><br>
>> >> >> >> > Mark, can you give us more information on the problems with PME<br>
>> >> >> >> > tuning?<br>
>> >> >> >> > Is<br>
>> >> >> >> > there a redmine?<br>
>> >> >> >> ><br>
>> >> >> >> ><br>
>> >> >> >> > Thanks<br>
>> >> >> >> > John<br>
>> >> >> >> ><br>
>> >> >> >> > On Mon, Sep 11, 2017 at 10:53 AM, Mark Abraham<br>
>> >> >> >> > <<a href="mailto:mark.j.abraham@gmail.com">mark.j.abraham@gmail.com</a>><br>
>> >> >> >> > wrote:<br>
>> >> >> >> >><br>
>> >> >> >> >> Hi,<br>
>> >> >> >> >><br>
>> >> >> >> >> Thanks. Was PME tuning active? Does it reproduce if that is<br>
>> >> >> >> >> disabled?<br>
>> >> >> >> >> Is<br>
>> >> >> >> >> the PME tuning still active? How many steps have taken place<br>
>> >> >> >> >> (at<br>
>> >> >> >> >> least<br>
>> >> >> >> >> as<br>
>> >> >> >> >> reported in the log file but ideally from processes)?<br>
>> >> >> >> >><br>
>> >> >> >> >> Mark<br>
>> >> >> >> >><br>
>> >> >> >> >> On Mon, Sep 11, 2017 at 4:42 PM Åke Sandgren<br>
>> >> >> >> >> <<a href="mailto:ake.sandgren@hpc2n.umu.se">ake.sandgren@hpc2n.umu.se</a>><br>
>> >> >> >> >> wrote:<br>
>> >> >> >> >>><br>
>> >> >> >> >>> My debugger run finally got to the lockup.<br>
>> >> >> >> >>><br>
>> >> >> >> >>> All processes are waiting on various MPI operations.<br>
>> >> >> >> >>><br>
>> >> >> >> >>> Attached a stack dump of all 56 tasks.<br>
>> >> >> >> >>><br>
>> >> >> >> >>> I'll keep the debug session running for a while in case<br>
>> >> >> >> >>> anyone<br>
>> >> >> >> >>> wants<br>
>> >> >> >> >>> some more detailed data.<br>
>> >> >> >> >>> This is a RelwithDeb build though so not everything is<br>
>> >> >> >> >>> available.<br>
>> >> >> >> >>><br>
>> >> >> >> >>> On 09/08/2017 11:28 AM, Berk Hess wrote:<br>
>> >> >> >> >>> > But you should be able to get some (limited) information by<br>
>> >> >> >> >>> > attaching a<br>
>> >> >> >> >>> > debugger to an aldready running process with a release<br>
>> >> >> >> >>> > build.<br>
>> >> >> >> >>> ><br>
>> >> >> >> >>> > If you plan on compiling and running a new case, use a<br>
>> >> >> >> >>> > release<br>
>> >> >> >> >>> > +<br>
>> >> >> >> >>> > debug<br>
>> >> >> >> >>> > symbols build. That should run as fast as a release build.<br>
>> >> >> >> >>> ><br>
>> >> >> >> >>> > Cheers,<br>
>> >> >> >> >>> ><br>
>> >> >> >> >>> > Berk<br>
>> >> >> >> >>> ><br>
>> >> >> >> >>> > On 2017-09-08 11:23, Åke Sandgren wrote:<br>
>> >> >> >> >>> >> We have, at least, one case that when run over 2 nodes, or<br>
>> >> >> >> >>> >> more,<br>
>> >> >> >> >>> >> quite<br>
>> >> >> >> >>> >> often (always) hangs, i.e. no more output in md.log or<br>
>> >> >> >> >>> >> otherwise<br>
>> >> >> >> >>> >> while<br>
>> >> >> >> >>> >> mdrun still consumes cpu time. It takes a random time<br>
>> >> >> >> >>> >> before<br>
>> >> >> >> >>> >> it<br>
>> >> >> >> >>> >> happens,<br>
>> >> >> >> >>> >> like 1-3 days.<br>
>> >> >> >> >>> >><br>
>> >> >> >> >>> >> The case can be shared if someone else wants to<br>
>> >> >> >> >>> >> investigate.<br>
>> >> >> >> >>> >> I'm<br>
>> >> >> >> >>> >> planning to run it in the debugger to be able to break and<br>
>> >> >> >> >>> >> look<br>
>> >> >> >> >>> >> at<br>
>> >> >> >> >>> >> states when it happens, but since it takes so long with<br>
>> >> >> >> >>> >> the<br>
>> >> >> >> >>> >> production<br>
>> >> >> >> >>> >> build it is not something i'm looking forward to.<br>
>> >> >> >> >>> >><br>
>> >> >> >> >>> >> On 09/08/2017 11:13 AM, Berk Hess wrote:<br>
>> >> >> >> >>> >>> Hi,<br>
>> >> >> >> >>> >>><br>
>> >> >> >> >>> >>> We are far behind schedule for the 2017 release. We are<br>
>> >> >> >> >>> >>> working<br>
>> >> >> >> >>> >>> hard<br>
>> >> >> >> >>> >>> on<br>
>> >> >> >> >>> >>> it, but I don't think we can promise a date yet.<br>
>> >> >> >> >>> >>><br>
>> >> >> >> >>> >>> We have a 2016.4 release planned for this week (might<br>
>> >> >> >> >>> >>> slip<br>
>> >> >> >> >>> >>> to<br>
>> >> >> >> >>> >>> next<br>
>> >> >> >> >>> >>> week). But if you can give us enough details to track<br>
>> >> >> >> >>> >>> down<br>
>> >> >> >> >>> >>> your<br>
>> >> >> >> >>> >>> hanging<br>
>> >> >> >> >>> >>> issue, we might be able to fix it in 2016.4.<br>
>> >> >> >> >>> ><br>
>> >> >> >> >>><br>
>> >> >> >> >>> --<br>
>> >> >> >> >>> Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden<br>
>> >> >> >> >>> Internet: <a href="mailto:ake@hpc2n.umu.se">ake@hpc2n.umu.se</a> Phone: <a href="tel:%2B46%2090%207866134" value="+46907866134">+46 90 7866134</a> Fax: +46<br>
>> >> >> >> >>> 90-580<br>
>> >> >> >> >>> 14<br>
>> >> >> >> >>> Mobile: <a href="tel:%2B46%2070%207716134" value="+46707716134">+46 70 7716134</a> WWW: <a href="http://www.hpc2n.umu.se" rel="noreferrer" target="_blank">http://www.hpc2n.umu.se</a><br>
>> >> >> >> >>> --<br>
>> >> >> >> >>> Gromacs Developers mailing list<br>
>> >> >> >> >>><br>
>> >> >> >> >>> * Please search the archive at<br>
>> >> >> >> >>><br>
>> >> >> >> >>> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists/GMX-<wbr>developers_List</a><br>
>> >> >> >> >>> before<br>
>> >> >> >> >>> posting!<br>
>> >> >> >> >>><br>
>> >> >> >> >>> * Can't post? Read<br>
>> >> >> >> >>> <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists</a><br>
>> >> >> >> >>><br>
>> >> >> >> >>> * For (un)subscribe requests visit<br>
>> >> >> >> >>><br>
>> >> >> >> >>><br>
>> >> >> >> >>><br>
>> >> >> >> >>><br>
>> >> >> >> >>> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/<wbr>mailman/listinfo/gromacs.org_<wbr>gmx-developers</a><br>
>> >> >> >> >>> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@<wbr>gromacs.org</a>.<br>
>> >> >> >> >><br>
>> >> >> >> >><br>
>> >> >> >> >> --<br>
>> >> >> >> >> Gromacs Developers mailing list<br>
>> >> >> >> >><br>
>> >> >> >> >> * Please search the archive at<br>
>> >> >> >> >><br>
>> >> >> >> >> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists/GMX-<wbr>developers_List</a><br>
>> >> >> >> >> before<br>
>> >> >> >> >> posting!<br>
>> >> >> >> >><br>
>> >> >> >> >> * Can't post? Read<br>
>> >> >> >> >> <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists</a><br>
>> >> >> >> >><br>
>> >> >> >> >> * For (un)subscribe requests visit<br>
>> >> >> >> >><br>
>> >> >> >> >><br>
>> >> >> >> >><br>
>> >> >> >> >> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/<wbr>mailman/listinfo/gromacs.org_<wbr>gmx-developers</a><br>
>> >> >> >> >> or<br>
>> >> >> >> >> send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@<wbr>gromacs.org</a>.<br>
>> >> >> >> ><br>
>> >> >> >> ><br>
>> >> >> >> ><br>
>> >> >> >> > --<br>
>> >> >> >> > Gromacs Developers mailing list<br>
>> >> >> >> ><br>
>> >> >> >> > * Please search the archive at<br>
>> >> >> >> ><br>
>> >> >> >> > <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists/GMX-<wbr>developers_List</a><br>
>> >> >> >> > before<br>
>> >> >> >> > posting!<br>
>> >> >> >> ><br>
>> >> >> >> > * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists</a><br>
>> >> >> >> ><br>
>> >> >> >> > * For (un)subscribe requests visit<br>
>> >> >> >> ><br>
>> >> >> >> ><br>
>> >> >> >> ><br>
>> >> >> >> > <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/<wbr>mailman/listinfo/gromacs.org_<wbr>gmx-developers</a><br>
>> >> >> >> > or<br>
>> >> >> >> > send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@<wbr>gromacs.org</a>.<br>
>> >> >> >> --<br>
>> >> >> >> Gromacs Developers mailing list<br>
>> >> >> >><br>
>> >> >> >> * Please search the archive at<br>
>> >> >> >> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists/GMX-<wbr>developers_List</a><br>
>> >> >> >> before<br>
>> >> >> >> posting!<br>
>> >> >> >><br>
>> >> >> >> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists</a><br>
>> >> >> >><br>
>> >> >> >> * For (un)subscribe requests visit<br>
>> >> >> >><br>
>> >> >> >><br>
>> >> >> >> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/<wbr>mailman/listinfo/gromacs.org_<wbr>gmx-developers</a><br>
>> >> >> >> or<br>
>> >> >> >> send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@<wbr>gromacs.org</a>.<br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> > --<br>
>> >> >> > Gromacs Developers mailing list<br>
>> >> >> ><br>
>> >> >> > * Please search the archive at<br>
>> >> >> > <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists/GMX-<wbr>developers_List</a><br>
>> >> >> > before<br>
>> >> >> > posting!<br>
>> >> >> ><br>
>> >> >> > * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists</a><br>
>> >> >> ><br>
>> >> >> > * For (un)subscribe requests visit<br>
>> >> >> ><br>
>> >> >> ><br>
>> >> >> > <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/<wbr>mailman/listinfo/gromacs.org_<wbr>gmx-developers</a><br>
>> >> >> > or<br>
>> >> >> > send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@<wbr>gromacs.org</a>.<br>
>> >> ><br>
>> >> ><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Gromacs Developers mailing list<br>
>> ><br>
>> > * Please search the archive at<br>
>> > <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists/GMX-<wbr>developers_List</a> before<br>
>> > posting!<br>
>> ><br>
>> > * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists</a><br>
>> ><br>
>> > * For (un)subscribe requests visit<br>
>> > <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/<wbr>mailman/listinfo/gromacs.org_<wbr>gmx-developers</a><br>
>> > or<br>
>> > send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@<wbr>gromacs.org</a>.<br>
>> --<br>
>> Gromacs Developers mailing list<br>
>><br>
>> * Please search the archive at<br>
>> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists/GMX-<wbr>developers_List</a> before<br>
>> posting!<br>
>><br>
>> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists</a><br>
>><br>
>> * For (un)subscribe requests visit<br>
>> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/<wbr>mailman/listinfo/gromacs.org_<wbr>gmx-developers</a> or<br>
>> send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@<wbr>gromacs.org</a>.<br>
><br>
><br>
<span class="gmail-HOEnZb"><font color="#888888">><br>
> --<br>
> Gromacs Developers mailing list<br>
><br>
> * Please search the archive at<br>
> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists/GMX-<wbr>developers_List</a> before<br>
> posting!<br>
><br>
> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists</a><br>
><br>
> * For (un)subscribe requests visit<br>
> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/<wbr>mailman/listinfo/gromacs.org_<wbr>gmx-developers</a> or<br>
> send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@<wbr>gromacs.org</a>.<br>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists/GMX-<wbr>developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/<wbr>Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/<wbr>mailman/listinfo/gromacs.org_<wbr>gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@<wbr>gromacs.org</a>.</font></span></blockquote></div><br></div></div></div>