Hi Justin and Mark,<br><br>Thanks once again for getting back.<br><br>I've found the problem - it's actually a known bug already: <br><br><a href="http://redmine.gromacs.org/issues/901">http://redmine.gromacs.org/issues/901</a><br>
<br>The dispersion correction is multiplied my the number of processes (I found this after taking a closer look at my md.log files to see where the energy was changing)! I guess this means I should use the serial version for meaningful binding energies. It also looks like it will be fixed for version 4.5.6<br>
<br>Thank you again, I really appreciate your help.<br><br>Steve<br><br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On 30/05/2012 9:42 PM, Stephen Cox wrote:<br>
> Hi Justin,<br>
><br>
> Thanks for getting back and posting the links.<br>
><br>
><br>
> On 5/29/12 6:22 AM, Stephen Cox wrote:<br>
> > Hi,<br>
> ><br>
> > I'm running a number of energy minimizations on a clathrate<br>
> supercell and I get<br>
> > quite significantly different values for the total energy<br>
> depending on the<br>
> > number of mpi processes / number of threads I use. More<br>
> specifically, some<br>
> > numbers I get are:<br>
> ><br>
> > #cores energy<br>
> > 1 -2.41936409202696e+04<br>
> > 2 -2.43726425776809e+04<br>
> > 3 -2.45516442350804e+04<br>
> > 4 -2.47003944216983e+04<br>
> ><br>
> > #threads energy<br>
> > 1 -2.41936409202696e+04<br>
> > 2 -2.43726425776792e+04<br>
> > 3 -2.45516442350804e+04<br>
> > 4 -2.47306458924815e+04<br>
> ><br>
> ><br>
> > I'd expect some numerical noise, but these differences seem to0<br>
> large for that.<br>
><br>
> The difference is only 2%, which by MD standards, is quite good,<br>
> I'd say ;)<br>
> Consider the discussion here:<br>
><br>
><br>
> I agree for MD this wouldn't be too bad, but I'd expect energy<br>
> minimization to get very close to the same local minimum from a given<br>
> starting configuration. The thing is I want to compute a binding curve<br>
> for my clathrate and compare to DFT values for the binding energy<br>
> (amongst other things), and the difference in energy between different<br>
> number of cores is rather significant for this purpose.<br>
<br>
Given the usual roughness of the PE surface to which you are minimizing,<br>
some variation in end point is expected.<br></blockquote><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
><br>
> Furthermore, if I only calculate the energy for nsteps = 0 (i.e. a<br>
> single point energy for identical structures) I get the same trend as<br>
> above (both mpi/openmp with domain/particle decomposition). Surely<br>
> there shouldn't be such a large difference in energy for a single<br>
> point calculation?<br>
<br>
nsteps = 0 is not strictly a single-point energy, since the constraints<br>
act before EM step 0. mdrun -s -rerun will give a single point. This<br>
probably won't change your observations. You should also be sure you're<br>
making observations with the latest release (4.5.5).<br></blockquote><div></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
If you can continue to observe this trend for more processors<br>
(overallocating?), then you may have evidence of a problem - but a full<br>
system description and an .mdp file will be in order also.<br>
<br>
Mark</blockquote><br><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
><br>
><br>
> <a href="http://www.gromacs.org/Documentation/Terminology/Reproducibility" target="_blank">http://www.gromacs.org/Documentation/Terminology/Reproducibility</a><br>
><br>
> To an extent, the information here may also be relevant:<br>
><br>
> <a href="http://www.gromacs.org/Documentation/How-tos/Extending_Simulations#Exact_vs_binary_identical_continuation" target="_blank">http://www.gromacs.org/Documentation/How-tos/Extending_Simulations#Exact_vs_binary_identical_continuation</a><br>
><br>
> > Before submitting a bug report, I'd like to check:<br>
> > a) if someone has seen something similar;<br>
><br>
> Sure. Energies can be different due to a whole host of factors<br>
> (discussed<br>
> above), and MPI only complicates matters.<br>
><br>
> > b) should I just trust the serial version?<br>
><br>
> Maybe, but I don't know that there's evidence to say that any of<br>
> the above tests<br>
> are more or less accurate than the others. What happens if you<br>
> run with mdrun<br>
> -reprod on all your tests?<br>
><br>
><br>
> Running with -reprod produces the same trend as above. If it was<br>
> numerical noise, I would have thought that the numbers would fluctuate<br>
> around some average value, not follow a definite trend where the<br>
> energy decreases with the number of cores/threads...<br>
><br>
><br>
> > c) have I simply done something stupid (grompp.mdp appended below);<br>
> ><br>
><br>
> Nope, looks fine.<br>
><br>
> -Justin<br>
><br>
> Thanks again for getting back to me.<br>
><br>
><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://lists.gromacs.org/pipermail/gmx-users/attachments/20120530/a4ed4a18/attachment-0001.html" target="_blank">http://lists.gromacs.org/pipermail/gmx-users/attachments/20120530/a4ed4a18/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 30 May 2012 07:51:02 -0400<br>
From: "Justin A. Lemkul" <<a href="mailto:jalemkul@vt.edu">jalemkul@vt.edu</a>><br>
Subject: Re: [gmx-users] Re: Possible bug: energy changes with the<br>
number of nodes for energy minimization<br>
To: Discussion list for GROMACS users <<a href="mailto:gmx-users@gromacs.org">gmx-users@gromacs.org</a>><br>
Message-ID: <<a href="mailto:4FC609A6.4090702@vt.edu">4FC609A6.4090702@vt.edu</a>><br>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br>
<br>
<br>
<br>
On 5/30/12 7:42 AM, Stephen Cox wrote:<br>
> Hi Justin,<br>
><br>
> Thanks for getting back and posting the links.<br>
><br>
><br>
> On 5/29/12 6:22 AM, Stephen Cox wrote:<br>
> > Hi,<br>
> ><br>
> > I'm running a number of energy minimizations on a clathrate supercell and<br>
> I get<br>
> > quite significantly different values for the total energy depending on the<br>
> > number of mpi processes / number of threads I use. More specifically, some<br>
> > numbers I get are:<br>
> ><br>
> > #cores energy<br>
> > 1 -2.41936409202696e+04<br>
> > 2 -2.43726425776809e+04<br>
> > 3 -2.45516442350804e+04<br>
> > 4 -2.47003944216983e+04<br>
> ><br>
> > #threads energy<br>
> > 1 -2.41936409202696e+04<br>
> > 2 -2.43726425776792e+04<br>
> > 3 -2.45516442350804e+04<br>
> > 4 -2.47306458924815e+04<br>
> ><br>
> ><br>
> > I'd expect some numerical noise, but these differences seem to0 large for<br>
> that.<br>
><br>
> The difference is only 2%, which by MD standards, is quite good, I'd say ;)<br>
> Consider the discussion here:<br>
><br>
><br>
> I agree for MD this wouldn't be too bad, but I'd expect energy minimization to<br>
> get very close to the same local minimum from a given starting configuration.<br>
> The thing is I want to compute a binding curve for my clathrate and compare to<br>
> DFT values for the binding energy (amongst other things), and the difference in<br>
> energy between different number of cores is rather significant for this purpose.<br>
><br>
<br>
I think the real issue comes down to how you're going to calculate binding<br>
energy. I would still expect that with sufficient MD sampling, the differences<br>
should be small or statistically insignificant given the nature of MD<br>
calculations. EM will likely be very sensitive to the nature of how it is run<br>
(MPI vs. serial, etc) since even the tiny rounding errors and other factors<br>
described below will cause changes in how the EM algorithm proceeds. For most<br>
purposes, such differences are irrelevant as EM is only a preparatory step for<br>
more intense calculations.<br></blockquote><div><br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> Furthermore, if I only calculate the energy for nsteps = 0 (i.e. a single point<br>
> energy for identical structures) I get the same trend as above (both mpi/openmp<br>
> with domain/particle decomposition). Surely there shouldn't be such a large<br>
> difference in energy for a single point calculation?<br>
><br>
<br>
That depends. Are you using the same .mdp file, just setting "nsteps = 0"? If<br>
so, that's not a good test. EM algorithms will make a change at step 0, the<br>
magnitude of which will again reflect the differences you're seeing. If you use<br>
the md integrator with a zero-step evaluation, that's a better test.<br></blockquote><div><br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
-Justin<br>
<br>
<br></blockquote><div><br></div></div>