<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 17, 2014 at 1:27 AM, Szilárd Páll <span dir="ltr"><<a href="mailto:pall.szilard@gmail.com" target="_blank">pall.szilard@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi,<br>
<br>
There have been attempts at putting together a benchmark suite, the<br>
first one quite a few years ago resulting in the gmxbench.sh script<br>
and few input systems; more additions have been made a couple of<br>
months ago, everything is here: <a href="http://git.gromacs.org/benchmarks.git" target="_blank">git.gromacs.org/benchmarks.git</a><br>
<span><br>
On Tue, Sep 30, 2014 at 6:22 PM, Mark Abraham <<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>> wrote:<br>
> Hi,<br>
><br>
> Cherry-picking Michael's email into its own thread:<br>
><br>
>> Is there a plan (long term) to do (essentially) automated performance<br>
>> tests so that we<br>
>> can perform consistent(ish) checks for new changes in code, then post the<br>
>> results in an<br>
>> easy(ish) way to interpret for others?<br>
><br>
> There's no organized plan. I've lately been trying to organize a dedicated<br>
> machine here so we can start to do some of this - had we had it and the<br>
> right kinds of tests then various bugs would not have gone unnoticed.<br>
<br>
</span>While a machine is useful, I think the what and how to benchmarks are<br>
to clarify first; given the difficulties in concretizing the benchmark<br>
setup in the past, these aspects require attention earlier rather than<br>
later, I think.</blockquote><div><br></div><div>Indeed, there are multiple objectives that a Gromacs benchmark suite might address...</div><div>1) a sysadmin/user would like to run one, to verify that they have not done a horrible job of installation</div><div>2) some sysadmins/users/devs would want to try to observe maximum performance on given hardware</div><div>3) devs would like a time-efficient way to be able to show that their proposed/actual code change is neutral-or-better</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> Some of the questions that previous attempts brought<br>
up:<br>
- facilitate comparing results (to other codes or older version of<br>
GROMACS) while avoiding the pitfall of using of "smallest" common<br>
denominator features/algorithms (like JACC or the STFC benchmarks);<br></blockquote><div><br></div><div>There's a Daresbury benchmark suite that attempted a large-scale cross-suite comparison, but I don't think we have much to benefit from facilitating that kind of comparison, for a range of reasons. There's merit in maintaining a benchmark that permits comparisons with other codes if the settings are ones that are actually reasonable to use - but e.g. a benchmark that stipulated the neighbour list update frequency and buffer size would be implementable but of doubtful real interest. Choosing a simulation package based on least-common-denominator performance is really not what we should encourage anybody to do.</div><div><br></div><div>Comparing with older versions of Gromacs is worthwhile only over a short time frame, and only to demonstrate that the expected performance behaviour is realized. There's no ongoing value in being able to run a current benchmark suite on 4.5, because it has none of the optimizations we've implemented since. So I think a benchmark set should evolve along with the code, and that backward compatibility should be a fair way down the list of priorities.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
- test algorithms or functionality: one may be interested in the<br>
algorithmic performance while others want to know how fast can one<br>
compute X.</blockquote><div><br></div><div>Indeed, this are my cases 3) & 2). It seems to me unlikely that their solution should have the same form. Use for case 3) might want to test performance on cases that are constructed to be a bit obtuse. For example, showing an improved bonded implementation suggests that you use a setup where you can observe any relevant difference to desirable precision after an efficient amount of computation. That's not necessarily in the regime where the code would be used. So some Martini polymer system with small-cutoff RF on a GPU might be good for a test of bonded-forces performance characteristics, while perhaps being doubtful as a useful model of anything, and thus not in a benchmark suite for 1) or 2).</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><br>
> In<br>
> principle, people could run that test suite on their own hardware, of<br>
> course.<br>
<br>
</span>I think the best would be if many people ran on many different hardware.<br></blockquote><div><br></div><div>There's value there, but only if there's a lightweight way to curate any shared results. "Hi, I compiled GROMACS at -O2 with the default gcc 4.4.7 on our new cluster" is of zero value... Results for cases 1) and 2) are interesting.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
However, I think reproducibility is quite tricky. It can only be<br>
ensured if we get a couple of identical machines, set them up with<br>
identical software and avoid most upgrades (e.g. kernel, libc[++],<br>
etc.) that can affect performance, keeping a machine as backup to<br>
avoid having to rerun all reference runs when the hardware breaks.<br>
Even this will only ensure observing performance on one particular<br>
piece of hardware with the set of algorithms, parallelization,<br>
optimizations actually used.</blockquote><div><br></div><div>Why is reproducibility across machines useful? The absolute performance number on any configuration is of nearly no interest. I think that the change over a short period of time on given hardware is what is of interest for 2) or 3), and that can be measured most reliably by running the old software version again. That doesn't scale for keeping an historical record of trends in performance, but I haven't been able to think of a need for that other than for released versions.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span>
> One option I've been toying with lately is dumping the mdrun performance<br>
> data matrix to XML (or something) so that some existing plotting machinery<br>
> can show the trend over time (and also observe a per-commit delta large<br>
> enough to vote -1 on Jenkins).<br>
<br>
</span>Isn't per commit is an overkill an ill-scaling setup? I think it's<br>
better to do less frequent (e.g. weekly) and per-reqeust performance<br>
regression tests. Proper testing anyway requires running dozens of<br>
combinations of input+launch configurations on several platforms. It's<br>
not a fun task, I know because I've been doing quite extensive<br>
performance testing semi-manually.<br></blockquote><div><br></div><div>Right, per-commit can't be done on a wide range of configurations. But we can test on something per-commit, which is better than we do right now. When we expect a change is the time to trigger some wider-scale testing (e.g. as I have done for some of the C->C++ changes that are now merged), but having any form of "canary in the mine" for unanticipated changes has decent value. It could probably be combined with e.g. tests of ensemble quality.</div><div><br></div><div>Mark</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
--<br>
Szilárd<br>
<div><div><br>
> I also mean to have a poke around with<br>
> <a href="http://www.phoromatic.com/" target="_blank">http://www.phoromatic.com/</a> to see if maybe it already has infrastructure we<br>
> could use.<br>
><br>
> Mark<br>
><br>
</div></div><span><font color="#888888">> --<br>
> Gromacs Developers mailing list<br>
><br>
> * Please search the archive at<br>
> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before<br>
> posting!<br>
><br>
> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
><br>
> * For (un)subscribe requests visit<br>
> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or<br>
> send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</font></span></blockquote></div><br></div></div>