<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 16, 2014 at 7:27 PM, Szilárd Páll <span dir="ltr"><<a href="mailto:pall.szilard@gmail.com" target="_blank">pall.szilard@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi,<br>
<br>
There have been attempts at putting together a benchmark suite, the<br>
first one quite a few years ago resulting in the gmxbench.sh script<br>
and few input systems; more additions have been made a couple of<br>
months ago, everything is here: <a href="http://git.gromacs.org/benchmarks.git" target="_blank">git.gromacs.org/benchmarks.git</a></blockquote><div><br></div><div>I get </div><div>fatal: remote error: access denied or repository not exported: /benchmarks.git<br></div><div><br></div><div>Is this not yet readable to all developers? Any reason this isn't in gerrit?</div><div><br></div><div>Roland</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
<br>
On Tue, Sep 30, 2014 at 6:22 PM, Mark Abraham <<a href="mailto:mark.j.abraham@gmail.com">mark.j.abraham@gmail.com</a>> wrote:<br>
> Hi,<br>
><br>
> Cherry-picking Michael's email into its own thread:<br>
><br>
>> Is there a plan (long term) to do (essentially) automated performance<br>
>> tests so that we<br>
>> can perform consistent(ish) checks for new changes in code, then post the<br>
>> results in an<br>
>> easy(ish) way to interpret for others?<br>
><br>
> There's no organized plan. I've lately been trying to organize a dedicated<br>
> machine here so we can start to do some of this - had we had it and the<br>
> right kinds of tests then various bugs would not have gone unnoticed.<br>
<br>
While a machine is useful, I think the what and how to benchmarks are<br>
to clarify first; given the difficulties in concretizing the benchmark<br>
setup in the past, these aspects require attention earlier rather than<br>
later, I think. Some of the questions that previous attempts brought<br>
up:<br>
- facilitate comparing results (to other codes or older version of<br>
GROMACS) while avoiding the pitfall of using of "smallest" common<br>
denominator features/algorithms (like JACC or the STFC benchmarks);<br>
- test algorithms or functionality: one may be interested in the<br>
algorithmic performance while others want to know how fast can one<br>
compute X.<br>
<br>
> In<br>
> principle, people could run that test suite on their own hardware, of<br>
> course.<br>
<br>
I think the best would be if many people ran on many different hardware.<br>
<br>
However, I think reproducibility is quite tricky. It can only be<br>
ensured if we get a couple of identical machines, set them up with<br>
identical software and avoid most upgrades (e.g. kernel, libc[++],<br>
etc.) that can affect performance, keeping a machine as backup to<br>
avoid having to rerun all reference runs when the hardware breaks.<br>
Even this will only ensure observing performance on one particular<br>
piece of hardware with the set of algorithms, parallelization,<br>
optimizations actually used.<br>
<br>
> One option I've been toying with lately is dumping the mdrun performance<br>
> data matrix to XML (or something) so that some existing plotting machinery<br>
> can show the trend over time (and also observe a per-commit delta large<br>
> enough to vote -1 on Jenkins).<br>
<br>
Isn't per commit is an overkill an ill-scaling setup? I think it's<br>
better to do less frequent (e.g. weekly) and per-reqeust performance<br>
regression tests. Proper testing anyway requires running dozens of<br>
combinations of input+launch configurations on several platforms. It's<br>
not a fun task, I know because I've been doing quite extensive<br>
performance testing semi-manually.<br>
<br>
--<br>
Szilárd<br>
<br>
> I also mean to have a poke around with<br>
> <a href="http://www.phoromatic.com/" target="_blank">http://www.phoromatic.com/</a> to see if maybe it already has infrastructure we<br>
> could use.<br>
><br>
> Mark<br>
<span class=""><font color="#888888">><br>
> --<br>
> Gromacs Developers mailing list<br>
><br>
> * Please search the archive at<br>
> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before<br>
> posting!<br>
><br>
> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
><br>
> * For (un)subscribe requests visit<br>
> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or<br>
> send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309
</div></div>