<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Oct 17, 2014 at 12:56 PM, Szilárd Páll <span dir="ltr"><<a href="mailto:pall.szilard@gmail.com" target="_blank">pall.szilard@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Fri, Oct 17, 2014 at 2:56 AM, Roland Schulz <<a href="mailto:roland@utk.edu">roland@utk.edu</a>> wrote:<br>
><br>
><br>
> On Thu, Oct 16, 2014 at 7:27 PM, Szilárd Páll <<a href="mailto:pall.szilard@gmail.com">pall.szilard@gmail.com</a>><br>
> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> There have been attempts at putting together a benchmark suite, the<br>
>> first one quite a few years ago resulting in the gmxbench.sh script<br>
>> and few input systems; more additions have been made a couple of<br>
>> months ago, everything is here: <a href="http://git.gromacs.org/benchmarks.git" target="_blank">git.gromacs.org/benchmarks.git</a><br>
><br>
><br>
> I get<br>
> fatal: remote error: access denied or repository not exported:<br>
> /benchmarks.git<br>
><br>
> Is this not yet readable to all developers?<br>
<br>
</span>Looks like it's not, I was not aware of that.<br>
<span class=""><br>
> Any reason this isn't in gerrit?<br>
<br>
</span>No idea, Rossen was managing this server, so he would probably be the<br>
right person to answer these questions.<br>
<br>
IMO git is not the best place for gigabytes of static data, so gerrit<br>
(unless it has improved in recent versions) will likely be even less<br>
appropriate.<br></blockquote><div><br></div><div>I think a good solution is to put large files on some cloud storage and have the link to it in git/gerrit.</div><div><br></div><div>Roland</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
--<br>
Szilárd<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
> Roland<br>
><br>
>><br>
>><br>
>><br>
>> On Tue, Sep 30, 2014 at 6:22 PM, Mark Abraham <<a href="mailto:mark.j.abraham@gmail.com">mark.j.abraham@gmail.com</a>><br>
>> wrote:<br>
>> > Hi,<br>
>> ><br>
>> > Cherry-picking Michael's email into its own thread:<br>
>> ><br>
>> >> Is there a plan (long term) to do (essentially) automated performance<br>
>> >> tests so that we<br>
>> >> can perform consistent(ish) checks for new changes in code, then post<br>
>> >> the<br>
>> >> results in an<br>
>> >> easy(ish) way to interpret for others?<br>
>> ><br>
>> > There's no organized plan. I've lately been trying to organize a<br>
>> > dedicated<br>
>> > machine here so we can start to do some of this - had we had it and the<br>
>> > right kinds of tests then various bugs would not have gone unnoticed.<br>
>><br>
>> While a machine is useful, I think the what and how to benchmarks are<br>
>> to clarify first; given the difficulties in concretizing the benchmark<br>
>> setup in the past, these aspects require attention earlier rather than<br>
>> later, I think. Some of the questions that previous attempts brought<br>
>> up:<br>
>> - facilitate comparing results (to other codes or older version of<br>
>> GROMACS) while avoiding the pitfall of using of "smallest" common<br>
>> denominator features/algorithms (like JACC or the STFC benchmarks);<br>
>> - test algorithms or functionality: one may be interested in the<br>
>> algorithmic performance while others want to know how fast can one<br>
>> compute X.<br>
>><br>
>> > In<br>
>> > principle, people could run that test suite on their own hardware, of<br>
>> > course.<br>
>><br>
>> I think the best would be if many people ran on many different hardware.<br>
>><br>
>> However, I think reproducibility is quite tricky. It can only be<br>
>> ensured if we get a couple of identical machines, set them up with<br>
>> identical software and avoid most upgrades (e.g. kernel, libc[++],<br>
>> etc.) that can affect performance, keeping a machine as backup to<br>
>> avoid having to rerun all reference runs when the hardware breaks.<br>
>> Even this will only ensure observing performance on one particular<br>
>> piece of hardware with the set of algorithms, parallelization,<br>
>> optimizations actually used.<br>
>><br>
>> > One option I've been toying with lately is dumping the mdrun performance<br>
>> > data matrix to XML (or something) so that some existing plotting<br>
>> > machinery<br>
>> > can show the trend over time (and also observe a per-commit delta large<br>
>> > enough to vote -1 on Jenkins).<br>
>><br>
>> Isn't per commit is an overkill an ill-scaling setup? I think it's<br>
>> better to do less frequent (e.g. weekly) and per-reqeust performance<br>
>> regression tests. Proper testing anyway requires running dozens of<br>
>> combinations of input+launch configurations on several platforms. It's<br>
>> not a fun task, I know because I've been doing quite extensive<br>
>> performance testing semi-manually.<br>
>><br>
>> --<br>
>> Szilárd<br>
>><br>
>> > I also mean to have a poke around with<br>
>> > <a href="http://www.phoromatic.com/" target="_blank">http://www.phoromatic.com/</a> to see if maybe it already has infrastructure<br>
>> > we<br>
>> > could use.<br>
>> ><br>
>> > Mark<br>
>> ><br>
>> > --<br>
>> > Gromacs Developers mailing list<br>
>> ><br>
>> > * Please search the archive at<br>
>> > <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before<br>
>> > posting!<br>
>> ><br>
>> > * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
>> ><br>
>> > * For (un)subscribe requests visit<br>
>> > <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a><br>
>> > or<br>
>> > send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
>> --<br>
>> Gromacs Developers mailing list<br>
>><br>
>> * Please search the archive at<br>
>> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before<br>
>> posting!<br>
>><br>
>> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
>><br>
>> * For (un)subscribe requests visit<br>
>> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or<br>
>> send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
><br>
><br>
><br>
><br>
> --<br>
> ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>
> <a href="tel:865-241-1537" value="+18652411537">865-241-1537</a>, ORNL PO BOX 2008 MS6309<br>
><br>
> --<br>
> Gromacs Developers mailing list<br>
><br>
> * Please search the archive at<br>
> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before<br>
> posting!<br>
><br>
> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
><br>
> * For (un)subscribe requests visit<br>
> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or<br>
> send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309
</div></div>