Hi,<div><br></div><div>As Berk said, we have considered creating a benchmark result database/submission both for showing statistics on the web as well as for performance regression testing. Actually, the task was even on a todo list a while ago, but for various reasons it got dropped. The only result was a very simple bash benchmark script which can be found here: git://<a href="http://git.gromacs.org/benchmarks.git">git.gromacs.org/benchmarks.git</a></div>
<div><br></div><div>IMHO the biggest barrier to having such a benchmark database/submission system is the lack of standardized GROMACS benchmark input set. The rest could be done with very little effort and pretty much anyone could help with it.</div>
<div><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
There are several things that would need to be checked for reliable<br>
number though: correct input tpr, empty machine, etc.<br>
So we can't publish the numbers as reliable numbers.<br></blockquote><div><br></div><div><div>Sure, but all that is doable and not rocket science, but mostly straightforward software engineering. If we had </div><div>
i) a standardized set of benchmark inputs and</div><div>ii) some simple working setup (or at least a design and a proof of concept dumb implementation),</div><div>we could ask for help. Without these no one can help with the actual implementation.</div>
</div><div><br></div><div>Cheers,</div><div>Szilárd</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
We were thinking about having two databases, one "certified" one<br>
and one open one where everyone can submit.<br>
<br>
Cheers,<br>
<br>
Berk<div class="HOEnZb"><div class="h5"><br>
<br>
On 10/24/12 17:05 , Raf Ponsaerts wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Dear gmx-developers,<br>
<br>
I don't know whether the following idea already has been proposed, but<br>
here I go...<br>
<br>
Would it be useful to have an online database where people would be able<br>
to submit their performance log after a run?<br>
<br>
It is probably not too difficult to automatically generate a small log<br>
file, such as md.log that contains the input parameters, "mega-flops<br>
accounting output" and the hardware specs (CPU-type, GPU-type, number of<br>
nodes, cores, threads, OS, kernel, parameters gmx-compilation...) that<br>
were used for the run. Otherwise, easily a script could be written that<br>
extracts such data from md.log.<br>
<br>
Depending on the willingness of the gmx-community, such a<br>
performance-logfile could be submitted to an online gmx-database.<br>
<br>
I guess that putting such data in a database (e.g. PostgreSQL) could be<br>
of value to the developers, when statistical information easily can be<br>
derived from such a database. Comparing side-by-side benchmarks is<br>
always tricky issue (different hardware, OS, GMX versions, ...). But<br>
still...<br>
<br>
Scripts that automatically analyze performance-data might also<br>
facilitate or speed up the revision-process, e.g. validate the effect of<br>
a git-commit if a developer keeps track of his performance data in a<br>
database. It could also be useful to code-testers or production users<br>
that would like to keep track - or get an automated overview while<br>
tweaking parameters.<br>
<br>
Since good performance is relevant but not the most important issue,<br>
do not take this proposal as some kind of feature-request!<br>
I guess that the core-devel team does not really have the time to work<br>
on this. But, there might also be other members (gmx-users) of the<br>
gmx-community that wish to spend some time to implement this. So, I<br>
would like to know how relevant and realistic you think this idea is.<br>
<br>
I'm curious to read your thoughts on this.<br>
<br>
regards,<br>
<br>
raf<br>
<br>
--------------------<br>
<br>
Raf Ponsaerts, Phd<br>
KULeuven, Belgium<br>
<br>
</blockquote>
<br>
-- <br>
gmx-developers mailing list<br>
<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/<u></u>mailman/listinfo/gmx-<u></u>developers</a><br>
Please don't post (un)subscribe requests to the list. Use the www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@<u></u>gromacs.org</a>.<br>
</div></div></blockquote></div><br></div>