<br><br><div class="gmail_quote">On Sun, Feb 5, 2012 at 3:53 PM, Shirts, Michael (mrs5pt) <span dir="ltr"><<a href="mailto:mrs5pt@eservices.virginia.edu">mrs5pt@eservices.virginia.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi, all-<br>
<br>
My opinion:<br>
<br>
I think there should probably be two classes of sets -- fast fully automated<br>
sets, and more sophisticated content validation sets.<br>
<br>
For the fast fully automated test, I would suggest:<br>
-testing a large range of input .mdps, tops and gros for whether they run<br>
through grompp and mdrun. Not performing whether the output is correct or<br>
not, because that is very hard to automate -- just testing whether it runs<br>
for, say 20-100 steps or so.<br></blockquote><div><br></div><div>Yes having that set of inputs is needed. Should we start a wiki page to start listing all the inputs we want to include? Or what would be the best way to collaborative creating this set of inputs?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Longer term, we should look more at validating code at a physical level.<br>
Clearly testing energy conservation is a good idea for integrators; it's<br>
fairly sensitive. I think we need to discuss a bit more about how to<br>
evaluate energy conservation. This actually can take a fair amount of time,<br>
and I'm thinking this is something that perhaps should wait for 5.0. For<br>
thermostats and barostats, I'm working on a very sensitive test of ensemble<br>
validity. I'll email a copy to the list when it's ready to go (1-2 weeks?),<br>
and this is something that can also be incorporated in an integrated testing<br>
regime, but again, these sorts of tests will take multiple hours, not<br>
seconds. That sort of testing level can't be part of the day to day build.<br></blockquote><div>Well even if the tests take ~2000CPU-hours, I think we (maybe even Erik by himself) have the resources to run this weekly.</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"><br></div>
<div class="im"><br>
> - What are the requirements for the new test set? E.g. how easy should it<br>
> be to see whats wrong when a test fails?<br>
<br>
</div>For the first set of tests, I can imagine that it would be nice to be able<br>
to look at the outputs of the tests, and diff different outputs<br>
corresponding to different code versions to help track down changes were.<br>
But I'm suspicious about making the evaluation of these tests decided on<br>
automatically at first. I agree such differences should EVENTUALLY be<br>
automated, but I'd prefer several months of investigation and discussion<br>
before deciding exactly what "correct" is.<br></blockquote><div><br></div><div>I think a wrong reference value is better than no reference value. Even a wrong reference value would allow us to detect if e.g. different compilers give significant different results (maybe some give the correct value). Also it would help to avoid adding additional bugs. Of course we shouldn't release the test set to the outside before we are relative sure that it actually correct.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
> Should the test support being run<br>
> under valgrind? Other?<br>
<br>
</div>Valgrind is incredibly slow and can fail for weird reasons -- I'm not sure<br>
it would add much to do it under valgrind.<br></blockquote><div>I have the current C++ tests (those written by Teemu) running under valgrind in Jenkins. It wasn't very hard to write a few suppression rules to make valgrind not report any false positives. Now Jenkins can automatically fail the build if the code has any memory errors. Obviously one woudn't run any of the long running tests with valgrind. But for the unit tests I think it might be very useful to catch bugs.</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I DON'T think we should have any test set that starts to look at more<br>
complicated features right now -- it will take months to get that working,<br>
and we need to get 4.6 out of the door on the order of weeks, so we can move<br>
on to the next improvements. 4.6 doesn't have to be perfectly flawless, as<br>
long as it's closer to perfect than 4.5.<br></blockquote><div><br></div><div>My reason for delaying the 4.6 release would not be to improve the 4.6 release. I agree with you we probably can't guarantee that the reference value are correct in time anyhow, so we probably wouldn't even want to ship the tests with 4.6. My worry is that as soon as 4.6 is out the focus is on adding new cool features instead of working on these boring tasks we should do, because they help us in the long run. E.g. if we would have agreed that we don't have a 4.6 release, the C++ conversion would most likely be much further along. And I don't see how we can create an incentive mechanism to work on these issues without somehow coupling it to releases.</div>
<div><br></div><div>Roland</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Best,<br>
~~~~~~~~~~~~<br>
Michael Shirts<br>
Assistant Professor<br>
Department of Chemical Engineering<br>
University of Virginia<br>
<a href="mailto:michael.shirts@virginia.edu">michael.shirts@virginia.edu</a><br>
<a href="tel:%28434%29-243-1821" value="+14342431821">(434)-243-1821</a><br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
--<br>
gmx-developers mailing list<br>
<a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
<br>
<br>
<br>
<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309<br>