<div>Hi </div>
<div>I have a few comments and suggestions that I hope will help. I have been designing software and building products for 30 years in industry in the area of compilers, optimizers, databases, imaging and other software where testing is a critical issue. In many cases this involved thousands or even millions of copies. I have been working on performance improvements to GROMACS and the difficulties of knowing if I broke something has been a big issue. <br>
<br>1) Unit and system test suites are developed as<i> products go through release cycles and as bugs are being fixed</i>. Trying to build
tests after a release or near the end of a release cycle does not work
because of limitations on resources, the desire to get the release out or because of work on future releases. <br><br>2) The important thing for 4.6 is to catch the big things and priority 1 bugs through the GROMACS community before it goes out and put in
place a simple manageable process going forward for testing. I would not release it with a priority 1 bug. (e.g. if Reaction Field is broken I would not release 4.6). <br><br>3) If conventions for a test framework can quickly be established and the developers of 4.6 functionality and recent critical bug fixes
have tests that they have used for their work that may reasonably be adapted to the new test framework they should be added. <br><br>4) The foundation of test suites is the unit/module level. If the base is not sound then the system/simulation will not be sound. At the module level it is almost always possible to determine what the 'right' answer is with relatively small datasets in short amounts of time for many parameter sets. A set of pass/fail test programs at module level is a good <i>long term</i> goal. It seems unrealistic to me to try to go back and build module level tests for GROMACS with limited resources, money and time. It is<i> more realistic to add tests as bugs are fixed and new features are added.</i> <br>
<br>5) Keep the test framework very simple. There are number of 'test environments' and products available. I have used a lot of them in commercial product development I would not recommend the use of any of them for GROMACS. <br>
<br>The US National Institute of Standards and Technology has test suites for compilers, databases and many other software technologies consisting of numerous programs that test functionality at a module level where each test program performs multiple tests sequentially, logs the results as text and returns a program exit indicating pass/fail. Test programs are typically grouped by functional area and run as a set of scripts. The tests that fail are clearly identified in the log and reading the test program code is simple because the tests are executed sequentially. Sometimes there are input/output files that are validated against a reference file. This simple model is used extensively for ANSI Standards compliance. Using this model unit/module level testing would be easy and application/simulation level tests could be run as scripts with diffs with configurable 'tolerances' with the same style of logging and pass/fail return method. <br>
<br>What would be needed: Test program and test script templates, conventions for test logging, individual program/script exit (pass/fail) , conventions for dataset naming and configurable tolerances for higher level application/simulation tests. The process also would need to be managed.<br>
</div>
<div> </div>
<div>6) Decisions about <i>how </i>to test functionality at the module level in industry is distributed among the developers that work to build initial or new features and to fix bugs. For <i>future releases</i> of GROMACS developers should be required to add regression test programs that will verify on a pass/fail basis all new functionality and priority 1 bug fixes. <br>
<br>Developers usually have the basic logic of such test programs in their own test
programs, simulations, files and criteria that they use to validate their own
development. Their test work is usually thrown away. It should not be too time consuming to use this code as the
basis of regression tests <i>in the future if developers understand</i>
that it is a requirement for code submission. Tests must be designed and the developers implementing new features
and fixing bugs should take primary responsibility for how to test
changes. <i>Unfortunately this takes time, </i><i>costs money, slows the development process and publication process</i>. <br></div>
<div> </div>
<div>7) It is important to try to get a second opinion about how something important is to be fixed or a new feature is to be added and have someone 'sign off' on another person's work and test program/script prior to incorporation into a release. It would be good to integrate this 'signoff' into the process.<br>
<br>Regards,<br>David<br></div>
<div> </div>
<div> </div><div> <br></div>
<div> </div>
<div class="gmail_quote">On Sat, Feb 11, 2012 at 7:35 AM, David van der Spoel <span dir="ltr"><<a href="mailto:spoel@xray.bmc.uu.se" target="_blank">spoel@xray.bmc.uu.se</a>></span> wrote:<br>
<blockquote style="border-left:1px solid rgb(204,204,204);margin:0px 0px 0px 0.8ex;padding-left:1ex" class="gmail_quote">
<div>On 2012-02-10 20:17, Roland Schulz wrote:<br></div>
<blockquote style="border-left:1px solid rgb(204,204,204);margin:0px 0px 0px 0.8ex;padding-left:1ex" class="gmail_quote">
<div><br><br>On Sun, Feb 5, 2012 at 3:53 PM, Shirts, Michael (mrs5pt)<br></div><<a href="mailto:mrs5pt@eservices.virginia.edu" target="_blank">mrs5pt@eservices.virginia.edu</a> <mailto:<a href="mailto:mrs5pt@eservices.virginia.edu" target="_blank">mrs5pt@eservices.<u></u>virginia.edu</a>>>
<div><br>wrote:<br><br> Hi, all-<br><br> My opinion:<br><br> I think there should probably be two classes of sets -- fast fully<br> automated<br> sets, and more sophisticated content validation sets.<br>
<br> For the fast fully automated test, I would suggest:<br> -testing a large range of input .mdps, tops and gros for whether<br> they run<br> through grompp and mdrun. Not performing whether the output is<br> correct or<br>
not, because that is very hard to automate -- just testing whether<br> it runs<br> for, say 20-100 steps or so.<br><br><br>Yes having that set of inputs is needed. Should we start a wiki page to<br>start listing all the inputs we want to include? Or what would be the<br>
best way to collaborative creating this set of inputs?<br></div></blockquote><br>A while ago I talked about this with Nicu who works with Siewert Jan Marrink. He is a software engineer by training and suggested the following. For each import parameter you take extreme values (e.g. timestep 5 fs and 0.001 fs) and a random value in between. Then there would be N^3 different parameter combinations for N parameters which probably is way too many combination, even if N would be only 20. Therefore you now pick a subset of, say 200 or 1000 out of these N^3 possible tests, and this becomes the test set. With such a set-up it is quite easy to see that we'd test at least the extreme value which are possible where things can go wrong. A few of these tests would actually be prohibited by grompp too, but in all likelihood not nearly enough.<br>
<br>At the time when Nicu & I discussed this we even considered publishing this, since I am not aware of another scientific code that has such rigorous testing tools.<br><br>
<blockquote style="border-left:1px solid rgb(204,204,204);margin:0px 0px 0px 0.8ex;padding-left:1ex" class="gmail_quote">
<div>
<div></div>
<div><br><br> Longer term, we should look more at validating code at a physical level.<br> Clearly testing energy conservation is a good idea for integrators; it's<br> fairly sensitive. I think we need to discuss a bit more about how to<br>
evaluate energy conservation. This actually can take a fair amount<br> of time,<br> and I'm thinking this is something that perhaps should wait for 5.0.<br> For<br> thermostats and barostats, I'm working on a very sensitive test of<br>
ensemble<br> validity. I'll email a copy to the list when it's ready to go (1-2<br> weeks?),<br> and this is something that can also be incorporated in an integrated<br> testing<br> regime, but again, these sorts of tests will take multiple hours, not<br>
seconds. That sort of testing level can't be part of the day to<br> day build.<br><br>Well even if the tests take ~2000CPU-hours, I think we (maybe even Erik<br>by himself) have the resources to run this weekly.<br>
<br><br><br> > - What are the requirements for the new test set? E.g. how easy<br> should it<br> > be to see whats wrong when a test fails?<br><br> For the first set of tests, I can imagine that it would be nice to<br>
be able<br> to look at the outputs of the tests, and diff different outputs<br> corresponding to different code versions to help track down changes<br> were.<br> But I'm suspicious about making the evaluation of these tests decided on<br>
automatically at first. I agree such differences should EVENTUALLY be<br> automated, but I'd prefer several months of investigation and discussion<br> before deciding exactly what "correct" is.<br><br>
<br>I think a wrong reference value is better than no reference value. Even<br>a wrong reference value would allow us to detect if e.g. different<br>compilers give significant different results (maybe some give the<br>correct value). Also it would help to avoid adding additional bugs. Of<br>
course we shouldn't release the test set to the outside before we are<br>relative sure that it actually correct.<br><br> > Should the test support being run<br> > under valgrind? Other?<br><br> Valgrind is incredibly slow and can fail for weird reasons -- I'm<br>
not sure<br> it would add much to do it under valgrind.<br><br>I have the current C++ tests (those written by Teemu) running under<br>valgrind in Jenkins. It wasn't very hard to write a few suppression<br>rules to make valgrind not report any false positives. Now Jenkins<br>
can automatically fail the build if the code has any memory errors.<br>Obviously one woudn't run any of the long running tests with valgrind.<br>But for the unit tests I think it might be very useful to catch bugs.<br>
<br> I DON'T think we should have any test set that starts to look at more<br> complicated features right now -- it will take months to get that<br> working,<br> and we need to get 4.6 out of the door on the order of weeks, so we<br>
can move<br> on to the next improvements. 4.6 doesn't have to be perfectly<br> flawless, as<br> long as it's closer to perfect than 4.5.<br><br><br>My reason for delaying the 4.6 release would not be to improve the 4.6<br>
release. I agree with you we probably can't guarantee that the reference<br>value are correct in time anyhow, so we probably wouldn't even want to<br>ship the tests with 4.6. My worry is that as soon as 4.6 is out the<br>
focus is on adding new cool features instead of working on these boring<br>tasks we should do, because they help us in the long run. E.g. if we<br>would have agreed that we don't have a 4.6 release, the C++ conversion<br>
would most likely be much further along. And I don't see how we can<br>create an incentive mechanism to work on these issues without somehow<br>coupling it to releases.<br><br>Roland<br><br><br> Best,<br> ~~~~~~~~~~~~<br>
Michael Shirts<br> Assistant Professor<br> Department of Chemical Engineering<br> University of Virginia<br></div></div> <a href="mailto:michael.shirts@virginia.edu" target="_blank">michael.shirts@virginia.edu</a> <mailto:<a href="mailto:michael.shirts@virginia.edu" target="_blank">michael.shirts@<u></u>virginia.edu</a>><br>
<a href="tel:%28434%29-243-1821" value="+14342431821" target="_blank">(434)-243-1821</a> <tel:%28434%29-243-1821><br><br><br><br> --<br> gmx-developers mailing list<br> <a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a> <mailto:<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@<u></u>gromacs.org</a>>
<div><br> <a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/<u></u>mailman/listinfo/gmx-<u></u>developers</a><br> Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@<u></u>gromacs.org</a><br></div> <mailto:<a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-<u></u>request@gromacs.org</a>>.<br>
<br><br><br><br><br><br><br>--<br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov/" target="_blank">cmb.ornl.gov</a> <<a href="http://cmb.ornl.gov/" target="_blank">http://cmb.ornl.gov</a>>
<div><br><a href="tel:865-241-1537" value="+18652411537" target="_blank">865-241-1537</a>, ORNL PO BOX 2008 MS6309<br><br><br></div></blockquote><font color="#888888"><br><br>-- <br>David van der Spoel, Ph.D., Professor of Biology<br>
Dept. of Cell & Molec. Biol., Uppsala University.<br>Box 596, 75124 Uppsala, Sweden. Phone: <a href="tel:%2B46184714205" value="+46184714205" target="_blank">+46184714205</a>.<br><a href="mailto:spoel@xray.bmc.uu.se" target="_blank">spoel@xray.bmc.uu.se</a> <a href="http://folding.bmc.uu.se/" target="_blank">http://folding.bmc.uu.se</a></font>
<div>
<div></div>
<div><br>-- <br>gmx-developers mailing list<br><a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a><br><a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/<u></u>mailman/listinfo/gmx-<u></u>developers</a><br>
Please don't post (un)subscribe requests to the list. Use the www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@<u></u>gromacs.org</a>.<br></div></div>
</blockquote></div><br>