<br><br><div class="gmail_quote">On Wed, Aug 25, 2010 at 9:58 AM, Mark Abraham <span dir="ltr"><<a href="mailto:mark.abraham@anu.edu.au">mark.abraham@anu.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><br><br>----- Original Message -----<br>From: "Esztermann, Ansgar" <<a href="mailto:Ansgar.Esztermann@mpi-bpc.mpg.de" target="_blank">Ansgar.Esztermann@mpi-bpc.mpg.de</a>><br>Date: Wednesday, August 25, 2010 22:49<br>
Subject: [gmx-developers] Test Suite, CTest, CDash<br>To: Discussion list for GROMACS development <<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a>><br><br>> Hello everyone,<br>
> <br>> <br>> a few years ago, I have been briefly involved in the gcc <br>> project. I was extremely impressed by the huge test suite and by <br>> the way it could be used to quickly detect regressions. Putting <br>
> aside for the moment the fact that "correct behaviour" is much <br>> easier to define for a compiler than for a simulation engine, I <br>> think that gromacs would profit from a similar test suite.<br>
<br></div>Agreed. There are huge combinatorial and chaotic problems, though. I'd expect there must be some literature on dealing with the former.<br></blockquote><div>I don't think their is any magic solution for it. One will never have have enough tests to check for all possible combinations. </div>
<div>Suggestions I have read (or are my own opinion):</div><div>- When ever a bug is found add a test which would have found the bug. Easy since one usually already has a simple test tpr. And the experience is that it happens quite often that the same or a very similar bug reappears in the future.</div>
<div>- Have very simple tests. Not units tests (found find integration bugs) but tests which are simple (easy to understand) and fast to run. </div><div>- Have very many tests. I think it would be good to collect tpr a large number of tpr which has shown problems in the past and make sure the collection covers a reasonable number of combinations of input files, methods, parameters and parallelization.</div>
<div>- Test for things which are not (very) dependent on the chaotic behavior. Test e.g. for system not exploding, minimum energy conservation, simulation not crashing, performance or energy after first (few) steps.</div>
<div>- Make non-brittle tests. Thus make sure that the tests don't fail if some internal things change non-relevant to the test. Thus e.g. don't check the energies to the last digit. </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
So I'm going to sound a bit negative here, but I'm skeptical that it's possible to do a sound job of a MD regression suite. Who's got a computer science theoretician looking for a project? :-)</blockquote>
<div><br></div><div>I'm sure it is impossible to write a regression suite which would find all possible errors for any software ;-). I read that some software has more code for tests than for the software itself (and still are not bug free). And while to many (brittle) tests are a maintainability problem, I think it is possible to add some good tests to GROMACS which would have made it likely to have found bugs earlier and easier.</div>
<div><br></div><div>Specific suggestions:</div><div>- Discuss and develop guidelines how good tests should look like</div><div>- Collects TPRs which have shown bugs before or otherwise are useful (test specific feature - e.g. PULL, ...) and run fast</div>
<div>- Test the results for all those TPRs in a way which is non-brittle.</div><div><br></div><div>Roland </div><div> </div></div><br>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov">cmb.ornl.gov</a><br>
865-241-1537, ORNL PO BOX 2008 MS6309<br>