<span style="border-collapse:collapse;font-family:arial, sans-serif;font-size:13px"><div style="color:rgb(80, 0, 80)"><span class="Apple-style-span" style="color: rgb(0, 0, 0); ">By the way, is this ever reviewed:</span></div>
<div style="color:rgb(80, 0, 80)"><span class="Apple-style-span" style="color: rgb(0, 0, 0); "><br></span></div><div style="color:rgb(80, 0, 80)"><span class="Apple-style-span" style="color: rgb(0, 0, 0); ">"Your mail to 'gmx-users' with the subject</span></div>
<div style="color:rgb(80, 0, 80)"><meta http-equiv="content-type" content="text/html; charset=utf-8"><span class="Apple-style-span" style="color: rgb(0, 0, 0); "><br> Re: [gmx-users] Why does the -append option exist?<br>
<br>Is being held until the list moderator can review it for approval.</span>"</div><div style="color:rgb(80, 0, 80)"><br></div><div style="color:rgb(80, 0, 80)"><br></div><div style="color:rgb(80, 0, 80)">
On Fri, Jun 3, 2011 at 9:24 PM, Mark Abraham <span dir="ltr"><<a href="mailto:Mark.Abraham@anu.edu.au" style="color:rgb(0, 0, 204)" target="_blank">Mark.Abraham@anu.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204, 204, 204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"><div>On 4/06/2011 8:26 AM, Dimitar Pachov wrote<br><br></div><div><br></div>If this is true, then it wants fixing, and fast, and will get it :-) However, it would be surprising for such a problem to exist and not have been reported up to now. This feature has been in the code for a year now, and while some minor issues have been fixed since the 4.5 release, it would surprise me greatly if your claim was true.<br>
<br>You're saying the equivalent of the steps below can occur:<br>1. Simulation wanders along normally and writes a checkpoint at step 1003<br>2. Random crash happens at step 1106<br>3. An -append restart from the old .tpr and the recent .cpt file will restart from step 1003<br>
4. Random crash happens at step 1059<br>5. Now a restart doesn't restart from step 1003, but some other step<div><br><br><blockquote type="cite"><div>and most importantly, the most important piece of data, that being the trajectory file, could be completely lost! I don't know the code behind the checkpointing & appending, but I can see how easy one can overwrite 100ns trajectories, for example, and "obtain" the same trajectories of size .... 0. <br>
</div></blockquote><br></div>I don't see how easy that is, without a concrete example, where user error is not possible.</div></blockquote><div><br></div></div><div>Here is an example:</div><div><br></div><div>========================</div>
<div><div><font size="1">[dpachov]$ ll -rth run1* \#run1*</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 11K May 2 02:59 run1.po.mdp</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 4.6K May 2 02:59 run1.grompp.out</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 3.5M May 13 19:09 run1.gro</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 2.3M May 14 00:40 run1.tpr</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 2.3M May 14 00:40 run1-i.tpr</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 0 May 29 21:53 run1.trr</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 1.2M May 31 10:45 run1.cpt</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 1.2M May 31 10:45 run1_prev.cpt</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 0 Jun 3 14:03 run1.xtc</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 0 Jun 3 14:03 run1.edr</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 15M Jun 3 17:03 run1.log</font></div>
</div><div>========================</div><div><br></div><div>Submitted by:</div><div>========================</div><div><font size="1">ii=1</font></div><div><font size="1">ifmpi="mpirun -np $NSLOTS"</font></div>
<div><font size="1">--------</font></div><div><font size="1"> if [ ! -f run${ii}-i.tpr ];then</font></div><div><div><font size="1"> cp run${ii}.tpr run${ii}-i.tpr </font></div><div><font size="1"> tpbconv -s run${ii}-i.tpr -until 200000 -o run${ii}.tpr </font></div>
<div><font size="1"> fi</font></div><div><font size="1"><br></font></div><div><font size="1"> k=`ls md-${ii}*.out | wc -l`</font></div><div><font size="1"> outfile="md-${ii}-$k.out"</font></div><div><font size="1"> if [[ -f run${ii}.cpt ]]; then</font></div>
<div><font size="1"> </font></div><div><font size="1"> $ifmpi `which mdrun` -s run${ii}.tpr -cpi run${ii}.cpt -v -deffnm run${ii} -npme 0 > $outfile 2>&1 </font></div><div><font size="1"><br></font></div>
<div><font size="1"> fi</font></div></div><div>=========================</div><div><br></div><div>From the end of run1.log:</div><div>=========================</div><div><div><font size="1">Started mdrun on node 0 Tue May 31 10:28:52 2011</font></div>
<div><font size="1"><br></font></div><div><font size="1"> Step Time Lambda</font></div><div><font size="1"> 51879390 103758.78000 0.00000</font></div><div><font size="1"><br></font></div>
<div><font size="1"> Energies (kJ/mol)</font></div><div><font size="1"> U-B Proper Dih. Improper Dih. CMAP Dih. LJ-14</font></div><div><font size="1"> 8.37521e+03 4.52303e+03 4.78633e+02 -1.23174e+03 2.87366e+03</font></div>
<div><font size="1"> Coulomb-14 LJ (SR) Disper. corr. Coulomb (SR) Coul. recip.</font></div><div><font size="1"> 3.02277e+04 9.48267e+04 -3.88596e+03 -7.43902e+05 -8.36436e+04</font></div><div>
<font size="1"> Potential Kinetic En. Total Energy Temperature Pres. DC (bar)</font></div><div><font size="1"> -6.91359e+05 1.29016e+05 -5.62342e+05 3.00159e+02 -1.24746e+02</font></div><div><font size="1"> Pressure (bar) Constr. rmsd</font></div>
<div><font size="1"> -2.43143e+00 0.00000e+00</font></div><div><font size="1"><br></font></div><div><font size="1">DD step 51879399 load imb.: force 225.5%</font></div><div><font size="1"><br></font></div><div><font size="1">Writing checkpoint, step 51879590 at Tue May 31 10:45:22 2011</font></div>
<div><font size="1"><br></font></div><div><font size="1"><br></font></div><div><font size="1"><br></font></div><div><font size="1"><br></font></div><div><font size="1">-----------------------------------------------------------</font></div>
<div><font size="1">Restarting from checkpoint, appending to previous log file.</font></div><div><font size="1"><br></font></div><div><font size="1">Log file opened on Fri Jun 3 17:03:20 2011</font></div><div><font size="1">Host: compute-1-13.local pid: 337 nodeid: 0 nnodes: 8</font></div>
<div><font size="1">The Gromacs distribution was built Tue Mar 22 09:26:37 EDT 2011 by</font></div><div><font size="1">dpachov@login-0-0.local (Linux 2.6.18-194.17.1.el5xen x86_64)</font></div></div><div><font size="1"><br>
</font></div><div><font size="1">:::</font></div><div><font size="1">:::</font></div><div><font size="1">:::</font></div><div><font size="1"><br></font></div><div><div><font size="1">Grid: 13 x 15 x 11 cells</font></div>
<div>
<font size="1">Initial temperature: 301.137 K</font></div><div><font size="1"><br></font></div><div><font size="1">Started mdrun on node 0 Fri Jun 3 13:58:07 2011</font></div><div><font size="1"><br></font></div><div><font size="1"> Step Time Lambda</font></div>
<div><font size="1"> 51879590 103759.18000 0.00000</font></div><div><font size="1"><br></font></div><div><font size="1"> Energies (kJ/mol)</font></div><div><font size="1"> U-B Proper Dih. Improper Dih. CMAP Dih. LJ-14</font></div>
<div><font size="1"> 8.47435e+03 4.61654e+03 3.99388e+02 -1.16765e+03 2.93920e+03</font></div><div><font size="1"> Coulomb-14 LJ (SR) Disper. corr. Coulomb (SR) Coul. recip.</font></div><div>
<font size="1"> 2.99294e+04 9.42035e+04 -3.87927e+03 -7.43250e+05 -8.35872e+04</font></div><div><font size="1"> Potential Kinetic En. Total Energy Temperature Pres. DC (bar)</font></div><div><font size="1"> -6.91322e+05 1.29433e+05 -5.61889e+05 3.01128e+02 -1.24317e+02</font></div>
<div><font size="1"> Pressure (bar) Constr. rmsd</font></div><div><font size="1"> -2.18259e+00 0.00000e+00</font></div><div><font size="1"><br></font></div><div><font size="1">DD step 51879599 load imb.: force 43.7%</font></div>
<div><font size="1"><br></font></div><div><font size="1">At step 51879600 the performance loss due to force load imbalance is 17.5 %</font></div><div><font size="1"><br></font></div><div><font size="1">NOTE: Turning on dynamic load balancing</font></div>
<div><font size="1"><br></font></div><div><font size="1">DD step 51879999 vol min/aver 0.643 load imb.: force 0.4%</font></div></div><div><font size="1"><br></font></div><div><font size="1">::</font></div><div><font size="1">::</font></div>
<div><font size="1">::</font></div><div><font size="1"><br></font></div><div><div><font size="1">DD step 51884999 vol min/aver 0.647 load imb.: force 0.3%</font></div><div><font size="1"><br></font></div><div><font size="1"> Step Time Lambda</font></div>
<div><font size="1"> 51885000 103770.00000 0.00000</font></div><div><font size="1"><br></font></div><div><font size="1"> Energies (kJ/mol)</font></div><div><font size="1"> U-B Proper Dih. Improper Dih. CMAP Dih. LJ-14</font></div>
<div><font size="1"> 8.33208e+03 4.72300e+03 5.31983e+02 -1.21532e+03 2.89586e+03</font></div><div><font size="1"> Coulomb-14 LJ (SR) Disper. corr. Coulomb (SR) Coul. recip.</font></div><div>
<font size="1"> 3.00900e+04 9.31785e+04 -3.87790e+03 -7.40841e+05 -8.36838e+04</font></div><div><font size="1"> Potential Kinetic En. Total Energy Temperature Pres. DC (bar)</font></div><div><font size="1"> -6.89867e+05 1.28721e+05 -5.61146e+05 2.99472e+02 -1.24229e+02</font></div>
<div><font size="1"> Pressure (bar) Constr. rmsd</font></div><div><font size="1"> -1.03491e+02 2.99840e-05</font></div></div><div>====================================</div><div><br></div><div>Last output files from restarts:</div>
<div>====================================</div><div><div><font size="1">[dpachov]$ ll -rth md-1-*out | tail -10</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.1K Jun 3 16:40 md-1-2428.out</font></div><div>
<font size="1">-rw-rw-r-- 1 dpachov dpachov 6.2K Jun 3 16:44 md-1-2429.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 5.9K Jun 3 16:46 md-1-2430.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 5.9K Jun 3 16:48 md-1-2431.out</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.1K Jun 3 16:50 md-1-2432.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 0 Jun 3 16:52 md-1-2433.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.2K Jun 3 16:55 md-1-2434.out</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.2K Jun 3 16:58 md-1-2435.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 5.9K Jun 3 17:03 md-1-2436.out</font></div><div><font size="1"><b>-rw-rw-r-- 1 dpachov dpachov 5.8K Jun 3 17:04 md-1-2437.out</b></font></div>
</div><div>====================================</div><div>+ around the time when the run1.xtc file seems to have been saved:</div><div>====================================</div><div><div><font size="1">[dpachov]$ ll -rth md-1-23[5-6][0-9]*out</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.2K Jun 3 13:37 md-1-2350.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.1K Jun 3 13:39 md-1-2351.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.2K Jun 3 13:43 md-1-2352.out</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.2K Jun 3 13:45 md-1-2353.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 5.9K Jun 3 13:46 md-1-2354.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 0 Jun 3 13:47 md-1-2355.out</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.1K Jun 3 13:49 md-1-2356.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.1K Jun 3 13:52 md-1-2357.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 12K Jun 3 13:57 md-1-2358.out</font></div>
<div><font size="1"><b>-rw-rw-r-- 1 dpachov dpachov 12K Jun 3 14:02 md-1-2359.out</b></font></div><div><font size="1"><b>-rw-rw-r-- 1 dpachov dpachov 6.0K Jun 3 14:03 md-1-2360.out</b></font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.2K Jun 3 14:06 md-1-2361.out</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 5.8K Jun 3 14:09 md-1-2362.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 5.9K Jun 3 14:10 md-1-2363.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.1K Jun 3 14:11 md-1-2364.out</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 5.8K Jun 3 14:12 md-1-2365.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.1K Jun 3 14:13 md-1-2366.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.1K Jun 3 14:14 md-1-2367.out</font></div>
<div><font size="1">-rw-rw-r-- 1 dpachov dpachov 6.0K Jun 3 14:17 md-1-2368.out</font></div><div><font size="1">-rw-rw-r-- 1 dpachov dpachov 5.9K Jun 3 14:18 md-1-2369.out</font></div></div><div>====================================</div>
<div><br></div><div>From md-1-2359.out:</div><div>=====================================</div><div><div><font size="1">:::::::</font></div><div><font size="1">Getting Loaded...</font></div><div><font size="1">Reading file run1.tpr, VERSION 4.5.4 (single precision)</font></div>
<div><font size="1"><br></font></div><div><font size="1">Reading checkpoint file run1.cpt generated: Tue May 31 10:45:22 2011</font></div><div><font size="1"><br></font></div><div><font size="1"><br></font></div><div><font size="1">Loaded with Money</font></div>
<div><font size="1"><br></font></div><div><font size="1">Making 2D domain decomposition 4 x 2 x 1</font></div><div><font size="1"><br></font></div><div><font size="1">WARNING: This run will generate roughly 4915 Mb of data</font></div>
<div><font size="1"><br></font></div><div><font size="1">starting mdrun 'run1'</font></div><div><font size="1">100000000 steps, 200000.0 ps (continuing from step 51879590, 103759.2 ps).</font></div><div><font size="1">step 51879590, will finish Wed Aug 17 14:21:59 2011</font></div>
<div><font size="1">imb F 44% </font></div><div><font size="1">NOTE: Turning on dynamic load balancing</font></div><div><font size="1"><br></font></div><div><font size="1">step 51879600, will finish Fri Jul 15 14:00:00 2011</font></div>
<div><font size="1">vol 0.64 imb F 0% step 51879700, will finish Mon Jun 27 02:19:09 2011</font></div><div><font size="1">vol 0.63 imb F 0% step 51879800, will finish Sat Jun 25 15:14:01 2011</font></div><div><font size="1">vol 0.64 imb F 1% step 51879900, will finish Sat Jun 25 02:11:53 2011</font></div>
<div><font size="1">vol 0.64 imb F 0% step 51880000, will finish Fri Jun 24 19:48:54 2011</font></div><div><font size="1">vol 0.64 imb F 1% step 51880100, will finish Fri Jun 24 15:55:19 2011</font></div></div><div><font size="1">::::::</font></div>
<div><div><font size="1">vol 0.67 imb F 0% step 51886400, will finish Fri Jun 24 02:51:45 2011</font></div><div><font size="1">vol 0.66 imb F 0% step 51886500, will finish Fri Jun 24 02:48:10 2011</font></div><div><font size="1">vol 0.66 imb F 0% step 51886600, will finish Fri Jun 24 02:47:33 2011</font></div>
</div><div>=====================================</div><div><br></div><div><div>From md-1-2360.out:</div><div>=====================================</div><div><div><font size="1">:::::::</font></div></div></div><div><div><font size="1">Getting Loaded...</font></div>
<div><font size="1">Reading file run1.tpr, VERSION 4.5.4 (single precision)</font></div><div><font size="1"><br></font></div><div><font size="1">Reading checkpoint file run1.cpt generated: Tue May 31 10:45:22 2011</font></div>
<div><font size="1"><br></font></div><div><font size="1"><br></font></div><div><font size="1">Loaded with Money</font></div><div><font size="1"><br></font></div><div><font size="1">Making 2D domain decomposition 4 x 2 x 1</font></div>
<div><font size="1"><br></font></div><div><font size="1">WARNING: This run will generate roughly 4915 Mb of data</font></div><div><font size="1"><br></font></div><div><font size="1">starting mdrun 'run1'</font></div>
<div><font size="1">100000000 steps, 200000.0 ps (continuing from step 51879590, 103759.2 ps).</font></div></div><div>=====================================</div><div><br></div><div>And from the last generated output md-1-2437.out (I think I killed the job at that point because of the above observed behavior):</div>
<div>=====================================</div><div><div><font size="1">:::::::</font></div></div><div><font size="1"><div>Getting Loaded...</div><div>Reading file run1.tpr, VERSION 4.5.4 (single precision)</div></font></div>
<div><font size="1"><span style="font-size:small">=====================================</span></font></div><div><br></div><div>I have at least 5-6 additional examples like this one. In some of them the *xtc file does have size greater than zero yet still very small, but it starts from some random frame (for example, in one of the cases it contains frames from ~91000ps to ~104000ps, but all frames before 91000ps are missing).</div>
<div><br></div><div>I realize there might be another problem, but the bottom line is that there is no mechanism that can prevent this from happening if many restarts are required, and particularly if the timing between these restarts is prone to be small (distributed computing could easily satisfy this condition).</div>
<div><br></div><div>Any suggestions, particularly related to the minimum resistance path to regenerate the missing data? :)</div><div style="color:rgb(80, 0, 80)"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204, 204, 204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"><div><br> <blockquote type="cite"><div>Using the checkpoint capability & appending make sense when many restarts are expected, but unfortunately it is exactly then when these options completely fail! As a new user of Gromacs, I must say I am disappointed, and would like to obtain an explanation of why the usage of these options is clearly stated to be safe when it is not, and why the append option is the default, and why at least a single warning has not been posted anywhere in the docs & manuals?</div>
</blockquote><br></div>I can understand and sympathize with your frustration if you've experienced the loss of a simulation. Do be careful when suggesting that others' actions are blame-worthy, however.</div></blockquote>
<div><br></div></div><div>I have never suggested this. As a user, I am entitled to ask. And since my questions were not clearly answered, I will repeat them in a structured way:</div><div><br></div><div>1. Why is the usage of these options (-cpi and -append) clearly stated to be safe when in fact it is not?</div>
<div>2. Why have you made the -append option the default in the most current GMX versions?</div><div>3. Why has not a single warning been posted anywhere in the docs & manuals? (this question is somewhat clear - because you did not know about such a problem, but people say "<span style="font-family:sans-serif;font-size:13px;line-height:20px">ignorance of the law excuses no one</span>", which means ignoring to put a warning for something that you were not 100% certain it would be error-free could not be an excuse)</div>
<div><br></div><div>I am blame-worthy - for blindly believing what was written in the manual without taking the necessary precautions. Lesson learned. </div><div style="color:rgb(80, 0, 80)"><div> </div></div>
<div style="color:rgb(80, 0, 80)"><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204, 204, 204);border-left-style:solid;padding-left:1ex">
<div text="#000000" bgcolor="#ffffff">However, developers' time rarely permits addressing "feature X doesn't work, why not?" in a productive way. Solving bugs can be hard, but will be easier (and solved faster!) if the user who thinks a problem exists follows good procedure. See <a href="http://www.chiark.greenend.org.uk/~sgtatham/bugs.html" style="color:rgb(0, 0, 204)" target="_blank">http://www.chiark.greenend.org.uk/~sgtatham/bugs.html</a><br>
<br></div></blockquote><div><br></div></div><div>Implying that I did not follow a certain procedure related to a certain problem without you knowing what my initial intention was is just a speculation. </div><div><br></div>
<div>At any instant, I do appreciate the time everybody unselfishly devotes to communicating with people experiencing problems. </div><div><br></div><div>Thanks,</div><div>Dimitar</div></span>