Hi,<div><br></div><div>yes that helps a lot. One more question. What filesystem on hopper 2 are you using for this test (home, scratch or proj, to see if it is Lustre or GPFS) ? And are you running the test on the login node or on the compute node?</div>
<div><br></div><div>Thanks</div><div>Roland<br><br><div class="gmail_quote">On Wed, Jun 8, 2011 at 1:17 PM, Dimitar Pachov <span dir="ltr"><<a href="mailto:dpachov@brandeis.edu">dpachov@brandeis.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hello,<br><br><div class="gmail_quote"><div><div></div><div class="h5">On Wed, Jun 8, 2011 at 4:21 AM, Sander Pronk <span dir="ltr"><<a href="mailto:pronk@cbr.su.se" target="_blank">pronk@cbr.su.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Hi Dimitar,<div><br></div><div>Thanks for the bug report. Would you mind trying the test program I attached on the same file system that you get the truncated files on? </div><div><br></div>
<div>compile it with gcc testje.c -o testio</div></div></blockquote><div><br></div></div></div><div>Yes, but no problem:</div><div><br></div><div>====</div><div><font size="1">[dpachov@login-0-0 NEWTEST]$ ./testio <br>TEST PASSED: ftell gives: 46</font><br>
</div><div>====</div><div><br></div><div>As for the other questions:</div><div><br></div><div>HPC OS version:</div><div>====</div><div><font size="1">[dpachov@login-0-0 NEWTEST]$ uname -a<br>Linux login-0-0.local 2.6.18-194.17.1.el5xen #1 SMP Mon Sep 20 07:20:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux<br>
[dpachov@login-0-0 NEWTEST]$ cat /etc/redhat-release <br>Red Hat Enterprise Linux Server release 5.2 (Tikanga)</font><br></div><div>====</div><div><br></div><div>GROMACS 4.5.4 built:</div><div>====</div><div><font size="1">module purge<br>
module load INTEL/intel-12.0<br>module load OPENMPI/1.4.3_INTEL_12.0<br>module load FFTW/2.1.5-INTEL_12.0 # not needed<br><br>#####<br># GROMACS settings</font><br><font size="1"><br>export CC=mpicc<br>export F77=mpif77<br>
export CXX=mpic++<br>export FC=mpif90<br>export F90=mpif90<br><br>make distclean<br><br>echo "XXXXXXX building single prec XXXXXX"<br><br>./configure --prefix=/home/dpachov/mymodules/GROMACS/EXEC/4.5.4-INTEL_12.0/SINGLE \<br>
--enable-mpi \<br> --enable-shared \</font><br> <font size="1">--program-prefix="" --program-suffix="" \<br> --enable-float --disable-fortran \<br> --with-fft=mkl \<br> --with-external-blas \<br> --with-external-lapack \<br>
--with-gsl \<br> --without-x \<br> CFLAGS="-O3 -funroll-all-loops" \<br> FFLAGS="-O3 -funroll-all-loops" \<br> CPPFLAGS="-I${MPI_INCLUDE} -I${MKL_INCLUDE} " \<br> LDFLAGS="-L${MPI_LIB} -L${MKL_LIB} -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -liomp5 "<br>
<br>make -j 8 && make install</font><br></div><div>====</div><div><br></div><div>Just did the same test on Hopper 2:</div><div><a href="http://www.nersc.gov/users/computational-systems/hopper/" target="_blank">http://www.nersc.gov/users/computational-systems/hopper/</a></div>
<div><br></div><div>with their built GROMACS 4.5.3 (gromacs/4.5.3(default)), and the result was the same as reported earlier. You could do the test there as well, if you have access, and see what you would get.</div><div>
<br></div><div>Hope that helps a bit.</div><div><br></div><div>Thanks,</div><div>Dimitar</div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word"><div><br></div><font color="#888888"><div>Sander</div><div><br></div></font></div><br><div style="word-wrap:break-word"><div><br></div><div><br></div><div><br><div><div><div></div><div class="h5">
<div>On Jun 7, 2011, at 23:21 , Dimitar Pachov wrote:</div>
<br></div></div><blockquote type="cite"><div><div></div><div class="h5"><div>Hello,</div><div><br></div><div>Just a quick update after a few shorts tests we (my colleague and I) quickly did. First, using </div><div><br></div>
<div>"<i>You can emulate this yourself by calling "sleep 10s" before mdrun and see if that's long enough to solve the latency issue in your case.</i>"</div>
<div><br></div><div>doesn't work for a few reasons, mainly because it doesn't seem to be a latency issue, but also because the load on a node is not affected by "sleep".</div><div><br></div><div>However, you can reproduce the behavior I have observed pretty easily. It seems to be related to the values of the pointers to the *xtc, *trr, *edr, etc files written at the end of the checkpoint file after abrupt crashes AND to the frequency of access (opening) to those files. How to test:</div>
<div> </div><div>1. In your input *mdp file put a high frequency of saving coordinates to, say, the *xtc (10, for example) and a low frequency for the *trr file (10,000, for example).</div><div>2. Run GROMACS (mdrun -s run.tpr -v -cpi -deffnm run)</div>
<div>3. Kill abruptly the run shortly after that (say, after 10-100 steps).</div><div>4. You should have a few frames written in the *xtc file, and the only one (the first) in the *trr file. The *cpt file should have different from zero values for "file_offset_low" for all of these files (the pointers have been updated).</div>
<div><br></div><div>5. Restart GROMACS (mdrun -s run.tpr -v -cpi -deffnm run). </div><div>6. Kill abruptly the run shortly after that (say, after 10-100 steps). Pay attention that the frequency for accessing/writing the *trr has not been reached. </div>
<div>7. You should have a few additional frames written in the *xtc file, while the *trr will still have only 1 frame (the first). The *cpt file now has updated all pointer values "file_offset_low", BUT the pointer to the *trr has acquired a value of 0. Obviously, we already now what will happen if we restart again from this last *cpt file. </div>
<div><br></div><div>8. Restart GROMACS (mdrun -s run.tpr -v -cpi -deffnm run). </div><div>9. Kill it. </div><div>10. File *trr has size zero. </div><div><br></div><div><br></div><div>Therefore, if a run is killed before the files are accessed for writing (depending on the chosen frequency), the file offset values reported in the *cpt file doesn't seem to be accordingly updated, and hence a new restart inevitably leads to overwritten output files.</div>
<div> </div><div>Do you think this is fixable?</div><div><br></div><div>Thanks,</div><div>Dimitar</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div class="gmail_quote">On Sun, Jun 5, 2011 at 6:20 PM, Roland Schulz <span dir="ltr"><<a href="mailto:roland@utk.edu" target="_blank">roland@utk.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Two comments about the discussion: </div><div><br></div><div>1) I agree that buffered output (Kernel buffers - not application buffers) should not affect I/O. If it does it should be filed as bug to the OS. Maybe someone can write a short test application which tries to reproduce this idea. Thus writing to a file from one node and immediate after one test program is killed on one node writing to it from some other node.</div>
<div><br></div><div>2) We lock files but only the log file. The idea is that we only need to guarantee that the set of files is only accessed by one application. This seems safe but in case someone sees a way of how the trajectory is opened without the log file being opened, please file a bug.</div>
<div><br></div><div>Roland</div><br><div class="gmail_quote"><div><div>On Sun, Jun 5, 2011 at 10:13 AM, Mark Abraham <span dir="ltr"><<a href="mailto:Mark.Abraham@anu.edu.au" target="_blank">Mark.Abraham@anu.edu.au</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>
<div text="#000000" bgcolor="#ffffff"><div><div>
On 5/06/2011 11:08 PM, Francesco Oteri wrote:
<blockquote type="cite">
Dear Dimitar, <br>
I'm following the debate regarding:<br>
<span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><br>
<div>
<div><font size="1"><br>
</font></div>
</div>
</span>
<blockquote type="cite">
<div class="gmail_quote">
<div class="gmail_quote">The point was not "why" I was getting
the restarts, but the fact itself that I was getting
restarts close in time, as I stated in my first post. I
actually also don't know whether jobs are deleted or
suspended. I've thought that a job returned back to the
queue will basically start from the beginning when later
moved to an empty slot ... so don't understand the
difference from that perspective.<br>
</div>
</div>
</blockquote>
<br>
In the second mail yoo say:<br>
<br>
<span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">
<div>Submitted by:</div>
<div>========================</div>
<div><font size="1">ii=1</font></div>
<div><font size="1">ifmpi="mpirun -np $NSLOTS"</font></div>
<div><font size="1">--------</font></div>
<div><font size="1"> if [ ! -f run${ii}-i.tpr ];then</font></div>
<div>
<div><font size="1"> cp run${ii}.tpr run${ii}-i.tpr </font></div>
<div><font size="1"> tpbconv -s run${ii}-i.tpr -until
200000 -o run${ii}.tpr </font></div>
<div><font size="1"> fi</font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1"> k=`ls md-${ii}*.out | wc -l`</font></div>
<div><font size="1"> outfile="md-${ii}-$k.out"</font></div>
<div><font size="1"> if [[ -f run${ii}.cpt ]]; then</font></div>
<div><font size="1"> </font></div>
<div><font size="1"> <b> $ifmpi `which mdrun` </b>-s
run${ii}.tpr -cpi run${ii}.cpt -v -deffnm run${ii} -npme 0
> $outfile 2>&1 </font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1"> fi</font></div>
</div>
<div>=========================<br>
<br>
<br>
If I understand well, you are submitting the SERIAL mdrun.
This means that multiple instances of mdrun are running at the
same time.<br>
Each instance of mdrun is an INDIPENDENT instance. Therefore
checkpoint files, one for each instance (i.e. one for each
CPU), are written at the same time.</div>
</span> </blockquote>
<br></div></div>
Good thought, but Dimitar's stdout excerpts from early in the thread
do indicate the presence of multiple execution threads. Dynamic load
balancing gets turned on, and the DD is 4x2x1 for his 8 processors.
Conventionally, and by default in the installation process, the
MPI-enabled binaries get an "_mpi" suffix, but it isn't enforced -
or enforceable :-)<br>
<br>
Mark<br>
</div>
<br></div></div>--<div><br>
gmx-users mailing list <a href="mailto:gmx-users@gromacs.org" target="_blank">gmx-users@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-users" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-users</a><br>
Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/Search" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/Search</a> before posting!<br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-users-request@gromacs.org" target="_blank">gmx-users-request@gromacs.org</a>.<br>
Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br></div></blockquote></div><font color="#888888"><br><br clear="all"><br>-- <br>
ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov/" target="_blank">cmb.ornl.gov</a><br>
<a href="tel:865-241-1537" value="+18652411537" target="_blank">865-241-1537</a>, ORNL PO BOX 2008 MS6309<br>
</font><br>--<br>
gmx-users mailing list <a href="mailto:gmx-users@gromacs.org" target="_blank">gmx-users@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-users" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-users</a><br>
Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/Search" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/Search</a> before posting!<br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-users-request@gromacs.org" target="_blank">gmx-users-request@gromacs.org</a>.<br>
Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br></blockquote></div><br><br clear="all"><br>-- <br><span style="font-family:arial, sans-serif;border-collapse:collapse"><span style="font-size:x-small"><font color="#999999">=====================================================<br>
</font></span><b><span style="font-size:x-small"><font color="#999999">Dimitar V Pachov</font></span></b><span style="font-size:x-small"><font color="#999999"><br><br>PhD Physics<br>Postdoctoral Fellow </font></span></span><div>
<span style="font-family:arial, sans-serif;border-collapse:collapse"><span style="font-size:x-small"><font color="#999999">HHMI & Biochemistry Department Phone: <a href="tel:%28781%29%20736-2326" value="+17817362326" target="_blank">(781) 736-2326</a><br>
Brandeis University, MS 057 Email: </font></span><a href="mailto:dpachov@brandeis.edu" target="_blank"><span style="font-size:x-small"><font color="#999999">dpachov@brandeis.edu</font></span></a><span style="font-size:x-small"><font color="#999999"><br>
=====================================================</font></span></span></div><br></div></div>
-- <br><div class="im">gmx-users mailing list <a href="mailto:gmx-users@gromacs.org" target="_blank">gmx-users@gromacs.org</a><br><a href="http://lists.gromacs.org/mailman/listinfo/gmx-users" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-users</a><br>
Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/Search" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/Search</a> before posting!<br>Please don't post (un)subscribe requests to the list. Use the <br>
www interface or send it to <a href="mailto:gmx-users-request@gromacs.org" target="_blank">gmx-users-request@gromacs.org</a>.<br>Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a></div>
</blockquote>
</div><br></div></div><br>--<div class="im"><br>
gmx-users mailing list <a href="mailto:gmx-users@gromacs.org" target="_blank">gmx-users@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-users" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-users</a><br>
Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/Search" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/Search</a> before posting!<br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-users-request@gromacs.org" target="_blank">gmx-users-request@gromacs.org</a>.<br>
Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br></div></blockquote></div><div><div></div><div class="h5"><br><br clear="all">
<br>-- <br><span style="font-family:arial, sans-serif;border-collapse:collapse"><span style="font-size:x-small"><font color="#999999">=====================================================<br>
</font></span><b><span style="font-size:x-small"><font color="#999999">Dimitar V Pachov</font></span></b><span style="font-size:x-small"><font color="#999999"><br><br>PhD Physics<br>Postdoctoral Fellow </font></span></span><div>
<span style="font-family:arial, sans-serif;border-collapse:collapse"><span style="font-size:x-small"><font color="#999999">HHMI & Biochemistry Department Phone: <a href="tel:%28781%29%20736-2326" value="+17817362326" target="_blank">(781) 736-2326</a><br>
Brandeis University, MS 057 Email: </font></span><a href="mailto:dpachov@brandeis.edu" target="_blank"><span style="font-size:x-small"><font color="#999999">dpachov@brandeis.edu</font></span></a><span style="font-size:x-small"><font color="#999999"><br>
=====================================================</font></span></span></div><br>
</div></div><br>--<br>
gmx-users mailing list <a href="mailto:gmx-users@gromacs.org">gmx-users@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-users" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-users</a><br>
Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/Search" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/Search</a> before posting!<br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-users-request@gromacs.org">gmx-users-request@gromacs.org</a>.<br>
Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br></blockquote></div><br><br clear="all"><br>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov">cmb.ornl.gov</a><br>
865-241-1537, ORNL PO BOX 2008 MS6309<br>
</div>