<br><br><div class="gmail_quote">On Wed, Oct 13, 2010 at 1:02 AM, Erik Lindahl <span dir="ltr"><<a href="mailto:lindahl@cbr.su.se" target="_blank">lindahl@cbr.su.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF"><div>Hi,</div><div><br></div><div>File flushing has been a huge issue to get working properly on AFS and other systems that have an extra layer of network disk cache. We also want to make sure the files are available e.g. on the frontend node of a cluster while the simulation is still running.</div>
</div></blockquote><div>Do we want to guarantee that it is available sooner than at each checkpoint (thus by default 15min)?</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF"><div>I think the proper solution is rather to have a separate IO thread so the disk operation can take all the latency in the world without delaying the run.</div></div></blockquote><div>This won't solve it for all cases. Depending on the write frequency (e.g. every 10 frames) the flushing time can take longer than computing the frames while the actually writing time (measured as the writing time with only infrequent flush) is fast enough to not cause significant overhead. In those situations the simulation would still wait on the IO thread. </div>
<div><br></div><div>Also this adds additional complexity. Not all systems like oversubscribing threads as far as I know. I know that older versions of Cray had problems and I heard their are also problems with BlueGene. Thus we would need to make the IO thread functionality optional which would add yet another duplication of code (both with and without IO thread).</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF"><div>You are more then welcome to play with it (but not in the release branch!) -</div>
</div></blockquote><div>No this is only going into the CollectiveIO branch and from their into the master branch. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF"><div> you might already have anaccount on the AFS-equipped clusters here, or we can arrange it!</div><div><span><br></span></div><div>
<span>Alternatively, sync with Sander and he might be able to test new code on AFS.</span></div></div></blockquote><div><br></div><div>Yes if I could get an account or Sander could test the CollectiveIO branch that would be great. I'll write Sander directly tomorrow.</div>
<div><br></div><div>Roland</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF"><font color="#888888">
</font><div><div></div><div><div><br></div><div><br>On Oct 12, 2010, at 23:26, Roland Schulz <<a href="mailto:roland@utk.edu" target="_blank">roland@utk.edu</a>> wrote:<br><br></div><div></div><blockquote type="cite">
<div>Erik, Berk,<div><br></div><div>you added flushing of trn, xtc and ern before the checkpointing functionality had been added. The additional flush can add quite a bit of unnecessary time especially with parallel file systems and/or MPI-IO. Am I right that with checkpointing it is not necessary anymore? The checkpointing is flushing the file before writing the checkpoint. Also gmx_fio_check_file_position is flushing the file before checking whether the file is too large for gmx_off_t</div>
<div>Roland<br clear="all"><br>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank"></a><a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309<br>
</div>
</div></blockquote></div></div></div></blockquote></div><br><br clear="all"><br>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309<br>