<br><br><div class="gmail_quote">On Thu, Sep 30, 2010 at 9:19 PM, Mark Abraham <span dir="ltr"><<a href="mailto:mark.abraham@anu.edu.au">mark.abraham@anu.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im"><br><br>----- Original Message -----<br>From: Roland Schulz <<a href="mailto:roland@utk.edu" target="_blank">roland@utk.edu</a>><br>Date: Friday, October 1, 2010 9:04<br>Subject: Re: [gmx-developers] Collective IO<br>
To: Discussion list for GROMACS development <<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a>><br><br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br></div><div class="gmail_quote"><div class="im"><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>On Thu, Sep 30, 2010 at 6:21 PM, Szilárd Páll <span dir="ltr"><<a>szilard.pall@cbr.su.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex"><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> Hi Roland,<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> Nice work, I'll definitely take a look at it!<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> Any idea on how does this improve scaling in general and at what<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> problem size starts to really matter? Does it introduce and overhead<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> in smaller simulations or it is only conditionally turned on?<br>
</blockquote><div><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br></div><div><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>At the moment it is always turned on for XTC when compiled with MPI. In serial or with threads nothing changes. At the moment we buffer at maximum 100 frames. If one uses less than 100 PP nodes than we buffer as many frames as the number of PP nodes. We also make sure that we don't buffer more than 250MB per node.</div>
<div><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br></div></div><div><div class="im"><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>The 100 frames and 250MB are both constants which should probably still be tuned.<br>
<br></div>Indeed - and the user should be able to tune them, too. They won't want to exceed their available physical memory, since buffering frames to virtual memory (if any) loses any gains from collective I/O.<br></div>
</div></blockquote><div>Honestly we hadn't thought much about the 250MB limit. We first wanted to get feedback on the approach and the code before doing more benchmarks and tuning these parameters. It is very likely that their are no cases which benefit from using more than 2MB per MPI process.</div>
<div><br></div><div>In case we limit the memory usage to 2MB should we still make it configurable? I think adding to many mdrun option gets confusing. Should we make the number of buffered frames a hidden mdrun option or an environment variable (the default would be that the number is auto-tuned)?</div>
<div><br></div><div>Roland</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="gmail_quote"><div><font color="#888888"><br>Mark<br><br></font></div>
<div><div></div><div class="h5"><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>If the IO time is negligible than of course you want see any speedup. The IO time shows mostly up as comm. energies (because only 1 node is in write_traj and the others are waiting at the next comm. energies). Usually the IO time because important if you either have more than 100 nodes and/or you write very frequently. <div>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br></div><div><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>It should not be slower in any case. The only case might be when one writes from many IO nodes to a non parallel file-system (e.g. NFS) that the parallel IO could be slower. We will test this tomorrow. Otherwise it should always be as fast or faster.</div>
<div><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br></div><div><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>Ryan & Roland</div>
<div><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div><div> <font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br> <font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> On Fri, Oct 1, 2010 at 12:11 AM, Roland Schulz <<a>roland@utk.edu</a>> wrote:<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > Hi,<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > we (Ryan&me) just uploaded our work on buffered MPI writing of XTC<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > trajectories. It can be found in the branch CollectiveIO.<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > We buffer a number of frames and use MPI IO to write those frames from a<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > number of nodes (see previous mails for details). The XTC trajectory is<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > written at least at every checkpoint guaranteeing that no frames are lost if<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > a simulation crashes.<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > We have tested it in serial, with PME, with threads, with multi and it seems<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > to work in all cases.<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > For 3 million atoms on 8192 writing every 1000 steps the performance is<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > increased from 21ns/day to 34ns/day and the time spent in comm.<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > energies decreases from 47% to 7%.<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > Feedback on the code change is very welcome. If you want to look at the<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > diff, I suggest to use:<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > git difftool afd66e48c4e608 #this is the origin/master from when we<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > uploaded the branch<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > Please let us know what you would like us to change before we merge this<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > into master.<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > Thanks<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > Ryan & Roland<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > --<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > 865-241-1537, ORNL PO BOX 2008 MS6309<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> ><br>
</div></div><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>> --<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > gmx-developers mailing list<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > <a>gmx-developers@gromacs.org</a><br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > <a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > Please don't post (un)subscribe requests to the list. Use the<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> > www interface or send it to <a>gmx-developers-request@gromacs.org</a>.<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> ><br> <font color="#888888"><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>--<br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> gmx-developers mailing list<br> <a><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank"><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> Please don't post (un)subscribe requests to the list. Use the<br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font> www interface or send it to <a>gmx-developers-request@gromacs.org</a>.<br>
</font></blockquote></div></div></div><div><div></div><div class="h5"><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br clear="all">
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font><br><font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>-- <br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>
<font style="font-style:normal;font-weight:normal;background-color:rgb(245, 248, 240);font-size:14px">> </font>865-241-1537, ORNL PO BOX 2008 MS6309<br>
</div></div> > -- <br><div><div></div><div class="h5">> gmx-developers mailing list<br>> <a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a><br>> <a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
> Please don't post (un)subscribe requests to the list. Use the <br>> www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.
</div></div><br>--<br>
gmx-developers mailing list<br>
<a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br></blockquote></div><br><br clear="all"><br>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov">cmb.ornl.gov</a><br>
865-241-1537, ORNL PO BOX 2008 MS6309<br>