<br><br><div class="gmail_quote">On Wed, Jul 7, 2010 at 1:14 AM, David van der Spoel <span dir="ltr"><<a href="mailto:spoel@xray.bmc.uu.se">spoel@xray.bmc.uu.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div class="im">On 7/7/10 1:57 AM, Roland Schulz wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
<br>
<br>
On Tue, Jul 6, 2010 at 7:18 PM, Shirts, Michael (mrs5pt)<br></div>
<<a href="mailto:mrs5pt@eservices.virginia.edu" target="_blank">mrs5pt@eservices.virginia.edu</a> <mailto:<a href="mailto:mrs5pt@eservices.virginia.edu" target="_blank">mrs5pt@eservices.virginia.edu</a>>><div>
<div></div><div class="h5"><br>
wrote:<br>
<br>
> BTW: Regarding parallel read of XTC for analysis tools. I suggest<br>
we add an<br>
> XTC meta-file to solve the problem of parallel read for XTC. To<br>
be able to<br>
> read frames in parallel we need to know the starting positions of<br>
the frame.<br>
> Using the bisect search for XTC in parallel will probably give poor<br>
> performance on most parallel IO systems (small random access IO<br>
pattern - is<br>
> what parallel IO systems don't like at all). Using TRR instead<br>
for parallel<br>
> analysis is also not such a good idea because even with parallel<br>
IO several<br>
> analysis will be IO bound and thus we could benefit from the XTC<br>
compression.<br>
> Thus an XTC file with a meta-file containing the starting<br>
positions should<br>
> give the best performance. A separate meta-file instead of adding the<br>
> positions to the header has the advantage that we don't change<br>
the current<br>
> format and thus don't break compatibility with 3rd party softare.<br>
Having a<br>
> separate meta-file has the disadvantage of the required<br>
bookkeeping to make<br>
> sure that the XTC file and the metafile are up-to date to each<br>
other, but I<br>
> think this shouldn't be to difficult to solve. And if a meta-file<br>
is missing<br>
> or not up-to date it is possible to generate it on the fly.<br>
<br>
I'm wondering if this is the sort of problem that eventually moving to<br>
something like netCDF might help solve. Clearly, it would be a<br>
difficult<br>
move, and would require interconversion utilities for backward<br>
compatibility.<br>
<br>
<br>
I looked into this. The compression of XTC is very good. And good<br>
compression is important if you want to have a good IO rate (of the<br>
uncompressed data). NetCDF3 doesn't support compressions (there are<br>
unsupported extensions). HDF5/NetCDF4 support compression but<br>
only parallel read of compressed data not parallel write of compressed<br>
data. Also the zlib compression would have a significantly lower<br>
compression ration than the XTC compression does.<br>
Thus none would do by itself all we would like to do. Of course one<br>
could do the XTC compression within a NetCDF/HDF5 container, but I don't<br>
see how this would help anyone. Without the full required support for<br>
compression the only other advantage I could see in moving to<br>
NetCDF/HDF5 is that is easier for others to program readers/writers (is<br>
already very easy since the library xdrfile has been released). And if<br>
we have our custom compression within NetCDF/HDF5 than reading those<br>
files wouldn't be any easier than reading/writing current XTC files.<br>
<br>
Without compression we could as well use TRR. Writing a parallel<br>
reader/writer for that is dead simple (since the position of each frame<br>
is known from the number of atoms).<br>
</div></div></blockquote>
<br>
<br>
A person here at UU (Daniel Spångberg) has developed a new trajectory library (TNG - trajectory next generation). We are about to submit a paper about it. Key advantages over xtc:<br>
<br>
- slightly better compression (slightly slower in the best form, but algorithm is tunable)<br>
- support for velocities<br>
- support for additional information (e.g. atom names) in one or more frames<br>
- random search supported without binary search<br>
- parallel compression<br></blockquote><div>parallel in the sense of multi-threaded? </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
- open source<br>
<br>
This will provide a very good basis for parallel trajectory I/O.<br></blockquote><div>This sounds great! </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
The main problem for parallel I/O is management of atom numbers in a domain decomposition setup. If atoms drift to another processor over time this will imply that bookkeeping has to deal with this, in particular when assembling the trajectories later for analysis. </blockquote>
<div><br></div><div> Yes. Without DD and TNG (or TRR) it is extremely easy. The question is how to do DD correct. This is what I tried to address. But in case it wasn't quite clear what I meant, I write some more detail. </div>
<div><br></div><div>There are two possible approaches I can think of for parallel I/O with DD (I describe here only how to write. Reading should be similar):</div><div>- If you want to minimize the communication and want to be able to write a single frame in parallel you can't write the atoms in their original order. If you would, the required sorting and small writes would kill the performance. As I understand you, writing in this non-sorted way would require the bookkeeping you mention. One needs to write both the atom data and their atom number to the trajectory (to be able to sort later) and then you need to do the sorting as post-processing step or the analysis tools have to be able to read a format where atoms are not sorted. </div>
<div>- Or you do the global sort of atoms as currently (evolving global communication) but in a way that you hide latencies and communication time. I think this is possible by writing less frequent and thus always >~50 frames at one time. This allows you to do the sorting as currently, and thus no post-pressing or bookkeeping in the analysis tools is required. A single frame is still written serially but you are able to write several frames in parallel.</div>
<div><br></div><div>I think the 2nd approach is much better. It should be easier to implement because it doesn't require any changes to analysis tools and the current writing functions only need few changes. Also the less frequent writes are anyhow needed for good performance because of I/O latency and because I/O systems the low bandwidth with small chunk size. And since the I/O bandwidth will always be much lower than the communication bandwidth (if optimized) the global communication required for the sort won't be a bottleneck.</div>
<div><br></div><div>The only thing the 2nd approach doesn't address is the memory requirement. If we want to be able to simulate systems larger than what fits into one node memory (may be interesting for Bluegene - for Cray it is > 200M atoms), then we need to do parallel IO within one frame. But the 2nd approach is easily extensible for that. If you sort the atoms anyhow (and do this partly in parallel) it is easy to send only the atoms within 1/N of a frame to one node and the next 1/N to the next node. If you already are writing only every 15min than the latency problems of small communication (for sorting) and small chunks (for writing) are not an issue (and they wouldn't be really that small anyhow if you system is bigger than your memory ;-)).</div>
<div><br></div><div>Roland </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
<br>
Roland<br>
<br>
<br>
Best,<br>
<br>
~~~~~~~~~~~~<br>
Michael Shirts<br>
Assistant Professor<br>
Department of Chemical Engineering<br>
University of Virginia<br></div>
<a href="mailto:michael.shirts@virginia.edu" target="_blank">michael.shirts@virginia.edu</a> <mailto:<a href="mailto:michael.shirts@virginia.edu" target="_blank">michael.shirts@virginia.edu</a>><div class="im">
<br>
(434)-243-1821<br>
<br>
<br>
--<br>
gmx-developers mailing list<br></div>
<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a> <mailto:<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a>><div class="im"><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a><br></div>
<mailto:<a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>>.<br>
<br>
<br>
<br>
<br>
--<br>
ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a> <<a href="http://cmb.ornl.gov" target="_blank">http://cmb.ornl.gov</a>><div class="im"><br>
865-241-1537, ORNL PO BOX 2008 MS6309<br>
<br>
</div></blockquote>
<br>
<br>
-- <br>
David.<br>
________________________________________________________________________<br>
David van der Spoel, PhD, Professor of Biology<br>
Dept. of Cell and Molecular Biology, Uppsala University.<br>
Husargatan 3, Box 596, 75124 Uppsala, Sweden<br>
phone: 46 18 471 4205 fax: 46 18 511 755<br>
<a href="mailto:spoel@xray.bmc.uu.se" target="_blank">spoel@xray.bmc.uu.se</a> <a href="mailto:spoel@gromacs.org" target="_blank">spoel@gromacs.org</a> <a href="http://folding.bmc.uu.se" target="_blank">http://folding.bmc.uu.se</a><br>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br><font color="#888888">
-- <br></font><div><div></div><div class="h5">
gmx-developers mailing list<br>
<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
Please don't post (un)subscribe requests to the list. Use the www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309<br>