<br><br><div class="gmail_quote">On Mon, Apr 16, 2012 at 2:51 PM, Erik Lindahl <span dir="ltr"><<a href="mailto:lindahl@cbr.su.se">lindahl@cbr.su.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Switching this discussion to gmx-developers - please participate if you feel you have something to say!<br>
<div class="im"><br>
<br>
On Apr 16, 2012, at 8:07 PM, Roland Schulz wrote:<br>
><br>
> Obvious external programs (e.g. VMD) won't bundle HDF5. But I doubt they have anything against supporting the HDF5 based formats on systems with HDF5 installed.<br>
<br>
</div>It is critical that the new formats work *everywhere*. This might sound excessive, but trust me - when we originally developed our current portable formats they required the XDR libraries that were available everywhere (read: as long as it was unix), and roughly 10 years ago that meant I had to spend several months writing a portable XDR implementation so we could get Gromacs working on Windows for Folding@Home. We're not going back to anything that isn't 100% portable :-)<br>
</blockquote><div><br></div><div>Obviously, by requiring that an external library works everywhere, we decide that no external library can ever be used (because it is impossible to test against all platforms/compilers) :-). Also I don't think it makes any sense to choose portability over every other important metric (maintainability, features, ...), no matter how small the portability difference and how huge the other advantages. Additionally. I think it should be up to the developers who want a certain portability to put in the effort to get it (as it is with other features).</div>
<div><br></div><div>I think we should have a list of supported platforms/compilers. As a developer it should be my duty to make sure that any code changes work on any of those (ideally Jenkins should do it for me) and I should be responsible to fix any bugs on any supported platform in my code. On the other hand, bugs on any non supported platform should be the responsibility of the person who wants it.</div>
<div><br></div><div>If we would require all developers to support all platforms, we would shift the responsibility to support esoteric platforms to the developers of new features, even if the majority has no interest in the platform. This is not how we handle new features and I don't see any reason why we should do that for portability.</div>
<div><br></div><div>I think we shouldn't require better portability to external code than we require for own own.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">
> Is it really easier to fix code some other Gromacs developer wrote? Given that the Gromacs code is often not well documented, IMO it is sometimes easier to fix code in external libraries than in Gromacs. Thus I think this is only an advantage as long as the Gromacs developers who originally wrote the code are still around. Also HDF5 is supported by the compute centers in the US. Thus we could ask one of the compute centers with help on HDF5 if we would required. Also HDF5 has commercial support.<br>
<br>
</div>I don't know about HDF5, but we have never had any significant problems in fixing portability issues in the GROMACS codebase. Again, HDF5 might be a model of portability, but before even thinking about that alternative the portability has to be tested!<br>
</blockquote><div><br></div><div>I agree it has to be tested. </div><div><br></div><div>But I think it is important to remember that maintaining portability in Gromacs has also problems and significant cost (=developer time). As an example, looking at Gerrit one sees quite a large number of changes which had to be reworked to fix MSVC issues. Also the large number of #ifdefs caused a significant number of bugs. And writing a new container format would add signification amount of low level (and thus non-trivial to port) code.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
> There is no question that HDF5 is much more versatile, but do we absolutely need that versatility?<br>
> No. But do we absolutely need NEC and Google nativeclient? I would wager that some of the nice to have but not required HDF5 features are more useful than NEC, NC, and Portland together.<br>
<br>
</div>Disagree. Nativeclient is likely to be Google's main vehicle for running things in the cloud, and NEC has on several instances had the fastest computers in the world (and there is rumor about an update to the Earth Simulator).</blockquote>
<div>Both is used by a very small part of the Gromacs community. Thus it shouldn't be more important than a feature used by that few people.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
And right now Portland is the only major compiler with support for GPU directives - it is an important compiler.<br></blockquote><div>GPU directives is a potential useful but currently not used feature. It should be treated as other comparable features.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">
> Certainly portability is important. But I don't think portability should be taken too far. We should try to decide solution based on the usefulness of the code to the Gromacs community (maybe with a slightly more importance to the needs of the core developers). Thus if hundreds of user would slightly benefit from non-essential tools provided by HDF5, while one user has to use an old/limited version (e.g. on native-client), I think the majority needs should be more important.<br>
<br>
</div>I think the first step is to be concrete about exactly what useful features we will get automatically with HDF5 that will be very difficult or impossible to implement in our own format?<br></blockquote><div>It is one option to do that research first. We have started it but are not finished. Another option is, we first agree on the requirements any potential external library has to fulfill. I think we currently don't have any guidelines for this.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Again, I like HDF5 as a potential more "general" optional format corresponding to our current TRR files. In this case it will be close to trivial to implement, and it will be very easy to slice data along any dimension directly into an analysis program.<br>
<br>
However, here we are primarily talking about the replacement to XTC, and with very fancy compression routines I think all those features disappear.</blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This format needs to be 100% portable </blockquote><div>Well even if all of TNG would only be available with HDF5, I don't see why we would need to have it 100% portable. The XTC compression is not orders of magnitudes worse.</div>
<div><br></div><blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
and easy to provide as a separate library for inclusion in other programs.</blockquote><div>My experience is that other projects are less reluctant to have external dependencies. E.g. VMD already has an optional dependency on NetCDF.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">And, let's focus on what we need rather than fun stuff that might come in handy once in a while :-)<br>
</blockquote><div>I think, this is my main point. I don't see how NEC and PGI doesn't fall into the "handy once in a while" category :-). </div><div><br></div><div>Roland</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im"><br>
> BTW: If we now discuss pro/cons of HDF5 and external libraries I suggest to move this discussion to gmx-developers. I moved the discussion back to private email because we were gathering information and I thought that information gathering is not of general interest. But the pro/cons of HDF5 (and external libraries in general) probably should be discussed on the list. Feel free to reply in public or let me know whether it is OK for me to forward to the list.<br>
<br>
</div>Good idea - taking it there!<br>
<br>
Cheers,<br>
<br>
Erik<br>
<br>
<br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309<br>