<br><br><div class="gmail_quote">On Sun, Jun 5, 2011 at 2:14 AM, Mark Abraham <span dir="ltr"><<a href="mailto:Mark.Abraham@anu.edu.au">Mark.Abraham@anu.edu.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div text="#000000" bgcolor="#ffffff"><div class="im">
On 5/06/2011 12:31 PM, Dimitar Pachov wrote:
<blockquote type="cite">As I said, the queue is like this: you submit the job, it
finds an empty node, it goes there, however seconds later
another user with higher privileges on that particular node
submits a job, his job kicks out my job, mine goes on the
queue again, it finds another empty node, goes there, then
another user with high privileges on that node submits a job,
which consequently kicks out my job again, and the cycle
repeats itself ... theoretically, it could continue forever,
depending on how many and where the empty nodes are, if any.</blockquote></div><div class="im">
<br></div>
You've said that *now* - but previously you've said nothing about
why you were getting lots of restarts. In my experience, PBS queues
suspend jobs rather than deleting them, in order that resources are
not wasted. Apparently other places do things this way. I think that
this information is highly relevant to explaining your observations.<div class="im"><br></div></div></blockquote><div><br></div><div><br></div><div>The point was not "why" I was getting the restarts, but the fact itself that I was getting restarts close in time, as I stated in my first post. I actually also don't know whether jobs are deleted or suspended. I've thought that a job returned back to the queue will basically start from the beginning when later moved to an empty slot ... so don't understand the difference from that perspective.</div>
<div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div text="#000000" bgcolor="#ffffff"><div class="im">
<br>
<blockquote type="cite">
<div class="gmail_quote">
<div>These many restarts suggest that the queue was full with
relatively short jobs ran by users with high privileges.
Technically, I cannot see why the same processes should be
running simultaneously because at any instant my job runs only
on one node, or it stays in the queuing list. <br>
</div>
</div>
</blockquote>
<br></div>
I/O can be buffered such that the termination of the process and the
completion of its I/O are asynchronous. Perhaps it *shouldn't* be
that way, but this is a problem for the administrators of your
cluster to address. They know how the file system works. If the next
job executes before the old one has finished output, then I think
the symptoms you observe might be possible.<br></div></blockquote><div><br></div><div>Yes, this is true, and I believe the timing of when the buffer is fully flushed is crucial in providing a possible explanation in the observed behavior. However, this bottleneck has been known for a long time, so I expected people had thought about that before confidently putting -append as a default. That's all.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div text="#000000" bgcolor="#ffffff">
<br>
Note that there is nothing GROMACS can do about that, unless somehow
GROMACS can apply a lock in the first mdrun that is respected by
your file system such that a subsequent mdrun cannot open the same
file until all pending I/O has completed. I'd expect proper HPC file
systems do that automatically, but I don't really know.<div><div></div><div class="h5"><br></div></div></div></blockquote><div><br></div><div>I am not an expert nor do I know the Gromacs coding, but could one have an option to specify certain timing before which Gromacs is prohibited to output/write any files after its initial start, i.e. some kind of suspension and/or waiting period? </div>
<div><br></div><div>I am also wondering about the checkpoint timing - the default is 15 min, but what would be the minimum? Since I have not tested it, what would happen if I specify 0.001 min, for example?</div><div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div text="#000000" bgcolor="#ffffff"><div><div class="h5">
<br>
<blockquote type="cite">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff">
<div>
<div>
<blockquote type="cite">
<div><br>
</div>
<div style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">
<div>From md-1-2360.out:</div>
<div>=====================================</div>
<div>
<div><font size="1">:::::::</font></div>
</div>
</div>
<div style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">
<div><font size="1">Getting Loaded...</font></div>
<div><font size="1">Reading file run1.tpr, VERSION
4.5.4 (single precision)</font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1">Reading checkpoint file run1.cpt
generated: Tue May 31 10:45:22 2011</font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1">Loaded with Money</font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1">Making 2D domain decomposition 4
x 2 x 1</font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1">WARNING: This run will generate
roughly 4915 Mb of data</font></div>
<div><font size="1"><br>
</font></div>
<div><font size="1">starting mdrun 'run1'</font></div>
<div><font size="1">100000000 steps, 200000.0 ps
(continuing from step 51879590, 103759.2 ps).</font></div>
</div>
<div style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">=====================================</div>
</blockquote>
<br>
</div>
</div>
These aren't showing anything other than that the restart is
coming from the same point each time.
<div><br>
<br>
<blockquote type="cite"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">
<div>And from the last generated output md-1-2437.out
(I think I killed the job at that point because of
the above observed behavior):</div>
<div>=====================================</div>
<div>
<div><font size="1">:::::::</font></div>
</div>
<div><font size="1">
<div>Getting Loaded...</div>
<div>Reading file run1.tpr, VERSION 4.5.4 (single
precision)</div>
</font></div>
<div><font size="1"><span style="font-size:small">=====================================</span></font></div>
<div><br>
</div>
<div>I have at least 5-6 additional examples like this
one. In some of them the *xtc file does have size
greater than zero yet still very small, but it
starts from some random frame (for example, in one
of the cases it contains frames from ~91000ps to
~104000ps, but all frames before 91000ps are
missing).</div>
</span></blockquote>
<br>
</div>
I think that demonstrating a problem requires that the set
of output files were fine before one particular restart, and
weird afterwards. I don't think we've seen that yet.
<div><br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I don't understand your point here. I am providing you with
all info I have. I am showing the output files of 3 restarts,
and they are different in a sense that the last two did not
progress further enough before another job restart occurred.
The first was fine before the restart, and the others were not
exactly fine after the restart. At this point I realize that
what I call "restart" and what you call "restart" might be two
different things. And here is where the problem might
be lying. </div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff">
<div> <br>
<blockquote type="cite"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">
<div>I realize there might be another problem, but the
bottom line is that there is no mechanism that can
prevent this from happening if many restarts are
required, and particularly if the timing between
these restarts is prone to be small (distributed
computing could easily satisfy this condition).</div>
<div><br>
</div>
<div>Any suggestions, particularly related to the
minimum resistance path to regenerate the missing
data? :)</div>
<div style="color:rgb(80, 0, 80)">
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff">
<div><br>
<blockquote type="cite">
<div>Using the checkpoint capability &
appending make sense when many restarts
are expected, but unfortunately it is
exactly then when these options completely
fail! As a new user of Gromacs, I must say
I am disappointed, and would like to
obtain an explanation of why the usage of
these options is clearly stated to be safe
when it is not, and why the append option
is the default, and why at least a single
warning has not been posted anywhere in
the docs & manuals?</div>
</blockquote>
<br>
</div>
I can understand and sympathize with your
frustration if you've experienced the loss of a
simulation. Do be careful when suggesting that
others' actions are blame-worthy, however.</div>
</blockquote>
<div><br>
</div>
</div>
<div>I have never suggested this. As a user, I am
entitled to ask. </div>
</span></blockquote>
<br>
</div>
Sure. However, talking about something that can "completely
fail"</div>
</blockquote>
<div><br>
</div>
<div>This is a fact, backed up by my evidences => I don't see
anything bad directed to anybody. </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> which makes you
"disappointed"</div>
</blockquote>
<div><br>
</div>
<div>This is me being honest => again not related to anybody
else. </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> and wanting to "obtain
an explanation"</div>
</blockquote>
<div><br>
</div>
<div>Well, this even is funny :) - many people want this,
especially in science. Is that bad?</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> about why something
doesn't work as stated and lacks "a single warning"</div>
</blockquote>
<div><br>
</div>
<div>Again a fact => again nothing bad here.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff"> suggests that someone
has done something less than appropriate</div>
</blockquote>
<div><br>
</div>
<div>This is a completely personal interpretation, and I am
personally not responsible of how people perceive information.
For unknown to me reason you moved into a very defensive mode.
What could I do? </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff">, and so blame-worthy.
It also assumes that the actions of a new user were correct,
and the actions of a developer with long experience were
not. </div>
</blockquote>
<div><br>
</div>
<div>Sorry, this is too much. Where was this suggested? It seems
to me you took it too personally. </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div text="#000000" bgcolor="#ffffff">This may or may not
prove to be true. Starting such a discussion from a
conciliatory (rather than antagonistic) stance is usually
more productive. The shared objective should be to fix the
problem, not prove that someone did something wrong.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Agree, and I did it. Again, your perception does not seem
to be correlated with my intended approach. <br>
</div>
</div>
</blockquote>
<br></div></div>
Words are open to interpretation. Communicating well requires that
you consider the impact of your words on your reader. You want
people who can address the problem to want to help. You don't want
them to feel defensive about the situation - whether you think that
would be an over-reaction or not.<div class="im"><br></div></div></blockquote><div><br></div><div>I got your point(s). However, I respectfully disagree with some of them. First, I believe it is much more important what information one's sentences bring rather than how specifically they are written. People are different, and hence, one cannot be like a chameleon constantly changing their ways of expressing themselves in order to adapt everybody's perceptions at any time for the sake of always satisfying the other side. Such a constant consideration of others is impossible, and definitely doesn't distinguish you as an individual as well as doesn't make you feel like a free person. Second, the impact of somebody's words is almost totally dependent on the listener/reader, and much less dependent on the person who communicates them. For example, you cannot persuade anybody in anything if that anybody doesn't decide for themselves to accept your words. How good you are in persuasion & communication is irrelevant. Third, the word "tone" generally refers to verbal communication and it is a very subjective call, the least to say, to judge "the tone" of written communication. Forth, cultural & expression differences may arise when people communicate on non-native languages, and this cannot be just ignored. </div>
<div><br></div><div>Therefore, I acknowledge the diversity of ways of expressing ourselves, and try not to experience negative feelings while reading plain written words on technical matters; otherwise, I would become unnecessary biased and prejudiced without having any real objective reason in being such. </div>
<div><br></div><div>Thanks,</div><div>Dimitar</div><div><br></div><div><br></div></div><br><br>