<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">@Eric, I can confirm that your scripts
will try to use all available cores for their jobs, because I
don't see anywhere a restriction for the number of OMP threads in
the tests with mdrun.<br>
<br>
I will disable the gmxapi testing if this is not fixed, because it
is choking the rest of the infrastructure.<br>
<br>
/Paul<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 29/09/2020 11:29, Erik Lindahl
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAEJJM8H1Ejyby4uo+qLadPPfn4TtfuCFLhBLTcLQPhh-iSwy2w@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">This could indeed be important for jobs in general
if they auto-detect the hardware.
<div><br>
</div>
<div>The way containerization work, you will always see that
there's e.g. 96 hardware threads on the server, and it is
quite possible to start 96 threads (or more) - but the cgroups
mechanism in the Linux kernel will limit the actual CPU usage
to the limits set (e.g. 4, with our max allowed value being 8
or 16, I think). That could then lead to a job trying to run
96 threads, but only having resources equivalent to two
physical cores (4 threads).</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Erik</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Sep 29, 2020 at 11:09
AM Paul bauer <<a href="mailto:paul.bauer.q@gmail.com"
moz-do-not-send="true">paul.bauer.q@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div>
<div>Eric, one question, how do you set the total number of
ranks for your jobs in the python script?<br>
<br>
I think the issue might be that you are oversubscribing
because you are trying to use the whole kubernetes
cluster, slowing everything down.</div>
<div><br>
</div>
<div>Cheers<br>
<br>
Paul<br>
<br>
P.s.: <a
href="https://gitlab.com/gromacs/gromacs/-/merge_requests/594"
target="_blank" moz-do-not-send="true">https://gitlab.com/gromacs/gromacs/-/merge_requests/594</a>
Is ready now and works as intended.<br>
</div>
<div><br>
</div>
<div>On 28/09/2020 21:15, Erik Lindahl wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">PS:<br>
<div><br>
</div>
<div>Note that there are two settings: The *request* you
set for CPU and memory is guaranteed when you execute
(although we count each hardware thread as a CPU),
while the limit is something you ask for, but aren't
guaranteed.</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Erik</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Sep 28, 2020
at 9:14 PM Erik Lindahl <<a
href="mailto:erik.lindahl@gmail.com" target="_blank"
moz-do-not-send="true">erik.lindahl@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>Given that several other test containers finish
in ~2 minutes, I think it's relatively unlikely
that random fluctuating resource contention would
systematically always affect the gmxapi test every
time it is run, but no other jobs :-)</div>
<div><br>
</div>
<div>The whole point of k8s/docker is that the
runtime environment is standardized; what
performance do you see if you run the same
container e.g. on your laptop, desktop, or any
other cloud resource when assigning two hardware
threads to it?</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Erik</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Sep 28,
2020 at 8:38 PM Eric Irrgang <<a
href="mailto:ericirrgang@gmail.com"
target="_blank" moz-do-not-send="true">ericirrgang@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
> On Sep 28, 2020, at 6:00 PM, Erik Lindahl
<<a href="mailto:erik.lindahl@gmail.com"
target="_blank" moz-do-not-send="true">erik.lindahl@gmail.com</a>>
wrote:<br>
> <br>
> - gmx-api. They both take 12-15 minutes on
two cores, and there are four of them. <br>
<br>
I think there is something wrong with the way
resources are detected in the CI Kubernetes
environment that is causing oversubscription. I
think at least 90% of that time is due to resource
contention. The jobs only take a few seconds when
run locally. I've mentioned this a couple of times
to Paul and Mark but we haven't been able to
prioritize it. It sounds like Mark and I may take
a closer look in October, but I can't troubleshoot
effectively because the run-time environment is
opaque to me.<br>
<br>
Otherwise, I agree.<br>
<br>
-- <br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a
href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a>
before posting!<br>
<br>
* Can't post? Read <a
href="http://www.gromacs.org/Support/Mailing_Lists"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a
href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>
or send a mail to <a
href="mailto:gmx-developers-request@gromacs.org"
target="_blank" moz-do-not-send="true">gmx-developers-request@gromacs.org</a>.<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Erik Lindahl <<a
href="mailto:erik.lindahl@dbb.su.se"
target="_blank"
moz-do-not-send="true">erik.lindahl@dbb.su.se</a>></div>
<div>Professor of Biophysics, Dept.
Biochemistry & Biophysics,
Stockholm University</div>
<div>Science for Life Laboratory, Box
1031, 17121 Solna, Sweden</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Erik Lindahl <<a
href="mailto:erik.lindahl@dbb.su.se"
target="_blank" moz-do-not-send="true">erik.lindahl@dbb.su.se</a>></div>
<div>Professor of Biophysics, Dept.
Biochemistry & Biophysics, Stockholm
University</div>
<div>Science for Life Laboratory, Box
1031, 17121 Solna, Sweden</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
</blockquote>
<p><br>
</p>
<pre cols="72">--
Paul Bauer, PhD
GROMACS Development Manager
KTH Stockholm, SciLifeLab
0046737308594</pre>
</div>
-- <br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a
href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a>
before posting!<br>
<br>
* Can't post? Read <a
href="http://www.gromacs.org/Support/Mailing_Lists"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a
href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>
or send a mail to <a
href="mailto:gmx-developers-request@gromacs.org"
target="_blank" moz-do-not-send="true">gmx-developers-request@gromacs.org</a>.</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Erik Lindahl <<a
href="mailto:erik.lindahl@dbb.su.se"
target="_blank" moz-do-not-send="true">erik.lindahl@dbb.su.se</a>></div>
<div>Professor of Biophysics, Dept. Biochemistry
& Biophysics, Stockholm University</div>
<div>Science for Life Laboratory, Box 1031, 17121
Solna, Sweden</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
</blockquote>
<p><br>
</p>
<pre class="moz-signature" cols="72">--
Paul Bauer, PhD
GROMACS Development Manager
KTH Stockholm, SciLifeLab
0046737308594</pre>
</body>
</html>