<div dir="ltr">This could indeed be important for jobs in general if they auto-detect the hardware.<div><br></div><div>The way containerization work, you will always see that there's e.g. 96 hardware threads on the server, and it is quite possible to start 96 threads (or more) - but the cgroups mechanism in the Linux kernel will limit the actual CPU usage to the limits set (e.g. 4, with our max allowed value being 8 or 16, I think). That could then lead to a job trying to run 96 threads, but only having resources equivalent to two physical cores (4 threads).</div><div><br></div><div>Cheers,</div><div><br></div><div>Erik</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 29, 2020 at 11:09 AM Paul bauer <<a href="mailto:paul.bauer.q@gmail.com">paul.bauer.q@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div>
<div>Eric, one question, how do you set the
total number of ranks for your jobs in the python script?<br>
<br>
I think the issue might be that you are oversubscribing because
you are trying to use the whole kubernetes cluster, slowing
everything down.</div>
<div><br>
</div>
<div>Cheers<br>
<br>
Paul<br>
<br>
P.s.: <a href="https://gitlab.com/gromacs/gromacs/-/merge_requests/594" target="_blank">https://gitlab.com/gromacs/gromacs/-/merge_requests/594</a> Is
ready now and works as intended.<br>
</div>
<div><br>
</div>
<div>On 28/09/2020 21:15, Erik Lindahl
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">PS:<br>
<div><br>
</div>
<div>Note that there are two settings: The *request* you set for
CPU and memory is guaranteed when you execute (although we
count each hardware thread as a CPU), while the limit is
something you ask for, but aren't guaranteed.</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Erik</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Sep 28, 2020 at 9:14
PM Erik Lindahl <<a href="mailto:erik.lindahl@gmail.com" target="_blank">erik.lindahl@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>Given that several other test containers finish in ~2
minutes, I think it's relatively unlikely that random
fluctuating resource contention would systematically
always affect the gmxapi test every time it is run, but no
other jobs :-)</div>
<div><br>
</div>
<div>The whole point of k8s/docker is that the runtime
environment is standardized; what performance do you see
if you run the same container e.g. on your laptop,
desktop, or any other cloud resource when assigning two
hardware threads to it?</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Erik</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Sep 28, 2020 at
8:38 PM Eric Irrgang <<a href="mailto:ericirrgang@gmail.com" target="_blank">ericirrgang@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
> On Sep 28, 2020, at 6:00 PM, Erik Lindahl <<a href="mailto:erik.lindahl@gmail.com" target="_blank">erik.lindahl@gmail.com</a>>
wrote:<br>
> <br>
> - gmx-api. They both take 12-15 minutes on two cores,
and there are four of them. <br>
<br>
I think there is something wrong with the way resources
are detected in the CI Kubernetes environment that is
causing oversubscription. I think at least 90% of that
time is due to resource contention. The jobs only take a
few seconds when run locally. I've mentioned this a couple
of times to Paul and Mark but we haven't been able to
prioritize it. It sounds like Mark and I may take a closer
look in October, but I can't troubleshoot effectively
because the run-time environment is opaque to me.<br>
<br>
Otherwise, I agree.<br>
<br>
-- <br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a>
before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>
or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Erik Lindahl <<a href="mailto:erik.lindahl@dbb.su.se" target="_blank">erik.lindahl@dbb.su.se</a>></div>
<div>Professor of Biophysics, Dept.
Biochemistry & Biophysics, Stockholm
University</div>
<div>Science for Life Laboratory, Box 1031,
17121 Solna, Sweden</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Erik Lindahl <<a href="mailto:erik.lindahl@dbb.su.se" target="_blank">erik.lindahl@dbb.su.se</a>></div>
<div>Professor of Biophysics, Dept. Biochemistry
& Biophysics, Stockholm University</div>
<div>Science for Life Laboratory, Box 1031, 17121
Solna, Sweden</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
</blockquote>
<p><br>
</p>
<pre cols="72">--
Paul Bauer, PhD
GROMACS Development Manager
KTH Stockholm, SciLifeLab
0046737308594</pre>
</div>
-- <br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Erik Lindahl <<a href="mailto:erik.lindahl@dbb.su.se" target="_blank">erik.lindahl@dbb.su.se</a>></div><div>Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm University</div><div>Science for Life Laboratory, Box 1031, 17121 Solna, Sweden</div></div></div></div></div></div></div></div></div>