<div dir="ltr">Hi,<div><br></div><div>Gilles: If this happens, I *suspect* this might only be in the hardware detection phase, right?</div><div><br></div><div>We might indeed have overlooked that, since ARM was originally just a low-end platform (think 32-bit Tegra...) where we never even thought of running anything multi-node.</div><div><br></div><div>We've already thought of adding cgroups awareness (for docker), so could you possibly assist me by showing a concrete example of this on A64fx:</div><div><br></div><div>1) lscpu </div><div><br></div><div>2) cgroups information, ideally both for the node and specific MPI processes</div><div><br></div><div>... since otherwise we're programming a bit blind :-)</div><div><br></div><div>Cheers,</div><div><br></div><div>Erik</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 11, 2020 at 8:47 AM Berk Hess <<a href="mailto:hess@kth.se">hess@kth.se</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
Then there is a strange bug somewhere.<br>
<br>
Are you using real MPI or thread-MPI. Do you know where this happens? <br>
Are you running with OpenMP support?<br>
<br>
Cheers,<br>
<br>
Berk<br>
<br>
On 2020-11-11 08:26, Gilles Gouaillardet wrote:<br>
> Berk,<br>
><br>
><br>
> There is a total of 52 cores, and my observation is that each MPI task <br>
> does spawn 52 threads.<br>
><br>
> So the worst case scenario is 48 MPI tasks each spawning 52 threads, <br>
> so a total of 48 * 52 threads on a single node<br>
><br>
><br>
> Cheers,<br>
><br>
><br>
> Gilles<br>
><br>
> On 11/11/2020 4:23 PM, Berk Hess wrote:<br>
>> On 2020-11-11 03:37, Gilles Gouaillardet wrote:<br>
>>> Erik and all,<br>
>>><br>
>>><br>
>>> I am kind of facing the exact opposite issue on an other ARM processor:<br>
>>><br>
>>> High end A64fx (Fugaku/FX1000) have 48 cores plus 2 or 4 assistant <br>
>>> cores.<br>
>>><br>
>>> A job is put in a cgroup of 48 cores (e.g. no assistant cores)<br>
>>><br>
>>> Worst case scenario, a flat mpi run (48 tasks) will spawn 48 * 52 <br>
>>> cores to spin up all the cores.<br>
>> What do you mean with 48 * 52 cores? There are only 52 cores. Gromacs <br>
>> will by default not spawn more threads in total then there are cores. <br>
>> If you ask for 48 MPI ranks with a real MPI library Gromacs will not <br>
>> spawn any additional threads. With thread-mpi it will spawn 48 or 52 <br>
>> total.<br>
>><br>
>> Cheers,<br>
>><br>
>> Berk<br>
>>><br>
>>> 1) GROMACS is not cgroup aware and hence consider there are 52 (or <br>
>>> 50) cores per node (this is a very minor issue)<br>
>>><br>
>>> 2) spawning such a high number of threads caused some crashes (weird <br>
>>> stack traces, I did not spend much time investigating)<br>
>>><br>
>>> 3) in the case of A64fx, all cores are up and running, ready to <br>
>>> crunch, and do not require any special tricks.<br>
>>><br>
>>><br>
>>> At this stage, I think the easiest path to address this on A64fx is <br>
>>> to add yet an other cmake option to<br>
>>><br>
>>> unconditionally skip the spin up phase.<br>
>>><br>
>>> This could be improved by adding a command line option, or an <br>
>>> environment variable, to change the default behavior<br>
>>><br>
>>> (default behavior should be a cmake option imho)<br>
>>><br>
>>><br>
>>> Any thoughts on how to best move forward?<br>
>>><br>
>>><br>
>>> Cheers,<br>
>>><br>
>>><br>
>>> Gilles<br>
>>><br>
>>> On 11/11/2020 6:08 AM, Erik Lindahl wrote:<br>
>>>> We might be able to work around the last aspect, but it will likely <br>
>>>> take a couple of weeks until I can lay hands on a new ARM-based <br>
>>>> Macbook.<br>
>>>><br>
>>>> Long story made short: The advanced power-saving features on ARM <br>
>>>> mean some cores are not visible until they are used, so we created <br>
>>>> a small hack where we "spin up" the CPU by exercising all cores.<br>
>>>><br>
>>>> We might anyway need to do something different with the new type of <br>
>>>> big.LITTLE cores where we have 4+4 or 8+4 cores, but I can't even <br>
>>>> start to work on that until I have suitable hardware. The good news <br>
>>>> is that such hardware was announced a couple of hours ago, with <br>
>>>> availability next week ;-)<br>
>>>><br>
>>>> Cheers,<br>
>>>><br>
>>>> Erik<br>
>>>><br>
>>>> On Tue, Nov 10, 2020 at 9:45 PM Mark Abraham <br>
>>>> <<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a> <mailto:<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>>> wrote:<br>
>>>><br>
>>>> No, the use of std::thread in eg hardware detection also requires<br>
>>>> a lower level threading implementation.<br>
>>>><br>
>>>> Mark<br>
>>>><br>
>>>> On Tue, Nov 10, 2020, 20:41 Berk Hess <<a href="mailto:hess@kth.se" target="_blank">hess@kth.se</a><br>
>>>> <mailto:<a href="mailto:hess@kth.se" target="_blank">hess@kth.se</a>>> wrote:<br>
>>>><br>
>>>> Hi,<br>
>>>><br>
>>>> Turning off GMX_THREAD_MPI in cmake should remove the<br>
>>>> dependency on pthreads.<br>
>>>><br>
>>>> Cheers,<br>
>>>><br>
>>>> Berk<br>
>>>><br>
>>>> On 2020-11-10 18:06, Guido Giuntoli wrote:<br>
>>>>><br>
>>>>> Hi,<br>
>>>>><br>
>>>>> Is there any way to disable the “Pthreads” dependency during<br>
>>>>> the configuration/compilation of GROMACS?<br>
>>>>><br>
>>>>> *Best regards | Mit freundlichen Grüßen*<br>
>>>>><br>
>>>>> **<br>
>>>>><br>
>>>>> *Guido Giuntoli***<br>
>>>>><br>
>>>>> **<br>
>>>>><br>
>>>>> HUAWEI TECHNOLOGIES Duesseldorf GmbH<br>
>>>>> Hansaallee 205, 40549 Dusseldorf, Germany, *<a href="http://www.huawei.com" rel="noreferrer" target="_blank">www.huawei.com</a>*<br>
>>>>> <<a href="http://www.huawei.com/" rel="noreferrer" target="_blank">http://www.huawei.com/</a>><br>
>>>>> Registered Office: Düsseldorf, Register Court Düsseldorf, HRB<br>
>>>>> 56063,<br>
>>>>> Managing Director: Li Peng, Li Jian, Shi Yanli**<br>
>>>>><br>
>>>>> Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf,<br>
>>>>> HRB 56063,<br>
>>>>> Geschäftsführer: Li Peng, Li Jian, Shi Yanli<br>
>>>>><br>
>>>>> *-----------------------------------------------------------------------------------------------* <br>
>>>>><br>
>>>>><br>
>>>>> *This e-mail and its attachments contain confidential<br>
>>>>> information from HUAWEI, which is intended only for the<br>
>>>>> person or entity whose address is listed above. Any use of<br>
>>>>> the information contained herein in any way (including, but<br>
>>>>> not limited to, total or partial disclosure, reproduction, or<br>
>>>>> dissemination) by persons other than the intended<br>
>>>>> recipient(s) is prohibited. If you receive this e-mail in<br>
>>>>> error, please notify the sender by phone or email immediately<br>
>>>>> and delete it!*<br>
>>>>><br>
>>>>><br>
>>>><br>
>>>> -- Gromacs Developers mailing list<br>
>>>><br>
>>>> * Please search the archive at<br>
>>>> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a><br>
>>>> <<a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a>><br>
>>>> before posting!<br>
>>>><br>
>>>> * Can't post? Read<br>
>>>> <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
>>>> <<a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a>><br>
>>>><br>
>>>> * For (un)subscribe requests visit<br>
>>>> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> <br>
>>>><br>
>>>> <<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>> <br>
>>>><br>
>>>> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a><br>
>>>> <mailto:<a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>>.<br>
>>>><br>
>>>> -- Gromacs Developers mailing list<br>
>>>><br>
>>>> * Please search the archive at<br>
>>>> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a><br>
>>>> <<a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a>><br>
>>>> before posting!<br>
>>>><br>
>>>> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
>>>> <<a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a>><br>
>>>><br>
>>>> * For (un)subscribe requests visit<br>
>>>> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> <br>
>>>><br>
>>>> <<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>> <br>
>>>><br>
>>>> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a><br>
>>>> <mailto:<a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>>.<br>
>>>><br>
>>>><br>
>>>><br>
>>>> -- <br>
>>>> Erik Lindahl <<a href="mailto:erik.lindahl@dbb.su.se" target="_blank">erik.lindahl@dbb.su.se</a> <mailto:<a href="mailto:erik.lindahl@dbb.su.se" target="_blank">erik.lindahl@dbb.su.se</a>>><br>
>>>> Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm <br>
>>>> University<br>
>>>> Science for Life Laboratory, Box 1031, 17121 Solna, Sweden<br>
>>>><br>
>><br>
<br>
-- <br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Erik Lindahl <<a href="mailto:erik.lindahl@dbb.su.se" target="_blank">erik.lindahl@dbb.su.se</a>></div><div>Professor of Biophysics, Dept. Biochemistry & Biophysics, Stockholm University</div><div>Science for Life Laboratory, Box 1031, 17121 Solna, Sweden</div></div></div></div></div></div></div></div></div>