<div dir="ltr"><div class="gmail_extra"><div><div class="gmail_signature">On Mon, Apr 25, 2016 at 8:46 PM, Mark Abraham <span dir="ltr"><<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>></span> wrote:<br></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<br><div><br><div class="gmail_quote"><span class=""><div dir="ltr">On Mon, Apr 25, 2016 at 7:19 PM Szilárd Páll <<a href="mailto:pall.szilard@gmail.com" target="_blank">pall.szilard@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Confirmed, we've ran into it last Friday and 5820 seemed to fix the issue, but as the author of the change noted, it's is unclear what the source of the crash is.<div><br></div><div>BTW: we need to add a "-pin on" test to the verification matrix to make sure the thread pinning code gets tested. It can be post-submit too, but we have none of those on the horizon so better add an otpion to the current ones IMO.</div></div></blockquote><div><br></div></span><div>Yes and no. Doing it that way<br>* doesn't test the code any more than that it doesn't crash,<br></div>* creates another degree of freedom of coverage for a matrix to manage, and<br><div>* creates a situation where Jenkins could thrash if enough threads get pinned to a common core<br><br></div><div>Much better is a unit test </div></div></div></div></blockquote><div><br></div><div><div>I partially agree, a unit test would be suitable to verify the functionality of the pinning code in isolation from the "outer world". However, it will not be able to control external conditions (e.g. affinity set outside mdrun) - which I was going to mention bu it slipped my mind. <br></div></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><div>that puts a bunch of unpinned threads doing some simple computation and observes the expected behaviour of similar pinned threads.</div></div></div></div></blockquote><div><br></div><div>Why is doing computation relevant to testing stuff? Pinned or not, threads will execute code correctly.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><div> That can run on every configuration because it can be a unit test that runs in milliseconds. Of course, we wrote this test before we changed the old working code, right?</div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_quote"><div><span class=""><font color="#888888"><br><br></font></span></div><span class=""><font color="#888888"><div>Mark<br></div></font></span><div><div class="h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><br clear="all"><div><div>--<br>Szilárd</div></div></div><div class="gmail_extra">
<br><div class="gmail_quote">On Mon, Apr 25, 2016 at 7:07 PM, Vedran Miletić <span dir="ltr"><<a href="mailto:rivanvx@gmail.com" target="_blank">rivanvx@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Gladly. This one? <a href="https://gerrit.gromacs.org/#/c/5820/" target="_blank">https://gerrit.gromacs.org/#/c/5820/</a><span><font color="#888888"><br><br></font></span></div><span><font color="#888888">V.<br></font></span></div><div><div><br><div class="gmail_quote"><div dir="ltr">pon, 25. tra 2016. u 19:06 Mark Abraham <<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>> napisao je:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><p dir="ltr">Hi,</p>
<p dir="ltr">Unsure offhand, but there's a fix in gerrit in this area if you want to try that?</p>
<p dir="ltr">Mark</p>
<br><div class="gmail_quote"></div><div class="gmail_quote"><div dir="ltr">On Mon, 25 Apr 2016 19:05 Vedran Miletić <<a href="mailto:rivanvx@gmail.com" target="_blank">rivanvx@gmail.com</a>> wrote:<br></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<br><div><br>since fa1360610d6fcf7eb263ce1181d9954074fd5151 "Make thread affinity failures always end up in log", I get crashes in mdrun when using tMPI on any simulation I tried (does not affect OpenMPI). I am seeing this on two machines using Fedora 23 and 24, GCC 5.3 and 6, respectively.<br><br>Backtrace is <br><br>#0 0x00007ffff78c966f in tMPI_Thread_getspecific (key=...) at /home/miletivn/workspace/gromacs/src/external/thread_mpi/src/pthreads.c:571<br>#1 0x00007ffff78cff34 in tMPI_Reduce (sendbuf=0x7fffffffa4dc, recvbuf=0x7fffffffa4d8, count=1, datatype=0x7ffff7dd6660 <tmpi_int>, op=TMPI_LAND, root=0, comm=0x0) at /home/miletivn/workspace/gromacs/src/external/thread_mpi/src/reduce.c:247<br>#2 0x00007ffff63038a5 in invalidWithinSimulation (cr=0x681bd0, invalidLocally=false) at /home/miletivn/workspace/gromacs/src/gromacs/mdrunutility/threadaffinity.cpp:73<br>#3 0x00007ffff6303c0b in get_thread_affinity_layout (fplog=0x689410, cr=0x681bd0, hwinfo=0x680230, threads=8, pin_offset=0, pin_stride=0x7fffffffc634, localityOrder=0x7fffffffc638) at /home/miletivn/workspace/gromacs/src/gromacs/mdrunutility/threadaffinity.cpp:142<br>...<br><br></div><div>Variable key looks like<br><br>$1 = {initialized = {value = 0, padding = '\000' <repeats 59 times>}, key = 0x0}<br><br></div><div>So key is uninitialized. Any idea why?<br></div><div><br></div><div>Regards,<br></div><div>Vedran<br></div></div></blockquote></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote></div>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote></div>
</div></div><br>--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br></blockquote></div><br></div>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote></div></div></div></div></div>
<br>--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br></blockquote></div><br></div></div>