<div dir="ltr">Hi,<br><br>Thanks Mark for the fixes, I'll review the change this afternoon.<br><br>Yesterday filed a redmine related to the NVIDIA OpenCL segfaults (<a href="http://redmine.gromacs.org/issues/1871" target="_blank">http://redmine.gromacs.org/issues/1871</a>) because in my testing I reproduced the issue with recent CUDA compiler/driver too. Not sure if this is a bug in the release 5.1 code or in the NVIDIA runtime, but given that on two ranks I did reproducibly get segfault with the three tests that jenkins complains about, the documentation changes that suggest CUDA >=v6.5 may not be enough.<div><div><br></div><div>Cheers,</div><div>--<br>Szilárd<div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 8, 2015 at 4:27 AM, Mark Abraham <span dir="ltr"><<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I've uploaded a patch that addresses a couple of the issues - the regressiontests are fine like I said - I think the segfaults are indeed coming from a broken version of CUDA (have updated the opencl test config to try 6.5). Agree we should probably bump the minimum version of CUDA for OpenCL and avoid trouble. </div><div><br></div><div>The empty-domain test (that I added to cover a hard-to-reproduce bug in our GPU stream handling) requires two ranks. I used to hard-code this in the CUDA days, which was OK then but not now with OpenCL needed in Jenkins, so my patch tries to rely better on the new automated resource assignment, but Jenkins can be the judge of that. I think we were also mis-managing the OpenCL version of the code that waited for non-local events before starting local events - that test case at least did its job (eventually).</div><div><br></div><div>Also added some error code strings that we might make more general use of in future.</div><div><br></div><div><a href="http://jenkins.gromacs.org/job/Gromacs_Gerrit_5_1-test-opencl-slave/15/" target="_blank">http://jenkins.gromacs.org/job/Gromacs_Gerrit_5_1-test-opencl-slave/15/</a><br></div><div><a href="https://gerrit.gromacs.org/#/c/5430/" target="_blank">https://gerrit.gromacs.org/#/c/5430/</a><span><font color="#888888"><br></font></span></div><span><font color="#888888"><div><br></div><div>Mark</div></font></span></div><div><div><br><div class="gmail_quote"><div dir="ltr">On Tue, Dec 8, 2015 at 4:36 AM Szilárd Páll <<a href="mailto:pall.szilard@gmail.com" target="_blank">pall.szilard@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div>All three segfaults produce a backtrace similar to this:<div><br></div><div><div>[...]</div><div>#2 0x00007fcf53618632 in ?? () from /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.1</div><div>#3 0x00007fcf58b6fdca in sync_ocl_event (stream=0x7fcf4c820160, ocl_event=0x7fcf4c031380)</div><div> at /mnt/workspace/Gromacs_Gerrit_5_1-test-opencl-slave/d27c5006/gromacs/src/gromacs/mdlib/nbnxn_ocl/nbnxn_ocl.cpp:331</div><div>#4 0x00007fcf58b70f7d in nbnxn_gpu_launch_cpyback (nb=0x7fcf4c030f40, nbatom=0x7fcf4c022ca0, </div><div> flags=1015, aloc=0)</div><div> at /mnt/workspace/Gromacs_Gerrit_5_1-test-opencl-slave/d27c5006/gromacs/src/gromacs/mdlib/nbnxn_ocl/nbnxn_ocl.cpp:952</div><div>#5 0x00007fcf58b65fcc in do_force_cutsVERLET ()</div><div> at /mnt/workspace/Gromacs_Gerrit_5_1-test-opencl-slave/d27c5006/gromacs/src/gromacs/mdlib/sim_util.cpp:1061</div><div>#6 0x00007fcf58b68e02 in do_force ()</div><div> at /mnt/workspace/Gromacs_Gerrit_5_1-test-opencl-slave/d27c5006/gromacs/src/gromacs/mdlib/sim_util.cpp:2009</div><div>#7 0x000000000041ac0e in do_md ()</div><div> at /mnt/workspace/Gromacs_Gerrit_5_1-test-opencl-slave/d27c5006/gromacs/src/programs/mdrun/md.cpp:1078</div><div>#8 0x000000000042835b in mdrunner ()</div><div> at /mnt/workspace/Gromacs_Gerrit_5_1-test-opencl-slave/d27c5006/gromacs/src/programs/mdrun/runner.cpp:1282</div><div>#9 0x000000000042528e in mdrunner_start_fn (arg=0xb8ddd0)</div><div> at /mnt/workspace/Gromacs_Gerrit_5_1-test-opencl-slave/d27c5006/gromacs/src/programs/mdrun/runner.cpp:186</div><div>[...]</div></div><div><br></div><div>This could be due to an old CUDA being used. I'll check that, but in any case, especially for NVIDIA OpenCL that we know it's been buggy (and as far as I know still is), we probably really should not use anything older than 7.0 or 7.5.</div><div><br></div><div>The other failures on the AMD test machine seem to be caused by the tests being called in an incompatible way, although I have the feeling that something is off with that too (because tMPI+OpenCL multi-GPU should work, I though).</div><div class="gmail_extra"><br clear="all"><div><div>--<br>Szilárd</div></div></div></div><div dir="ltr"><div class="gmail_extra">
<br><div class="gmail_quote">On Mon, Dec 7, 2015 at 2:46 PM, Vedran Miletić <span dir="ltr"><<a href="mailto:rivanvx@gmail.com" target="_blank">rivanvx@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Szilard, Mark,<br>
<br>
thanks for looking into this.<br>
<br>
2015-12-07 14:29 GMT+01:00 Szilárd Páll <<a href="mailto:pall.szilard@gmail.com" target="_blank">pall.szilard@gmail.com</a>>:<br>
> <a href="http://jenkins.gromacs.org/job/Gromacs_Gerrit_5_1-test-opencl-slave/14" rel="noreferrer" target="_blank">http://jenkins.gromacs.org/job/Gromacs_Gerrit_5_1-test-opencl-slave/14</a><br>
<br>
Didn't know we had that one. Very nice.<br>
<div><div><br>
Regards,<br>
Vedran<br>
<br>
--<br>
Vedran Miletić<br>
<a href="http://vedranmileti.ch/" rel="noreferrer" target="_blank">http://vedranmileti.ch/</a><br>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</div></div></blockquote></div><br></div></div>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote></div>
</div></div><br>--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br></blockquote></div><br></div></div></div></div>