<div dir="ltr">Hi,<div><br></div><div>Looks like I needed to purge the installs of old drivers before adding new ones on bs_nix1310. I didn't need to do that on bs_nix1204, but I've learned a lesson now :-).</div><div><br></div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Nov 20, 2017 at 11:36 AM Szilárd Páll <<a href="mailto:pall.szilard@gmail.com">pall.szilard@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hi,<br></div><div><br></div><div>CC 2.0 devices are indeed unsupported by CUDA 9.0 RT so I could imagine that they "forgot" to test the 384-series (9.0 driver version) drivers them properly.</div><div><br></div>I suggest trying a different driver. We have been using 384.90 on a couple of dev-machines, although none with CC 2.0 cards. Otherwise, reverting to 375 would be the best option.<br><br></div>Cheers,<br></div><div class="gmail_extra"><br clear="all"><div><div class="m_-128764281819720521gmail_signature" data-smartmail="gmail_signature">--<br>Szilárd</div></div></div><div class="gmail_extra">
<br><div class="gmail_quote">On Mon, Nov 20, 2017 at 8:13 AM, Mark Abraham <span dir="ltr"><<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Hi guys,</p>
<p dir="ltr">Thanks for the ideas. I can use CUDA_VISIBLE_DEVICES to get the test to run on any of the devices, but only with some of the possible combinations of the three devices visible. Several working combinations includes the old GT 480 GPU, but that shouldn't matter because no builds are using CUDA 9.0. And nvidia-smi reports that all devices are in the default non-exclusive mode. mdrun is perfectly happy with the GPUs. </p><span class="m_-128764281819720521HOEnZb"><font color="#888888">
<p dir="ltr">Mark</p></font></span><div class="m_-128764281819720521HOEnZb"><div class="m_-128764281819720521h5">
<br><div class="gmail_quote"><div dir="ltr">On Mon, 20 Nov 2017 08:07 Millad Ghane <<a href="mailto:mghane@cs.uh.edu" target="_blank">mghane@cs.uh.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello Mark,<br>
<br>
It might be due to existence of another program that makes the GPU<br>
busy for itself. If there is a program that uses GPU, the CUDA driver<br>
might halt your program and return that error. It makes sense, right?<br>
No sharing.<br>
<br>
<br>
Bests,<br>
Millad<br>
<br>
<br>
On Sun, Nov 19, 2017 at 11:26 PM, Mark Abraham <<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>> wrote:<br>
> Hi,<br>
><br>
> I seem to have broken bs_nix1310 while trying to install CUDA 9.0 + a driver<br>
> than can use it. It worked fine on bs_nix1204, but something is different<br>
> about this machine. It doesn't seem to be just that it has one really old<br>
> GPU.<br>
><br>
> On gerrit, you will see errors from Jenkins builds on that slave consistent<br>
> with<br>
><br>
> cudaFuncGetAttributes failed: all CUDA-capable devices are busy or<br>
> unavailable<br>
><br>
> for some of the PME-related unit tests, until we sort out how to get a<br>
> functional driver working on it. Sorry for the inconvenience!<br>
><br>
> (More details at <a href="https://redmine.gromacs.org/boards/6/topics/851?r=855" rel="noreferrer" target="_blank">https://redmine.gromacs.org/boards/6/topics/851?r=855</a> for<br>
> those with access to it.)<br>
><br>
> Mark<br>
><br>
> --<br>
> Gromacs Developers mailing list<br>
><br>
> * Please search the archive at<br>
> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before<br>
> posting!<br>
><br>
> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
><br>
> * For (un)subscribe requests visit<br>
> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or<br>
> send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
</blockquote></div>
</div></div><br>--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br></blockquote></div><br></div>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote></div>