<p dir="ltr">Hi guys,</p>
<p dir="ltr">Thanks for the ideas. I can use CUDA_VISIBLE_DEVICES to get the test to run on any of the devices, but only with some of the possible combinations of the three devices visible. Several working combinations includes the old GT 480 GPU, but that shouldn't matter because no builds are using CUDA 9.0. And nvidia-smi reports that all devices are in the default non-exclusive mode. mdrun is perfectly happy with the GPUs. </p>
<p dir="ltr">Mark</p>
<br><div class="gmail_quote"><div dir="ltr">On Mon, 20 Nov 2017 08:07 Millad Ghane <<a href="mailto:mghane@cs.uh.edu">mghane@cs.uh.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello Mark,<br>
<br>
It might be due to existence of another program that makes the GPU<br>
busy for itself. If there is a program that uses GPU, the CUDA driver<br>
might halt your program and return that error. It makes sense, right?<br>
No sharing.<br>
<br>
<br>
Bests,<br>
Millad<br>
<br>
<br>
On Sun, Nov 19, 2017 at 11:26 PM, Mark Abraham <<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>> wrote:<br>
> Hi,<br>
><br>
> I seem to have broken bs_nix1310 while trying to install CUDA 9.0 + a driver<br>
> than can use it. It worked fine on bs_nix1204, but something is different<br>
> about this machine. It doesn't seem to be just that it has one really old<br>
> GPU.<br>
><br>
> On gerrit, you will see errors from Jenkins builds on that slave consistent<br>
> with<br>
><br>
> cudaFuncGetAttributes failed: all CUDA-capable devices are busy or<br>
> unavailable<br>
><br>
> for some of the PME-related unit tests, until we sort out how to get a<br>
> functional driver working on it. Sorry for the inconvenience!<br>
><br>
> (More details at <a href="https://redmine.gromacs.org/boards/6/topics/851?r=855" rel="noreferrer" target="_blank">https://redmine.gromacs.org/boards/6/topics/851?r=855</a> for<br>
> those with access to it.)<br>
><br>
> Mark<br>
><br>
> --<br>
> Gromacs Developers mailing list<br>
><br>
> * Please search the archive at<br>
> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before<br>
> posting!<br>
><br>
> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
><br>
> * For (un)subscribe requests visit<br>
> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or<br>
> send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
</blockquote></div>