<div>As far as I can tell, the errors are not related to GROMACS, but incorrect system configuration or incorrectly built/linked mdrun (see specific comments inline).</div><div><br></div>On Mon, Dec 3, 2012 at 3:02 PM, Justin Lemkul <span dir="ltr"><<a href="mailto:jalemkul@vt.edu" target="_blank">jalemkul@vt.edu</a>></span> wrote:<br>
<div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
Hi All,<br>
<br>
I'm having a strange problem and I'm hoping I can get some help diagnosing it. I compiled the beta release on our AMD cluster that has Tesla S2050 GPU's, but I haven't been able to successfully run anything yet. Our admins don't know much specifically about Gromacs, and it seems everything should be working, but somehow mdrun is not detecting CUDA or finding the GPU card on the compute nodes.<br>
<br>
We have CUDA 3.1, 3.2, and 4.0 available on the cluster, and I can replicate this problem with both 3.2 and 4.0. It appears that mdrun is linked properly to the CUDA libraries:<br>
<br>
$ ldd mdrun<br>
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaaacc8000)<br>
libgmxpreprocess.so.6 => /home/jalemkul/ATHENA/<u></u>software/gromacs-46beta1/bin/.<u></u>./lib/libgmxpreprocess.so.6 (0x00002aaaaaecc000)<br>
libmd.so.6 => /home/jalemkul/ATHENA/<u></u>software/gromacs-46beta1/bin/.<u></u>./lib/libmd.so.6 (0x00002aaaab1b2000)<br>
libgmx.so.6 => /home/jalemkul/ATHENA/<u></u>software/gromacs-46beta1/bin/.<u></u>./lib/libgmx.so.6 (0x00002aaaab7a2000)<br>
libm.so.6 => /lib64/libm.so.6 (0x00002aaaabfd0000)<br>
libcudart.so.4 => /cm/shared/apps/cuda40/<u></u>toolkit/4.0.17/lib64/<u></u>libcudart.so.4 (0x00002aaaac253000)<br>
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaac4c1000)<br>
libfftw3f.so.3 => /home/jalemkul/ATHENA/<u></u>software/fftw-3.3.3/lib/<u></u>libfftw3f.so.3 (0x00002aaaac6dc000)<br>
libstdc++.so.6 => /cm/shared/apps/gcc/4.3.4/<u></u>lib64/libstdc++.so.6 (0x00002aaaaca58000)<br>
libgomp.so.1 => /cm/shared/apps/gcc/4.3.4/<u></u>lib64/libgomp.so.1 (0x00002aaaacd5f000)<br>
libgcc_s.so.1 => /cm/shared/apps/gcc/4.3.4/<u></u>lib64/libgcc_s.so.1 (0x00002aaaacf67000)<br>
libc.so.6 => /lib64/libc.so.6 (0x00002aaaad17d000)<br>
/lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)<br>
librt.so.1 => /lib64/librt.so.1 (0x00002aaaad4d5000)<br>
<br>
We have a module system, so by simply using 'module load cuda40' I get all of the CUDA stuff loaded properly, i.e.:<br>
<br>
$ echo $LD_LIBRARY_PATH<br>
/cm/shared/apps/gcc/4.3.4/lib:<u></u>/cm/shared/apps/gcc/4.3.4/<u></u>lib64:/cm/shared/apps/cuda40/<u></u>toolkit/4.0.17/lib64:/cm/<u></u>local/apps/cuda40/libs/270.41.<u></u>19/usr/lib64:/cm/shared/apps/<u></u>cuda40/sdk/4.0.17/C/lib:/cm/<u></u>shared/apps/cuda40/sdk/4.0.17/<u></u>OpenCL/common/lib/Linux64<br>
<br>
So it appears mdrun should be able to load the necessary libraries. When I execute the run:<br>
<br>
$ mdrun -deffnm md -nb gpu -gpu_id 0<br>
<br>
I get the following in the .log file:<br>
<br>
Log file opened on Mon Dec 3 08:55:51 2012<br>
Host: athena002 pid: 21368 nodeid: 0 nnodes: 1<br>
Gromacs version: VERSION 4.6-beta1<br>
Precision: single<br>
MPI library: thread_mpi<br>
OpenMP support: enabled<br>
GPU support: enabled<br>
invsqrt routine: gmx_software_invsqrt(x)<br>
CPU acceleration: SSE2<br>
FFT library: fftw-3.3.3-sse2<br>
Large file support: enabled<br>
RDTSCP usage: enabled<br>
Built on: Thu Nov 29 21:38:43 EST 2012<br>
Built by: jalemkul@athena1 [CMAKE]<br>
Build OS/arch: Linux 2.6.18-194.11.4.el5 x86_64<br>
Build CPU vendor: AuthenticAMD<br>
Build CPU brand: AMD Opteron(tm) Processor 6134<br>
Build CPU family: 16 Model: 9 Stepping: 1<br>
Build CPU features: apic clfsh cmov cx8 cx16 htt lahf_lm misalignsse mmx msr nonstop_tsc pdpe1gb popcnt pse rdtscp sse2 sse3 sse4a<br>
C compiler: /cm/shared/apps/gcc/4.3.4/bin/<u></u>gcc GNU gcc (GCC) 4.3.4<br>
C compiler flags: -msse2 -Wextra -Wno-missing-field-<u></u>initializers -Wno-sign-compare -Wall -Wno-unused -Wunused-value -fomit-frame-pointer -funroll-all-loops -O3 -DNDEBUG<br>
C++ compiler: /cm/shared/apps/gcc/4.3.4/bin/<u></u>g++ GNU g++ (GCC) 4.3.4<br>
C++ compiler flags: -msse2 -Wextra -Wno-missing-field-<u></u>initializers -Wno-sign-compare -Wall -Wno-unused -Wunused-value -fomit-frame-pointer -funroll-all-loops -O3 -DNDEBUG<br>
CUDA compiler: nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2011 NVIDIA Corporation;Built on Thu_May_12_11:09:45_PDT_2011;<u></u>Cuda compilation tools, release 4.0, V0.2.1221<br>
CUDA driver: 0.0<br>
CUDA runtime: 0.0<br></blockquote><div><br></div><div>This is very fishy, it should be 4.0!</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
>From the last two lines, it seems the the driver and runtime libraries are not found. Later in the .log file, it seems that the GPU itself is not found:<br>
<br>
NOTE: Error occurred during GPU detection:<br>
CUDA driver version is insufficient for CUDA runtime version<br></blockquote><div><br></div><div>This is the very error the CUDA runtime returns and it means that the libcudart.so (which is the CUDA runtime library) is not compatible with the GPU driver. You can check the driver version by running nvidia-smi. </div>
<div><br></div><div>However, this sounds quite fishy because based on the library path above you are supposed to be using CUDA 4.0 with 270.41 drivers which should be compatible.</div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Can not use GPU acceleration, will fall back to CPU kernels.<br>
<br>
<br>
No GPUs detected<br>
<br>
<br>
------------------------------<u></u>-------------------------<br>
Program mdrun, VERSION 4.6-beta1<br>
Source code file: /home/jalemkul/gromacs-4.6-<u></u>beta1/src/gmxlib/gmx_detect_<u></u>hardware.c, line: 567<br>
<br>
Fatal error:<br>
Some of the requested GPUs do not exist, behave strangely, or are not compatible:<br>
GPU #0: inexistent<br></blockquote><div><br></div><div>You get this error because you explicitly requested GPU #0 which appears to not exist because it could not be detected.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
For more information and tips for troubleshooting, please check the GROMACS<br>
website at <a href="http://www.gromacs.org/Documentation/Errors" target="_blank">http://www.gromacs.org/<u></u>Documentation/Errors</a><br>
------------------------------<u></u>-------------------------<br>
<br>
Any ideas on troubleshooting, or things I can tell our admins to help move things along? I've got 4.6beta1 working fine on a local workstation in our lab, but the installation on the unversity's cluster is nonfunctional from the standpoint of the GPU.<br>
</blockquote><div><br></div><div>Try to compile and run a simple code, something from the CUDA SDK is ideal, e.g. deviceQuery. If that works with the same compiler setup, run-settings on the same hardware (and shows the CUDe driver/runtime versions correctly!), than it could be a library pre-loading issue or in worst case a GROMACS bug.</div>
<div><br></div><div>Cheers,</div><div>--<br>Szilárd<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
-Justin<span class=""><font color="#888888"><br>
<br>
-- <br>
==============================<u></u>==========<br>
<br>
Justin A. Lemkul, Ph.D.<br>
Research Scientist<br>
Department of Biochemistry<br>
Virginia Tech<br>
Blacksburg, VA<br>
jalemkul[at]<a href="http://vt.edu" target="_blank">vt.edu</a> | (540) 231-9080<br>
<a href="http://www.bevanlab.biochem.vt.edu/Pages/Personal/justin" target="_blank">http://www.bevanlab.biochem.<u></u>vt.edu/Pages/Personal/justin</a><br>
<br>
==============================<u></u>==========<br>
-- <br>
gmx-developers mailing list<br>
<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/<u></u>mailman/listinfo/gmx-<u></u>developers</a><br>
Please don't post (un)subscribe requests to the list. Use the www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@<u></u>gromacs.org</a>.<br>
</font></span></blockquote></div><br></div>