<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 29, 2014 at 10:39 AM, Vedran Miletić <span dir="ltr"><<a href="mailto:rivanvx@gmail.com" target="_blank">rivanvx@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi,<br>
<br>
can one use both MPI and CUDA to run Gromacs on multiple machines with<br>
mutliple GPUs? </blockquote><div><br></div><div>Yes</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Wiki page on Acceleration and parallelization [1]<br>
doesn't handle this particular scenario, and comments on bug 1135 [2]<br>
suggest GPU acceleration is meant to be used on single machine only. I<br>
might be missing something, because when I actually try to run it on a<br>
cluster with 5 machines, 1 GPU per machine like, assuming -gpu_id<br>
takes local GPU IDs<br>
<br>
mpirun -np 5 -hostfile ... mdrun_mpi -v -deffnm ... -gpu_id 00000<br></blockquote><div><br></div><div>That's not going to do what you probably think it would do. See (from mdrun -h) "The argument of -gpu_id is a string of digits (without delimiter) representing device id-s of the GPUs to be used. For example, ”02” specifies using GPUs 0 and 2 in the first and second</div>
<div>PP ranks per compute node respectively." The "per compute node" is critical here. Your -gpu_id requires there will be five PP ranks *per node* to address. That your MPI hostfile made this possible is a separate question for you to consider. You sound like you want to use</div>
<div><br></div><div>mpirun -np 5 -hostfile ... mdrun_mpi -v -deffnm ... -gpu_id 0<br></div><div><br></div><div>We do it this way because it is a much more scaleable way of expressing the underlying requirement that a PP rank on a node maps to a GPU on a node, when there's more than one GPU per node. <a href="http://www.gromacs.org/Documentation/Acceleration_and_parallelization#Using_multi-simulations_and_GPUs">http://www.gromacs.org/Documentation/Acceleration_and_parallelization#Using_multi-simulations_and_GPUs</a> hints at this, but that page doesn't cover your case. I'll add it.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
it seems to run, and nvidia-smi indicates gmx_mpi running a process.<br>
Very nice! However, the output of mdrun_mpi doesn't seem to be<br>
correct:<br>
<br>
1 GPU detected on host akston:<br>
#0: NVIDIA GeForce GTX 660, compute cap.: 3.0, ECC: no, stat: compatible<br>
<br>
1 GPU user-selected for this run.<br>
Mapping of GPU ID to the 5 PP ranks in this node: 0,0,0,0,0<br>
NOTE: You assigned GPUs to multiple MPI processes.<br>
<br>
I certainly did not assign GPU to multiple MPI processes, because<br>
there are 5 GPUs which have no info about them printed on screen.</blockquote><div><br></div><div>Actually you did such an assignment, as covered above. We only report the detection from the lowest PP rank on the lowest-ranked node, because we haven't bothered to serialize the data to check things are sane. Usually such a machine is sufficiently homogeneous that this is not an issue.</div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> So,<br>
is mdrun output wrong here and the code just works? Or is it that<br>
mdrun should warn even harder that this scenario is unsupported?<br></blockquote><div><br></div><div>Works fine, but the output could be more specific. Documenting all cases sanely where someone will find it is hard.</div>
<div><br></div><div>Mark</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Thanks in advance.<br>
<br>
Regards,<br>
Vedran<br>
<br>
[1] <a href="http://www.gromacs.org/Documentation/Acceleration_and_parallelization" target="_blank">http://www.gromacs.org/Documentation/Acceleration_and_parallelization</a><br>
[2] <a href="http://bugzilla.gromacs.org/issues/1135" target="_blank">http://bugzilla.gromacs.org/issues/1135</a><br>
<span class=""><font color="#888888">--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
</font></span></blockquote></div><br></div></div>