<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi, <div><br></div><div>most likely the Ethernet is the problem here. I compiled some numbers for the DPPC </div><div>benchmark in the paper "Speeding up parallel GROMACS on high-latency networks", </div><div><a href="http://www3.interscience.wiley.com/journal/114205207/abstract?CRETRY=1&SRETRY=0">http://www3.interscience.wiley.com/journal/114205207/abstract?CRETRY=1&SRETRY=0</a></div><div>which are for version 3.3, but PME will behave similarly. If you did not already use </div><div>separate PME nodes, this is worth a try, since on Ethernet the performance will drastically</div><div>depend on the number of nodes involved in the FFT. I also have a tool which finds the</div><div>optimal PME settings for a given number of nodes, by varying the number of PME nodes</div><div>and the fourier grid settings. I can send it to you if you want.</div><div><br></div><div>Carsten</div><div><br></div><div><br><div><div>On Nov 9, 2008, at 10:30 PM, Yawar JQ wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><font face="arial,helvetica,sans-serif">I was wondering if anyone could comment on these benchmark results for the d.dppc benchmark?</font></div> <div><font face="arial,helvetica,sans-serif"></font> </div> <div> <table style="WIDTH: 179pt; BORDER-COLLAPSE: collapse" border="0" cellspacing="0" cellpadding="0" width="238"> <colgroup> <col style="WIDTH: 35pt; mso-width-source: userset; mso-width-alt: 1718" width="47"> <col style="WIDTH: 77pt; mso-width-source: userset; mso-width-alt: 3730" width="102"> <col style="WIDTH: 67pt; mso-width-source: userset; mso-width-alt: 3254" width="89"> </colgroup><tbody> <tr style="HEIGHT: 15pt" height="20"> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; WIDTH: 35pt; HEIGHT: 15pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" height="20" width="47"><font face="Calibri">Nodes</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; WIDTH: 77pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" width="102"><font face="Calibri">Cutoff (ns/day)</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; WIDTH: 67pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" width="89"><font face="Calibri">PME (ns/day)</font></td></tr> <tr style="HEIGHT: 15pt" height="20"> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; HEIGHT: 15pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" height="20"><font face="Calibri">4</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">1.331</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">0.797</font></td></tr> <tr style="HEIGHT: 15pt" height="20"> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; HEIGHT: 15pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" height="20"><font face="Calibri">8</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">2.564</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">1.497</font></td></tr> <tr style="HEIGHT: 15pt" height="20"> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; HEIGHT: 15pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" height="20"><font face="Calibri">16</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">4.5</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">1.92</font></td></tr> <tr style="HEIGHT: 15pt" height="20"> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; HEIGHT: 15pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" height="20"><font face="Calibri">32</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">8.308</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">0.575</font></td></tr> <tr style="HEIGHT: 15pt" height="20"> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; HEIGHT: 15pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" height="20"><font face="Calibri">64</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">13.5</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">0.275</font></td></tr> <tr style="HEIGHT: 15pt" height="20"> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; HEIGHT: 15pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" height="20"><font face="Calibri">128</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">20.093</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">-</font></td></tr> <tr style="HEIGHT: 15pt" height="20"> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; HEIGHT: 15pt; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65" height="20"><font face="Calibri">192</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">21.6</font></td> <td style="BORDER-BOTTOM: #d4d0c8; BORDER-LEFT: #d4d0c8; BACKGROUND-COLOR: transparent; BORDER-TOP: #d4d0c8; BORDER-RIGHT: #d4d0c8" class="xl65"><font face="Calibri">-</font></td></tr></tbody></table></div> <div> </div> <div><font face="arial,helvetica,sans-serif">It seems to scale relatively well up to 32-64 nodes without PME. This seems slightly better than the benchmark results for Gromacs 3 on <a href="http://www.gromacs.org/">www.gromacs.org</a>. </font></div> <div><font face="arial,helvetica,sans-serif"></font> </div> <div><font face="arial,helvetica,sans-serif">Can someone comment on the magnitude of the performance hit and lack of scaling with PME is worrying me. </font></div> <div><font face="arial,helvetica,sans-serif"></font> </div> <div><font face="arial,helvetica,sans-serif">For the PME runs, I set rlist,rvdw,rouloumb=1.2 and the rest set to the defaults. I can try it with some other settings, larger spacing for the grid, but I'm not sure how much more that would help. Is there a more standardized system I should use for testing PME scaling?</font></div> <div><font face="arial,helvetica,sans-serif"></font> </div> <div><font face="arial,helvetica,sans-serif">This is with GNU compilers and parallelization with OpenMPI 1.2. I'm not sure what we're using for the FFTW The compute nodes are Dell m600 blades w/ 16GB of RAM and dual quad core Intel Xeon 3GHz processors. I believe it's all ethernet interconnects.</font></div> <div><font face="arial,helvetica,sans-serif"></font> </div> <div><font face="arial,helvetica,sans-serif">Thanks,</font></div> <div><font face="arial,helvetica,sans-serif">YQ</font></div> _______________________________________________<br>gmx-users mailing list <a href="mailto:gmx-users@gromacs.org">gmx-users@gromacs.org</a><br><a href="http://www.gromacs.org/mailman/listinfo/gmx-users">http://www.gromacs.org/mailman/listinfo/gmx-users</a><br>Please search the archive at http://www.gromacs.org/search before posting!<br>Please don't post (un)subscribe requests to the list. Use the <br>www interface or send it to gmx-users-request@gromacs.org.<br>Can't post? Read http://www.gromacs.org/mailing_lists/users.php</blockquote></div><br></div></body></html>