<table cellspacing="0" cellpadding="0" border="0" ><tr><td valign="top" style="font: inherit;"><br>Hi Mark,<br><br>Many thanks for your fast response!<br><br><!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:PunctuationKerning/>
<w:DrawingGridVerticalSpacing>7.8 磅</w:DrawingGridVerticalSpacing>
<w:DisplayHorizontalDrawingGridEvery>0</w:DisplayHorizontalDrawingGridEvery>
<w:DisplayVerticalDrawingGridEvery>2</w:DisplayVerticalDrawingGridEvery>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:Compatibility>
<w:SpaceForUL/>
<w:BalanceSingleByteDoubleByteWidth/>
<w:DoNotLeaveBackslashAlone/>
<w:ULTrailSpace/>
<w:DoNotExpandShiftReturn/>
<w:AdjustLineHeightInTable/>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:UseFELayout/>
</w:Compatibility>
<w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
</w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" LatentStyleCount="156">
</w:LatentStyles>
</xml><![endif]--><!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:普通表格;
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin:0cm;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman";
mso-ansi-language:#0400;
mso-fareast-language:#0400;
mso-bidi-language:#0400;}
</style>
<![endif]-->
<p class="MsoNormal"><i style=""><span lang="EN-US">What's
the network hardware? Can other machine load influence your network
performance?</span></i></p>
<p class="MsoNormal"><span lang="EN-US">The supercomputer system is based on the
Cray Gemini interconnect technology. I suppose this is a fast network hardware...</span></p><p class="MsoNormal"><br><span lang="EN-US"></span></p>
<p class="MsoNormal"><i style=""><span lang="EN-US">Are
the systems in the NVT ensemble? Use diff to check the .mdp files differ only
how you think they do.</span></i></p>
<p class="MsoNormal"><span lang="EN-US">The systems are in NPT ensemble. I saw some
discussions on the mailing list that NPT ensemble is superior to NVT ensemble
for REMD. And the .mdp files differ only in the temperature.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><i style=""><span lang="EN-US">What
are the values of nstlist and<a name="OLE_LINK12"> nstcalcenergy</a>?</span></i></p>
<p class="MsoNormal"><span lang="EN-US">Previously, nstlist=5</span>, <a style="background-color: rgb(255, 255, 255);" name="OLE_LINK15"><span style=""><span style=""><span style="background-image: none; background-repeat: repeat; background-attachment: scroll; background-position: 0% 0%; -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;" lang="EN-US">nstcalcenergy</span></span></span></a><span style="background-color: rgb(255, 255, 255);"><span style="background-image: none; background-repeat: repeat; background-attachment: scroll; background-position: 0% 0%; -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;" lang="EN-US">=1</span></span></p>
<span style=""></span>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: NimbusRomNo9L-Regu;" lang="EN-US">Thank you for
pointing this out. I checked the manual again that this option affects the
performance in parallel simulations because calculating energies requires global
communication between all processes. So I have set this option to -1 this time.
This should be one reason for the low parallel efficiency.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: NimbusRomNo9L-Regu;" lang="EN-US">And after I
changed </span><span style="background: rgb(255, 255, 255) none repeat scroll 0% 0%; -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;" lang="EN-US">nstcalcenergy=</span><span lang="EN-US">-1, I</span><span style="font-size: 11pt; font-family: NimbusRomNo9L-Regu;" lang="EN-US"> found there was a 3% improvement on the efficiency compared with those when
</span><span lang="EN-US">nstcalcenergy=1.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p style="font-style: italic;" class="MsoNormal"><span lang="EN-US">Take a look at the execution time breakdown
at the end of the .log files, and do so for more than one replica. With the
current implementation, every simulation has to synchronize and communicate
every handful of steps, which means that large scale parallelism won't work
efficiently unless you have<a name="OLE_LINK17"> fast network hardware</a> that
is dedicated to your job. This effect shows up in the "Rest" row of
the time breakdown. With <a name="OLE_LINK14"></a><a name="OLE_LINK13"><span style="">Infiniband</span></a>, I'd expect you should
only be losing about 10% of the run time total. The 30-fold loss you have upon
going from 24->42 replicas keeping 4 CPUs/replica suggests some other
contribution, however.</span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">I checked the time breakdown in the log
files for short REMD simulations. For the REMD simulaiton with 168 cores for 42
replicas, as you see below, the “Rest” makes up as surprisingly high as <b style=""><u>96.6%</u></b> of the time for one of the
replicas. This parameter is almost the same level for the other replicas. For
the REMD simulation with 96 cores for 24 replicas, the “Rest” takes up about
24%. I was also aware of your post: </span></p>
<p class="MsoNormal"><span lang="EN-US"><a href="http://www.mail-archive.com/gmx-users@gromacs.org/msg37507.html">http://www.mail-archive.com/gmx-users@gromacs.org/msg37507.html</a></span></p>
<p class="MsoNormal"><span lang="EN-US">As you suggested such big loss should be
ascribed to other factors. Do you think it is the network hardware to blame or
there are other reasons please? Any suggestion would be greatly appreciated<br></span></p>
<p class="MsoNormal"><span lang="EN-US"> </span></p>
<p class="MsoNormal"><span lang="EN-US">Computing:<span style=""> </span>Nodes<span style=""> </span>Number<span style="">
</span>G-Cycles<span style=""> </span>Seconds<span style=""> </span>%</span></p>
<p class="MsoNormal"><span lang="EN-US">-----------------------------------------------------------------------</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Domain decomp.<span style=""> </span>4<span style=""> </span>442<span style="">
</span><span style=""> </span>2.604<span style=""> </span>1.2<span style="">
</span>0.0</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>DD
comm. load<span style=""> </span>4<span style=""> </span>6<span style=""> </span>0.001<span style=""> </span>0.0<span style="">
</span>0.0</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Comm. coord.<span style=""> </span>4<span style=""> </span>2201<span style=""> </span>1.145<span style=""> </span>0.5<span style="">
</span>0.0</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Neighbor search<span style=""> </span>4<span style="">
</span>442<span style=""> </span>14.964<span style=""> </span>7.1<span style="">
</span>0.2</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Force<span style=""> </span><span style=""> </span>4<span style="">
</span>2201<span style=""> </span>175.303<span style=""> </span>83.5<span style="">
</span>2.0</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Wait
+ Comm. F<span style=""> </span>4<span style=""> </span>2201<span style=""> </span>1.245<span style=""> </span>0.6<span style="">
</span>0.0</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>PME
mesh<span style=""> </span>4<span style=""> </span>2201<span style=""> </span>30.314<span style=""> </span>14.4<span style="">
</span>0.3</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Write traj.<span style=""> </span>4<span style=""> </span>11<span style=""> </span>17.346<span style=""> </span>8.3<span style="">
</span>0.2</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Update<span style=""> </span>4<span style=""> </span>2201<span style=""> </span>2.004<span style=""> </span>1.0<span style="">
</span>0.0</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Constraints<span style=""> </span>4<span style=""> </span>2201<span style=""> </span>26.593<span style=""> </span>12.7<span style="">
</span>0.3</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Comm. energies<span style=""> </span>4<span style=""> </span>442<span style=""> </span>28.722<span style=""> </span>13.7<span style="">
</span>0.3</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Rest<span style=""> </span>4<span style=""> </span>8426.029<span style=""> </span>4012.4<span style="">
</span>96.6</span></p>
<p class="MsoNormal"><span lang="EN-US">-----------------------------------------------------------------------</span></p>
<p class="MsoNormal"><span lang="EN-US"><span style=""> </span>Total<span style=""> </span>4<span style=""> </span>8726.270<span style=""> </span>4155.4<span style="">
</span>100.0</span></p>
<br><br>Qiong<br><br>On 7/02/2011 9:52 PM, Qiong Zhang wrote:
<blockquote type="cite">
<table border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td style="font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; font-size: inherit; line-height: inherit; font-size-adjust: inherit; font-stretch: inherit;" valign="top">
<p class="yiv1366269415MsoNormal"><span lang="EN-US">Dear all
gmx-users,</span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US"> </span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">I have </span><span lang="EN-US">recently </span><span lang="EN-US">been
testing the REMD
simulations. I was running simulations on a
supercomputer system<span class="yiv1366269415highlightedsearchterm"> </span>ba<span class="yiv1366269415highlightedsearchterm">se</span>d
on the AMD Opteron 12-core (2.1 GHz) processors. The
Gromacs 4.5.3
version was used.</span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US"> </span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">I have a system of
5172 atoms, of which 138
atoms belong to solute and the other are water
molecules. An exponential
distribution of temperatures was generated ranging
from 276 to 515 K in total
of 42 replicas or from 298 to 420 K in total of 24
replicas, ensuring that the
exchange ratio between all adjacent replicas is about
0.25. The replica
exchange was carried out every 0.5ps. The integrate
step size was 2fs.</span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US"> </span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">For the above
system, when REMD is
simulated over 24 replicas, the simulation speed is
reasonably fast. However,
when REMD is simulated over 42 replicas, the
simulation speed is awfully slow.Please see the
following table for the speed.<br>
</span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">----------------------------------------------------------------------------</span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">Replica number<span style=""> </span>CPU number<span style=""> </span>speed</span></p>
<p class="yiv1366269415MsoNormal" style="margin-left: 90pt;"><span style="" lang="EN-US"><span style="">24<span style="">
</span></span></span><span lang="EN-US">96<span style=""> </span>58015steps/15minutes</span></p>
<p class="yiv1366269415MsoNormal" style="margin-left: 90pt;"><span style="" lang="EN-US"><span style="">42<span style="">
</span></span></span><span lang="EN-US">42<span style=""> </span><span style=""> </span><a rel="nofollow" name="OLE_LINK5">865steps/15minutes</a></span></p>
<p class="yiv1366269415MsoNormal" style="margin-left: 90pt;"><span style="" lang="EN-US"><span style="">42<span style="">
</span></span></span><span lang="EN-US">84<span style=""> </span>1175<a rel="nofollow" name="OLE_LINK7">steps/15minutes</a></span></p>
<p class="yiv1366269415MsoNormal" style="margin-left: 84.75pt;"><span style="" lang="EN-US"><span style="">42<span style="">
</span></span></span><span lang="EN-US">168<span style=""> </span>1875steps/15minutes</span></p>
<div style="border-style: none none solid; border-color: windowtext; border-width: medium medium 1pt; padding: 0cm 0cm 1pt;">
<p class="yiv1366269415MsoNormal" style="border: medium none ; padding: 0cm;"><span lang="EN-US">42<span style="">
</span>336<span style="">
</span>2855steps/15minutes</span></p>
</div>
<p class="yiv1366269415MsoNormal"><span lang="EN-US"> </span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">The command line
for the mdrun
is:</span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">aprun -n (CPU
number here) mdrun_d -s
md.tpr -multi (replica number here) -replex 250</span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US"> </span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">My questions are :<br>
</span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">1) why the REMD
for the 42
replicas is so slow for the same system? <br>
</span></p>
<p class="yiv1366269415MsoNormal"><span lang="EN-US">2) On what aspects
can I improve the operating
efficiency please?<br>
</span></p>
</td>
</tr>
</tbody>
</table>
</blockquote>
<br>
What's the network hardware? Can other machine load influence your
network performance?<br>
<br>
Are the systems in the NVT ensemble? Use diff to check the .mdp
files differ only how you think they do.<br>
<br>
What are the values of nstlist and nstcalcenergy?<br>
<br>
Take a look at the execution time breakdown at the end of the .log
files, and do so for more than one replica. With the current
implementation, every simulation has to synchronize and communicate
every handful of steps, which means that large scale parallelism
won't work efficiently unless you have fast network hardware that is
dedicated to your job. This effect shows up in the "Rest" row of the
time breakdown. With Infiniband, I'd expect you should only be
losing about 10% of the run time total. The 30-fold loss you have
upon going from 24->42 replicas keeping 4 CPUs/replica suggests
some other contribution, however.<br>
<br>
Mark</td></tr></table><br>