<table cellspacing="0" cellpadding="0" border="0" ><tr><td valign="top" style="font: inherit;"><!--[if gte mso 9]><xml>
 <w:WordDocument>
  <w:View>Normal</w:View>
  <w:Zoom>0</w:Zoom>
  <w:PunctuationKerning/>
  <w:DrawingGridVerticalSpacing>7.8 磅</w:DrawingGridVerticalSpacing>
  <w:DisplayHorizontalDrawingGridEvery>0</w:DisplayHorizontalDrawingGridEvery>
  <w:DisplayVerticalDrawingGridEvery>2</w:DisplayVerticalDrawingGridEvery>
  <w:ValidateAgainstSchemas/>
  <w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
  <w:IgnoreMixedContent>false</w:IgnoreMixedContent>
  <w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
  <w:Compatibility>
   <w:SpaceForUL/>
   <w:BalanceSingleByteDoubleByteWidth/>
   <w:DoNotLeaveBackslashAlone/>
   <w:ULTrailSpace/>
   <w:DoNotExpandShiftReturn/>
   <w:AdjustLineHeightInTable/>
   <w:BreakWrappedTables/>
   <w:SnapToGridInCell/>
   <w:WrapTextWithPunct/>
   <w:UseAsianBreakRules/>
   <w:DontGrowAutofit/>
   <w:UseFELayout/>
  </w:Compatibility>
  <w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
 </w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
 <w:LatentStyles DefLockedState="false" LatentStyleCount="156">
 </w:LatentStyles>
</xml><![endif]--><!--[if gte mso 10]>
<style>
 /* Style Definitions */
 table.MsoNormalTable
        {mso-style-name:普通表格;
        mso-tstyle-rowband-size:0;
        mso-tstyle-colband-size:0;
        mso-style-noshow:yes;
        mso-style-parent:"";
        mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
        mso-para-margin:0cm;
        mso-para-margin-bottom:.0001pt;
        mso-pagination:widow-orphan;
        font-size:10.0pt;
        font-family:"Times New Roman";
        mso-fareast-font-family:"Times New Roman";
        mso-ansi-language:#0400;
        mso-fareast-language:#0400;
        mso-bidi-language:#0400;}
</style>
<![endif]-->

<p class="MsoNormal"><span lang="EN-US">Hi Mark,</span></p>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>

<p class="MsoNormal"><a name="OLE_LINK19"><span lang="EN-US">Your analyses are
quite reasonable. The low-temperature replicas are indeed doing much more work
than the high-temperature replicas. As you said, the lowest temperature replica
in the 24-replica should take an amount of time comparable to that of the
lowest in the 42-replica. So for my case, the load imbalance across replicas is
only partly to blame. Now I can exclude factors from the REMD parameters
themselves. I will ask the system admin for possible explanations.</span></a></p>

<p class="MsoNormal"><span style=""><span lang="EN-US">&nbsp;</span></span></p>

<p class="MsoNormal"><span style=""><span lang="EN-US">May I
ask when you do REMD with NVT ensemble, is it right that all your replicas are
running with the same Volume as the lowest-temperature replica? Or do you
equilibrate each replica with NPT ensemble, then NVT ensemble, and then feed
the equilibrated structures to NVT REMD simulations?</span></span></p>

<p class="MsoNormal"><span style=""><span lang="EN-US">&nbsp;</span></span></p>

<p class="MsoNormal"><span style=""><span lang="EN-US">Thank
you for all your helpful suggestions!</span></span></p>

<span style=""></span>

<p class="MsoNormal"><span lang="EN-US">&nbsp;</span></p>

<p class="MsoNormal"><span lang="EN-US">Qiong</span></p>

<br><br><br>
              Hi Mark,<br>
              <br>
              Many thanks for your fast response!<br>
              <br>
              
              <p class="yiv1054013002MsoNormal"><i style=""><span lang="EN-US">What's
                    the network hardware? Can other machine load
                    influence your network
                    performance?</span></i></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">The supercomputer
                  system is based on the
                  Cray Gemini interconnect technology. I suppose this is
                  a fast network hardware...</span></p>
              <p class="yiv1054013002MsoNormal"><br>
                <span lang="EN-US"></span></p>
              <p class="yiv1054013002MsoNormal"><i style=""><span lang="EN-US">Are
                    the systems in the NVT ensemble? Use diff to check
                    the .mdp files differ only
                    how you think they do.</span></i></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">The systems are in
                  NPT ensemble. I saw some
                  discussions on the mailing list that NPT ensemble is
                  superior to NVT ensemble
                  for REMD. And the .mdp files differ only in the
                  temperature.</span></p>
            
          
        
      
    
    <br>
    Maybe so, but under NPT the density varies with T, and so with
    replica. This means the size of neighbour lists varies, and the cost
    of the computation (PME or not) varies. The generalized ensemble is
    limited by the progress of the slowest replica. If using PME, in
    theory, you can juggle the contribution of the various terms to
    balance the computation load across the replicas, but this is not
    easy to do.<br>
    <span lang="EN-US">&nbsp;</span>
    <blockquote type="cite">
      <table border="0" cellpadding="0" cellspacing="0">
        <tbody>
          <tr>
            <td style="font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; font-size: inherit; line-height: inherit; font-size-adjust: inherit; font-stretch: inherit;" valign="top">
              <p class="yiv1054013002MsoNormal"><i style=""><span lang="EN-US">What
                    are the values of nstlist and<a rel="nofollow" name="OLE_LINK12">
                      nstcalcenergy</a>?</span></i></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">Previously,
                  nstlist=5</span>, <a rel="nofollow" style="background-color: rgb(255, 255, 255);" name="OLE_LINK15"><span style=""><span style=""><span style="background-image: none; background-repeat: repeat; background-attachment: scroll; background-position: 0% 0%;" lang="EN-US">nstcalcenergy</span></span></span></a><span style="background-color: rgb(255, 255, 255);"><span style="background-image: none; background-repeat: repeat; background-attachment: scroll; background-position: 0% 0%;" lang="EN-US">=1</span></span></p>
              <span style=""></span>
              <p class="yiv1054013002MsoNormal"><span style="font-size: 11pt; font-family: NimbusRomNo9L-Regu;" lang="EN-US">Thank
                  you for
                  pointing this out. I checked the manual again that
                  this option affects the
                  performance in parallel simulations because
                  calculating energies requires global
                  communication between all processes. So I have set
                  this option to -1 this time.
                  This should be one reason for the low parallel
                  efficiency.</span></p>
              <p class="yiv1054013002MsoNormal"><span style="font-size: 11pt; font-family: NimbusRomNo9L-Regu;" lang="EN-US">And
                  after I
                  changed </span><span style="background: rgb(255, 255, 255) none repeat scroll 0% 0%; -moz-background-clip: border; -moz-background-origin: padding; -moz-background-inline-policy: continuous;" lang="EN-US">nstcalcenergy=</span><span lang="EN-US">-1,
                  I</span><span style="font-size: 11pt; font-family: NimbusRomNo9L-Regu;" lang="EN-US"> found there was a
                  3% improvement on the efficiency compared with those
                  when
                </span><span lang="EN-US">nstcalcenergy=1.</span></p>
            </td>
          </tr>
        </tbody>
      </table>
    </blockquote>
    <br>
    Yep. nstpcouple and nsttcouple also influence this.<br>
    <span lang="EN-US">&nbsp;<br>
    </span>
    <blockquote type="cite">
      <table border="0" cellpadding="0" cellspacing="0">
        <tbody>
          <tr>
            <td style="font-family: inherit; font-style: inherit; font-variant: inherit; font-weight: inherit; font-size: inherit; line-height: inherit; font-size-adjust: inherit; font-stretch: inherit;" valign="top">
              <p style="font-style: italic;" class="yiv1054013002MsoNormal"><span lang="EN-US">Take a look at the execution time
                  breakdown
                  at the end of the .log files, and do so for more than
                  one replica. With the
                  current implementation, every simulation has to
                  synchronize and communicate
                  every handful of steps, which means that large scale
                  parallelism won't work
                  efficiently unless you have<a rel="nofollow" name="OLE_LINK17"> fast network hardware</a> that
                  is dedicated to your job. This effect shows up in the
                  "Rest" row of
                  the time breakdown. With <a rel="nofollow" name="OLE_LINK14"></a><a rel="nofollow" name="OLE_LINK13"><span style="">Infiniband</span></a>,
                  I'd expect you should
                  only be losing about 10% of the run time total. The
                  30-fold loss you have upon
                  going from 24-&gt;42 replicas keeping 4 CPUs/replica
                  suggests some other
                  contribution, however.</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">&nbsp;</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">I checked the time
                  breakdown in the log
                  files for short REMD simulations. For the REMD
                  simulaiton with 168 cores for 42
                  replicas, as you see below, the “Rest” makes up as
                  surprisingly high as <b style=""><u>96.6%</u></b> of
                  the time for one of the
                  replicas. This parameter is almost the same level for
                  the other replicas. For
                  the REMD simulation with 96 cores for 24 replicas, the
                  “Rest” takes up about
                  24%. I was also aware of your post: </span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><a rel="nofollow" target="_blank" href="http://www.mail-archive.com/gmx-users@gromacs.org/msg37507.html">http://www.mail-archive.com/gmx-users@gromacs.org/msg37507.html</a></span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">As you suggested
                  such big loss should be
                  ascribed to other factors. Do you think it is the
                  network hardware to blame or
                  there are other reasons please? Any suggestion would
                  be greatly appreciated<br>
                </span></p>
            </td>
          </tr>
        </tbody>
      </table>
    </blockquote>
    <br>
    I expect the load imbalance across replicas is partly to blame. Look
    at the sum of Force + PME mesh (in seconds) across the generalized
    ensemble. That's where the simulation work is all done, and I expect
    your low-temperature replicas are doing much more work than your
    high-temperature replicas. Unfortunately 4.5.3 doesn't allow the
    user to know enough detail here. Future versions of GROMACS will -
    work in progress.<br>
    <br>
    Strictly, though, your rate-limiting lowest temperature replica in
    the 24-replica regime should take an amount of time comparable to
    that of the lowest in the 42-replica regime (22K difference is not
    that significant) - and similar to a run other than as part of a
    replica-exchange simulation. Your reported data is not consistent
    with that, so I think your jobs are also experiencing differing
    degrees of network or filesystem contention at different times. Your
    sysadmins can comment on that.<br>
    <br>
    Mark<br>
    <br>
    
      
        
          
            
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">&nbsp;</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">Computing:<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>Nodes<span style="">&nbsp;&nbsp;&nbsp;&nbsp; </span>Number<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>G-Cycles<span style="">&nbsp;&nbsp;&nbsp; </span>Seconds<span style="">&nbsp;&nbsp;&nbsp;&nbsp; </span>%</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">-----------------------------------------------------------------------</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Domain
                  decomp.<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>442<span style="">&nbsp;&nbsp;&nbsp;
                  </span><span style="">&nbsp;&nbsp;&nbsp;&nbsp;</span>2.604<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>1.2<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.0</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>DD
comm.
                  load<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>6<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>0.001<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>0.0<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.0</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Comm.
                  coord.<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>2201<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>1.145<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>0.5<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.0</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Neighbor
                  search<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>442<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>14.964<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>7.1<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.2</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Force<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span><span style="">&nbsp;&nbsp;&nbsp;&nbsp;</span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>2201<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>175.303<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>83.5<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>2.0</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Wait
+
                  Comm. F<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>2201<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>1.245<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>0.6<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.0</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>PME
                  mesh<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>2201<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>30.314<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>14.4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.3</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Write
                  traj.<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>11<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>17.346<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>8.3<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.2</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Update<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>2201<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>2.004<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>1.0<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.0</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Constraints<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>2201<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>26.593<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>12.7<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.3</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Comm.
                  energies<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>442<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>28.722<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>13.7<span style="">&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>0.3</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Rest<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>8426.029<span style="">&nbsp;&nbsp;&nbsp;&nbsp; </span>4012.4<span style="">&nbsp;&nbsp;&nbsp;
                  </span>96.6</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US">-----------------------------------------------------------------------</span></p>
              <p class="yiv1054013002MsoNormal"><span lang="EN-US"><span style="">&nbsp;</span>Total<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </span>4<span style="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
                  </span>8726.270<span style="">&nbsp;&nbsp;&nbsp;&nbsp; </span>4155.4<span style="">&nbsp;&nbsp;
                  </span>100.0</span></p>
              <br>
              <br>
              Qiong<br>
              </td></tr></table><br>