<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /><br><p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Hi, Mark,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Thanks for the
reply! <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">It seemed that I
got something messed up. At the beginning, I used ‘constraints = all-bonds’ and
</span><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:0pt">‘domain decomposition’.
W</span><span lang="EN-US" style="font-size:12.0pt">hen </span><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:0pt">the </span><span lang="EN-US" style="font-size:12.0pt">simulation scale to more than 2 processes, an error
like below will occur: <o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;mso-layout-grid-align:
none;text-autospace:none"><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:
0pt">####################<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Fatal error: There
is no domain decomposition for 6 nodes that is compatible with the given box
and a minimum cell size of 2.06375 nm<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Change the number
of nodes or mdrun option -rcon or -dds or your LINCS settings<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">Look in the log
file for details on the domain decomposition<o:p></o:p></span></p>
<p class="MsoNormal" align="left" style="text-align:left;mso-layout-grid-align:
none;text-autospace:none"><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:
0pt">####################<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt">I refer to the
manual and found no answer. Then I turned to use ‘</span><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:0pt">particle decomposition’, tried
all kind of method, including change mpich to lammpi, change Gromacs from V4.05
to V4.07,adjusting the mdp file (e.g. ‘</span><span lang="EN-US" style="font-size:12.0pt">constraints = hbonds</span><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:0pt">’ or no PME), and none of these
take effect! I thought I have tried ‘</span><span lang="EN-US" style="font-size:
12.0pt">constraints = hbonds</span><span lang="EN-US" style="font-size:12.0pt;
mso-font-kerning:0pt">’ with ‘domain decomposition’, at least with lammpi. <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:
0pt">However, when I tried ‘</span><span lang="EN-US" style="font-size:12.0pt">constraints
= hbonds</span><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:0pt">’
</span><span lang="EN-US" style="font-size:12.0pt">and </span><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:0pt">‘domain decomposition’ under
mpich today, it scaled to more than 2 processes well! And now it also scaled
well under lammpi using ‘</span><span lang="EN-US" style="font-size:12.0pt">constraints
= hbonds</span><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:0pt">’
</span><span lang="EN-US" style="font-size:12.0pt">and </span><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:0pt">‘domain decomposition’!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:
0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:
0pt">So, it seemed the key is ‘</span><span lang="EN-US" style="font-size:12.0pt">constraints
= hbonds</span><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:0pt">’
for ‘domain decomposition’.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:
0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:
0pt">Of course, the simulation still crashed when using </span><span lang="EN-US" style="font-size:12.0pt">‘</span><span lang="EN-US" style="font-size:
12.0pt;mso-font-kerning:0pt">particle decomposition’ with ‘</span><span lang="EN-US" style="font-size:12.0pt">constraints = hbonds or all-bonds’, and I
don’t know why.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:12.0pt;mso-font-kerning:
0pt">I use double precision version and NTP ensemble to perform a PCA!<o:p></o:p></span></p><br><div id="addMailSign"><span> </span></div><div id="mailcontent"><br><blockquote dir="ltr" style="border-left:2px solid #000000;margin-left:10px;margin-right:0px;padding-left:5px;padding-right:0px;font-family:宋体;font-style:normal;font-variant:normal;font-weight:normal;font-size: 9pt;line-height:normal;font-size-adjust:none;font-stretch:normal;"><div style="font-family: 宋体; font-style: normal; font-variant: normal; font-weight: normal; font-size: 9pt; line-height: normal; font-size-adjust: none; font-stretch: normal;">----- Original Message -----</div><p>From: xhomes@sohu.com<br>Date: Tuesday, June 1, 2010 11:53<br>Subject: [gmx-users] “Fatal error in PMPI_Bcast: Other MPI error, …..” occurs when using the ‘particle decomposition’ option.<br>To: gmx-users <gmx-users@gromacs.org></p><div class="sohu_quote" style=""><br><br>> Hi, everyone of gmx-users,<br>> <br>> I met a problem when I use the ‘particle decomposition’ option <br>> in a NTP MD simulation of Engrailed Homeodomain (En) in CL- <br>> neutralized water box. It just crashed with an error “Fatal <br>> error in PMPI_Bcast: Other MPI error, error stack: …..”. <br>> However, I’ve tried the ‘domain decomposition’ and everything is <br>> ok! I use the Gromacs 4.05 and 4.07, the MPI lib is mpich2-<br>> 1.2.1p1. The system box size is 5.386(nm)3. The MDP file list as <br>> below:<br>> ########################################################<br>> title = En<br>> ;cpp = /lib/cpp<br>> ;include = -I../top<br>> define = <br>> integrator = md<br>> dt = 0.002<br>> nsteps = 3000000<br>> nstxout = 500<br>> nstvout = 500<br>> nstlog = 250<br>> nstenergy = 250<br>> nstxtcout = 500<br>> comm-<br>> mode = Linear<br>> nstcomm = 1<br>> <br>> ;xtc_grps = Protein<br>> energygrps = protein non-protein<br>> <br>> nstlist = 10<br>> ns_type = grid<br>> pbc = xyz default xyz<br>> ;periodic_molecules = <br>> yes default no<br>> rlist = 1.0<br>> <br>> coulombtype = PME<br>> rcoulomb = 1.0<br>> vdwtype = Cut-off<br>> rvdw = 1.4<br>> fourierspacing = 0.12<br>> fourier_nx = 0<br>> fourier_ny = 0<br>> fourier_nz = 0<br>> pme_order = 4<br>> ewald_rtol = 1e-5<br>> optimize_fft = yes<br>> <br>> tcoupl = v-rescale<br>> tc_grps = protein non-protein<br>> tau_t = 0.1 0.1<br>> ref_t = 298 298<br>> Pcoupl = Parrinello-Rahman<br>> pcoupltype = isotropic<br>> tau_p = 0.5<br>> compressibility = 4.5e-5<br>> ref_p = 1.0<br>> <br>> gen_vel = yes<br>> gen_temp = 298<br>> gen_seed = 173529<br>> <br>> constraints = hbonds<br>> lincs_order = 10<br>> ########################################################<br>> <br>> When I conduct MD using “nohup mpiexec -np 2 mdrun_dmpi -s <br>> 11_Trun.tpr -g 12_NTPmd.log -o 12_NTPmd.trr -c 12_NTPmd.pdb -e <br>> 12_NTPmd_ener.edr -cpo 12_NTPstate.cpt &”, everything is OK.<br>> <br>> Since the system doesn’t support more than 2 processes under <br>> ‘domain decomposition’ option, it took me about 30 days to <br>> calculate a 6ns trajectory. Then I decide to use the ‘particle </div><br><br>Why no more than 2? What GROMACS version? Why are you using double precision with temperature coupling?<br><br>MPICH has known issues. Use OpenMPI.<div class="sohu_quote" style=""><br><br>> decomposition’ option. The command line is “nohup mpiexec -np 6 <br>> mdrun_dmpi -pd -s 11_Trun.tpr -g 12_NTPmd.log -o 12_NTPmd.trr -c <br>> 12_NTPmd.pdb -e 12_NTPmd_ener.edr -cpo 12_NTPstate.cpt &”. And I <br>> got the crash in the nohup file like below:<br>> ####################<br>> Fatal error in PMPI_Bcast: Other MPI error, error stack:<br>> PMPI_Bcast(1302)......................: MPI_Bcast(buf=0x8fedeb0, <br>> count=60720, MPI_BYTE, root=0, MPI_COMM_WORLD) failed<br>> MPIR_Bcast(998).......................: <br><wbr>> MPIR_Bcast_scatter_ring_al<wbr>lgather(842): <br>> MPIR_Bcast_binomial(187)..............: <br>> MPIC_Send(41).........................: <br>> MPIC_Wait(513)........................: <br>> MPIDI_CH3I_Progress(150)..............: <br><wbr>> MPID_nem_mpich2_blocking_r<wbr>ecv(948)....: <br>> MPID_nem_tcp_connpoll(1720)...........: <br>> state_commrdy_handler(1561)...........: <br>> MPID_nem_tcp_send_queued(127).........: writev to socket failed -<br>> Bad address<br>> rank 0 in job 25 cluster.cn_52655 caused <br>> collective abort of all ranks<br>> exit status of rank 0: killed by signal 9<br>> ####################<br>> <br>> And the ends of the log file list as below:<br>> ####################<br>> ……..<br>> ……..<br>> ……..<br>> ……..<br>> <br>> bQMMM = FALSE<br>> <br>> QMconstraints = 0<br>> QMMMscheme = 0<br>> <br>> scalefactor = 1<br>> qm_opts:<br>> <br>> ngQM = 0<br>> ####################<br>> <br>> I’ve search the gmx-users mail list and tried to adjust the md <br>> parameters, and no solution was found. The "mpiexec -np x" <br>> option doesn't work except when x=1. I did found that when the <br>> whole En protein is constrained using position restraints <br>> (define = -DPOSRES), the ‘particle decomposition’ option works. <br>> However this is not the kind of MD I want to conduct.<br>> <br>> Could anyone help me about this problem? And I also want to know <br>> how can I accelerate this kind of MD (long time simulation of <br>> small system) using Gromacs? Thinks a lot!<br>> <br>> (Further information about the simulated system: The system has <br>> one En protein (54 residues, 629 atoms), total 4848 spce waters, <br>> and 7 Cl- used to neutralize the system. The system has been <br>> minimized first. A 20ps MD is also performed for the waters and <br>> ions before EM.)</div><br><br>This should be bread-and-butter with either decomposition up to at least 16 processors, for a correctly compiled GROMACS with a useful MPI library.<br><br>Mark<br>-- <br><p></p></blockquote></div>