Hi All,<br><br>I'm trying to d.lzm gromacs benchmarks with 64 node machine, but dynamic load balancing performance is very low. <br><br>Any <span class="il">suggestion</span> will be of great help.<br><br>Thanks.<br>
<br>Deniz KARASU<br><br>Log file opened on Sat Feb 13 17:23:37 2010<br>Host: <a href="http://d077.uybhm.itu.edu.tr">d077.uybhm.itu.edu.tr</a> pid: 20157 nodeid: 0 nnodes: 64<br>The Gromacs distribution was built Thu Sep 10 11:45:26 EEST 2009 by<br>
<a href="mailto:mds.fatma@lnode1.uybhm.itu.edu.tr">mds.fatma@lnode1.uybhm.itu.edu.tr</a> (Linux 2.6.18-53.1.14.el5_lustre.1.6.5.1smp x86_64)<br><br><br> :-) G R O M A C S (-:<br><br> Good ROcking Metal Altar for Chronical Sinners<br>
<br> :-) VERSION 4.0.5 (-:<br><br><br> Written by David van der Spoel, Erik Lindahl, Berk Hess, and others.<br> Copyright (c) 1991-2000, University of Groningen, The Netherlands.<br>
Copyright (c) 2001-2008, The GROMACS development team,<br> check out <a href="http://www.gromacs.org">http://www.gromacs.org</a> for more information.<br><br> This program is free software; you can redistribute it and/or<br>
modify it under the terms of the GNU General Public License<br> as published by the Free Software Foundation; either version 2<br> of the License, or (at your option) any later version.<br><br>
:-) /AKDENIZ/HOME005/users/mds.fatma/rs/software/bin/mdrun (-:<br><br><br>++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++<br>B. Hess and C. Kutzner and D. van der Spoel and E. Lindahl<br>GROMACS 4: Algorithms for highly efficient, load-balanced, and scalable<br>
molecular simulation<br>J. Chem. Theory Comput. 4 (2008) pp. 435-447<br>-------- -------- --- Thank You --- -------- --------<br><br><br>++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++<br>D. van der Spoel, E. Lindahl, B. Hess, G. Groenhof, A. E. Mark and H. J. C.<br>
Berendsen<br>GROMACS: Fast, Flexible and Free<br>J. Comp. Chem. 26 (2005) pp. 1701-1719<br>-------- -------- --- Thank You --- -------- --------<br><br><br>++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++<br>E. Lindahl and B. Hess and D. van der Spoel<br>
GROMACS 3.0: A package for molecular simulation and trajectory analysis<br>J. Mol. Mod. 7 (2001) pp. 306-317<br>-------- -------- --- Thank You --- -------- --------<br><br><br>++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++<br>
H. J. C. Berendsen, D. van der Spoel and R. van Drunen<br>GROMACS: A message-passing parallel molecular dynamics implementation<br>Comp. Phys. Comm. 91 (1995) pp. 43-56<br>-------- -------- --- Thank You --- -------- --------<br>
<br>parameters of the run:<br> integrator = md<br> nsteps = 5000<br> init_step = 0<br> ns_type = Grid<br> nstlist = 5<br> ndelta = 2<br>
nstcomm = 1<br> comm_mode = Linear<br> nstlog = 0<br> nstxout = 0<br> nstvout = 0<br> nstfout = 0<br> nstenergy = 0<br>
nstxtcout = 0<br> init_t = 0<br> delta_t = 0.004<br> xtcprec = 1000<br> nkx = 0<br> nky = 0<br> nkz = 0<br>
pme_order = 4<br> ewald_rtol = 1e-05<br> ewald_geometry = 0<br> epsilon_surface = 0<br> optimize_fft = FALSE<br> ePBC = xyz<br> bPeriodicMols = FALSE<br>
bContinuation = FALSE<br> bShakeSOR = FALSE<br> etc = Berendsen<br> epc = No<br> epctype = Isotropic<br> tau_p = 1<br> ref_p (3x3):<br>
ref_p[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}<br> ref_p[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}<br> ref_p[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}<br> compress (3x3):<br> compress[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}<br>
compress[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}<br> compress[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}<br> refcoord_scaling = No<br> posres_com (3):<br> posres_com[0]= 0.00000e+00<br>
posres_com[1]= 0.00000e+00<br> posres_com[2]= 0.00000e+00<br> posres_comB (3):<br> posres_comB[0]= 0.00000e+00<br> posres_comB[1]= 0.00000e+00<br> posres_comB[2]= 0.00000e+00<br> andersen_seed = 815131<br>
rlist = 0.9<br> rtpi = 0.05<br> coulombtype = Cut-off<br> rcoulomb_switch = 0<br> rcoulomb = 1.4<br> vdwtype = Cut-off<br> rvdw_switch = 0<br>
rvdw = 1.4<br> epsilon_r = 1<br> epsilon_rf = 1<br> tabext = 1<br> implicit_solvent = No<br> gb_algorithm = Still<br> gb_epsilon_solvent = 80<br>
nstgbradii = 1<br> rgbradii = 2<br> gb_saltconc = 0<br> gb_obc_alpha = 1<br> gb_obc_beta = 0.8<br> gb_obc_gamma = 4.85<br> sa_surface_tension = 2.092<br>
DispCorr = No<br> free_energy = no<br> init_lambda = 0<br> sc_alpha = 0<br> sc_power = 0<br> sc_sigma = 0.3<br> delta_lambda = 0<br>
nwall = 0<br> wall_type = 9-3<br> wall_atomtype[0] = -1<br> wall_atomtype[1] = -1<br> wall_density[0] = 0<br> wall_density[1] = 0<br> wall_ewald_zfac = 3<br>
pull = no<br> disre = No<br> disre_weighting = Conservative<br> disre_mixed = FALSE<br> dr_fc = 1000<br> dr_tau = 0<br> nstdisreout = 100<br>
orires_fc = 0<br> orires_tau = 0<br> nstorireout = 100<br> dihre-fc = 1000<br> em_stepsize = 0.01<br> em_tol = 10<br> niter = 20<br>
fc_stepsize = 0<br> nstcgsteep = 1000<br> nbfgscorr = 10<br> ConstAlg = Lincs<br> shake_tol = 0.0001<br> lincs_order = 4<br> lincs_warnangle = 30<br>
lincs_iter = 1<br> bd_fric = 0<br> ld_seed = 1993<br> cos_accel = 0<br> deform (3x3):<br> deform[ 0]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}<br> deform[ 1]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}<br>
deform[ 2]={ 0.00000e+00, 0.00000e+00, 0.00000e+00}<br> userint1 = 0<br> userint2 = 0<br> userint3 = 0<br> userint4 = 0<br> userreal1 = 0<br>
userreal2 = 0<br> userreal3 = 0<br> userreal4 = 0<br>grpopts:<br> nrdf: 2636.83 23.9984 42933.2<br> ref_t: 300 300 300<br> tau_t: 0.1 0.1 0.1<br>
anneal: No No No<br>ann_npoints: 0 0 0<br> acc: 0 0 0<br> nfreeze: N N N<br> energygrp_flags[ 0]: 0<br>
efield-x:<br> n = 0<br> efield-xt:<br> n = 0<br> efield-y:<br> n = 0<br> efield-yt:<br> n = 0<br> efield-z:<br> n = 0<br> efield-zt:<br> n = 0<br> bQMMM = FALSE<br>
QMconstraints = 0<br> QMMMscheme = 0<br> scalefactor = 1<br>qm_opts:<br> ngQM = 0<br><br>Initializing Domain Decomposition on 64 nodes<br>Dynamic load balancing: auto<br>
Will sort the charge groups at every domain (re)decomposition<br>Initial maximum inter charge-group distances:<br> two-body bonded interactions: 0.571 nm, LJ-14, atoms 439 442<br> multi-body bonded interactions: 0.571 nm, Proper Dih., atoms 439 442<br>
Minimum cell size due to bonded interactions: 0.628 nm<br>Maximum distance for 5 constraints, at 120 deg. angles, all-trans: 0.825 nm<br>Estimated maximum distance required for P-LINCS: 0.825 nm<br>This distance will limit the DD cell size, you can override this with -rcon<br>
Scaling the initial minimum size with 1/0.8 (option -dds) = 1.25<br>Optimizing the DD grid for 64 cells with a minimum initial size of 1.031 nm<br>The maximum allowed number of cells is: X 5 Y 5 Z 4<br>Domain decomposition grid 4 x 4 x 4, separate PME nodes 0<br>
Domain decomposition nodeid 0, coordinates 0 0 0<br><br>Using two step summing over 11 groups of on average 5.8 processes<br><br>Table routines are used for coulomb: FALSE<br>Table routines are used for vdw: FALSE<br>
Cut-off's: NS: 0.9 Coulomb: 1.4 LJ: 1.4<br>System total charge: 0.000<br>Generated table with 1200 data points for 1-4 COUL.<br>Tabscale = 500 points/nm<br>Generated table with 1200 data points for 1-4 LJ6.<br>Tabscale = 500 points/nm<br>
Generated table with 1200 data points for 1-4 LJ12.<br>Tabscale = 500 points/nm<br><br>Enabling SPC water optimization for 7156 molecules.<br><br>Configuring nonbonded kernels...<br>Testing x86_64 SSE support... present.<br>
<br><br>Removing pbc first time<br><br>Initializing Parallel LINear Constraint Solver<br><br>++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++<br>B. Hess<br>P-LINCS: A Parallel Linear Constraint Solver for molecular simulation<br>
J. Chem. Theory Comput. 4 (2008) pp. 116-122<br>-------- -------- --- Thank You --- -------- --------<br><br>The number of constraints is 1407<br>There are inter charge-group constraints,<br>will communicate selected coordinates each lincs iteration<br>
117 constraints are involved in constraint triangles,<br>will apply an additional matrix expansion of order 4 for couplings<br>between constraints inside triangles<br><br>++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++<br>
S. Miyamoto and P. A. Kollman<br>SETTLE: An Analytical Version of the SHAKE and RATTLE Algorithms for Rigid<br>Water Models<br>J. Comp. Chem. 13 (1992) pp. 952-962<br>-------- -------- --- Thank You --- -------- --------<br>
<br><br>Linking all bonded interactions to atoms<br>There are 379 inter charge-group virtual sites,<br>will an extra communication step for selected coordinates and forces<br><br>The initial number of communication pulses is: X 1 Y 1 Z 2<br>
The initial domain decomposition cell size is: X 1.43 nm Y 1.43 nm Z 1.24 nm<br><br>The maximum allowed distance for charge groups involved in interactions is:<br> non-bonded interactions 1.400 nm<br>
two-body bonded interactions (-rdd) 1.400 nm<br> multi-body bonded interactions (-rdd) 1.239 nm<br> virtual site constructions (-rcon) 1.239 nm<br> atoms separated by up to 5 constraints (-rcon) 1.239 nm<br>
<br>When dynamic load balancing gets turned on, these settings will change to:<br>The maximum number of communication pulses is: X 2 Y 2 Z 2<br>The minimum size for domain decomposition cells is 0.905 nm<br>The requested allowed shrink of DD cells (option -dds) is: 0.80<br>
The allowed shrink of domain decomposition cells is: X 0.63 Y 0.63 Z 0.73<br>The maximum allowed distance for charge groups involved in interactions is:<br> non-bonded interactions 1.400 nm<br> two-body bonded interactions (-rdd) 1.400 nm<br>
multi-body bonded interactions (-rdd) 0.905 nm<br> virtual site constructions (-rcon) 0.905 nm<br> atoms separated by up to 5 constraints (-rcon) 0.905 nm<br><br><br>Making 3D domain decomposition grid 4 x 4 x 4, home cell index 0 0 0<br>
<br>Center of mass motion removal mode is Linear<br>We have the following groups for center of mass motion removal:<br> 0: rest<br><br>++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++<br>H. J. C. Berendsen, J. P. M. Postma, A. DiNola and J. R. Haak<br>
Molecular dynamics with coupling to an external bath<br>J. Chem. Phys. 81 (1984) pp. 3684-3690<br>-------- -------- --- Thank You --- -------- --------<br><br>There are: 22824 Atoms<br>There are: 383 VSites<br>Charge group distribution at step 0: 119 121 124 123 127 118 128 126 117 112 124 118 126 120 130 120 121 136 124 123 118 117 125 130 122 129 127 123 125 125 113 119 124 127 124 124 123 119 128 129 123 128 126 121 119 124 118 129 131 118 119 119 122 128 129 124 121 123 125 120 120 120 116 131<br>
Grid: 6 x 6 x 5 cells<br><br>Constraining the starting coordinates (step 0)<br><br>Constraining the coordinates at t0-dt (step 0)<br>RMS relative constraint deviation after constraining: 3.57e-05<br>Initial temperature: 311.264 K<br>
<br>Started mdrun on node 0 Sat Feb 13 17:23:39 2010<br><br> Step Time Lambda<br> 0 0.00000 0.00000<br><br> Energies (kJ/mol)<br> G96Angle Proper Dih. Improper Dih. LJ-14 Coulomb-14<br>
2.20938e+03 1.06206e+03 5.21012e+02 5.34001e+02 1.67617e+04<br> LJ (SR) LJ (LR) Coulomb (SR) Coulomb (LR) Potential<br> 4.37552e+04 -1.85437e+03 -3.77685e+05 -2.78734e+03 -3.17483e+05<br>
Kinetic En. Total Energy Temperature Pressure (bar) Cons. rmsd ()<br> 5.90556e+04 -2.58428e+05 3.11564e+02 1.98804e+02 3.56693e-05<br><br>DD step 4 load imb.: force 262.0%<br><br>At step 5 the performance loss due to force load imbalance is 19.1 %<br>
<br>NOTE: Turning on dynamic load balancing<br><br>DD load balancing is limited by minimum cell size in dimension Y Z<br>DD step 4999 vol min/aver 0.453! load imb.: force 42.2%<br><br> Step Time Lambda<br>
5000 20.00000 0.00000<br><br>Writing checkpoint, step 5000 at Sat Feb 13 17:23:57 2010<br><br> Energies (kJ/mol)<br> G96Angle Proper Dih. Improper Dih. LJ-14 Coulomb-14<br>
2.18559e+03 1.08758e+03 5.08072e+02 5.73181e+02 1.67070e+04<br> LJ (SR) LJ (LR) Coulomb (SR) Coulomb (LR) Potential<br> 4.39756e+04 -1.84574e+03 -3.78315e+05 -9.08535e+03 -3.24209e+05<br>
Kinetic En. Total Energy Temperature Pressure (bar) Cons. rmsd ()<br> 5.81564e+04 -2.66053e+05 3.06820e+02 -3.44878e+02 9.68320e-05<br><br> <====== ############### ==><br> <==== A V E R A G E S ====><br>
<== ############### ======><br><br> Energies (kJ/mol)<br> G96Angle Proper Dih. Improper Dih. LJ-14 Coulomb-14<br> 2.13937e+03 1.08823e+03 4.88467e+02 5.56312e+02 1.66991e+04<br>
LJ (SR) LJ (LR) Coulomb (SR) Coulomb (LR) Potential<br> 4.37569e+04 -1.85173e+03 -3.78660e+05 -7.85919e+03 -3.23642e+05<br> Kinetic En. Total Energy Temperature Pressure (bar) Cons. rmsd ()<br>
5.84560e+04 -2.65186e+05 3.08400e+02 -2.56636e+02 0.00000e+00<br><br> Total Virial (kJ/mol)<br> 2.14238e+04 1.20840e+02 1.11414e+02<br> 1.21134e+02 2.14442e+04 1.16878e+01<br> 1.11918e+02 1.23263e+01 2.12292e+04<br>
<br> Pressure (bar)<br> -2.67468e+02 -2.17401e+01 -1.48656e+01<br> -2.17802e+01 -2.66730e+02 1.77342e-01<br> -1.49344e+01 9.02074e-02 -2.35709e+02<br><br> Total Dipole (Debye)<br> -3.97323e+02 -3.59815e+02 -1.52774e+02<br>
<br> T-Protein T-CL- T-SOL<br> 2.99534e+02 3.00276e+02 3.08949e+02<br><br> <====== ############################### ==><br> <==== R M S - F L U C T U A T I O N S ====><br>
<== ############################### ======><br><br> Energies (kJ/mol)<br> G96Angle Proper Dih. Improper Dih. LJ-14 Coulomb-14<br> 6.39796e+01 4.10873e+01 2.95910e+01 3.76420e+01 4.88986e+01<br>
LJ (SR) LJ (LR) Coulomb (SR) Coulomb (LR) Potential<br> 5.84609e+02 2.13849e+00 1.10640e+03 1.67778e+03 1.10444e+03<br> Kinetic En. Total Energy Temperature Pressure (bar) Cons. rmsd ()<br>
3.05395e+02 1.10173e+03 1.61119e+00 1.60301e+02 0.00000e+00<br><br> Total Virial (kJ/mol)<br> 1.65615e+03 1.02322e+03 1.00778e+03<br> 1.02179e+03 1.66559e+03 1.04738e+03<br> 1.00766e+03 1.04676e+03 1.69082e+03<br>
<br> Pressure (bar)<br> 2.28103e+02 1.41246e+02 1.39669e+02<br> 1.41035e+02 2.27116e+02 1.45691e+02<br> 1.39680e+02 1.45604e+02 2.31456e+02<br><br> Total Dipole (Debye)<br> 3.19197e+02 1.87684e+02 1.24709e+02<br>
<br> T-Protein T-CL- T-SOL<br> 5.84167e+00 7.10486e+01 1.65761e+00<br><br><br> M E G A - F L O P S A C C O U N T I N G<br><br> RF=Reaction-Field FE=Free Energy SCFE=Soft-Core/Free Energy<br>
T=Tabulated W3=SPC/TIP3p W4=TIP4p (single or pairs)<br> NF=No Forces<br><br> Computing: M-Number M-Flops % Flops<br>-----------------------------------------------------------------------<br>
LJ 480.457532 15855.099 1.9<br> Coulomb 688.452307 18588.212 2.3<br> Coulomb [W3] 66.644451 5331.556 0.6<br> Coulomb + LJ 362.642477 13780.414 1.7<br>
Coulomb + LJ [W3] 156.776518 14266.663 1.7<br> Coulomb + LJ [W3-W3] 2604.244668 638039.944 77.6<br> Outer nonbonded loop 930.259077 9302.591 1.1<br> 1,4 nonbonded interactions 15.573114 1401.580 0.2<br>
NS-Pairs 3507.987455 73667.737 9.0<br> Reset In Box 7.889882 23.670 0.0<br> CG-CoM 23.253414 69.760 0.0<br> Angles 11.487297 1929.866 0.2<br>
Propers 4.330866 991.768 0.1<br> Impropers 2.730546 567.954 0.1<br> Virial 130.461087 2348.300 0.3<br> Update 116.058207 3597.804 0.4<br>
Stop-CM 116.058207 1160.582 0.1<br> Calc-Ekin 116.081414 3134.198 0.4<br> Lincs 15.846123 950.767 0.1<br> Lincs-Mat 266.595492 1066.382 0.1<br>
Constraint-V 139.069412 1112.555 0.1<br> Constraint-Vir 123.201821 2956.844 0.4<br> Settle 35.801468 11563.874 1.4<br> Virtual Site 3 0.140028 5.181 0.0<br>
Virtual Site 3fd 1.205241 114.498 0.0<br> Virtual Site 3fad 0.430086 75.695 0.0<br> Virtual Site 3out 0.140028 12.182 0.0<br>-----------------------------------------------------------------------<br>
Total 821915.676 100.0<br>-----------------------------------------------------------------------<br><br><br> D O M A I N D E C O M P O S I T I O N S T A T I S T I C S<br>
<br> av. #atoms communicated per step for force: 2 x 146675.8<br> av. #atoms communicated per step for vsites: 2 x 122.7<br> av. #atoms communicated per step for LINCS: 2 x 1993.1<br><br> Average load imbalance: 63.9 %<br>
Part of the total run time spent waiting due to load imbalance: 6.8 %<br> Steps where the load balancing was limited by -rdd, -rcon and/or -dds: X 0 % Y 19 % Z 19 %<br><br>NOTE: 6.8 % performance was lost due to load imbalance<br>
in the domain decomposition.<br><br><br> R E A L C Y C L E A N D T I M E A C C O U N T I N G<br><br> Computing: Nodes Number G-Cycles Seconds %<br>-----------------------------------------------------------------------<br>
Domain decomp. 64 1001 542.428 228.8 19.9<br> Vsite constr. 64 5001 13.336 5.6 0.5<br> Comm. coord. 64 5001 201.623 85.0 7.4<br> Neighbor search 64 1001 468.949 197.8 17.2<br>
Force 64 5001 286.647 120.9 10.5<br> Wait + Comm. F 64 5001 525.059 221.4 19.2<br> Vsite spread 64 5001 57.706 24.3 2.1<br> Write traj. 64 1 0.739 0.3 0.0<br>
Update 64 5001 17.965 7.6 0.7<br> Constraints 64 5001 168.205 70.9 6.2<br> Comm. energies 64 5001 432.254 182.3 15.8<br> Rest 64 16.536 7.0 0.6<br>
-----------------------------------------------------------------------<br> Total 64 2731.446 1152.0 100.0<br>-----------------------------------------------------------------------<br>
<br>NOTE: 16 % of the run time was spent communicating energies,<br> you might want to use the -nosum option of mdrun<br><br><br> Parallel run - timing based on wallclock.<br><br> NODE (s) Real (s) (%)<br>
Time: 18.000 18.000 100.0<br> (Mnbf/s) (GFlops) (ns/day) (hour/ns)<br>Performance: 1424.445 45.662 96.019 0.250<br>Finished mdrun on node 0 Sat Feb 13 17:23:57 2010<br><br>