<html>
<head>
<style>
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
</style>
</head>
<body class='hmmessage'>
<br><br>> Date: Fri, 31 Jul 2009 07:49:49 +0200<br>> From: spoel@xray.bmc.uu.se<br>> To: gmx-users@gromacs.org<br>> Subject: Re: [gmx-users] very strange domain composition statistics<br>> <br>> Mark Abraham wrote:<br>> > Jennifer Williams wrote:<br>> >> Hi ,<br>> >><br>> >> I am having some problems when running in parallel. Although my jobs <br>> >> run to completion I am getting some worrying domain decomposition <br>> >> statistics in particular the average load imbalance and the <br>> >> performance loss due to load imbalance see below:<br>> > <br>> > Please report your GROMACS version number. If it's not the latest <br>> > (4.0.5), then you should probably update and see if it's a problem that <br>> > may have been fixed between those releases. You might also try it <br>> > without your freeze groups, especially if they dominate the system.<br>> <br>> In addition, you do not specify how may processors you used, nor the <br>> division over processors that mdrun makes and not the expected <br>> performance either. From the numbers below it seems like you used 1 or 2 <br>> processors at most. The large number is definitely erroneous though.<br>> <br><br>For the cycle count table the number of processors seems to be 6.<br><br>It seems that the cycle counters have overflowed.<br>On what kind of architecture with what kind of compilers are you running this?<br><br>Berk<br><br>> > <br>> >> D O M A I N D E C O M P O S I T I O N S T A T I S T I C S<br>> >><br>> >> av. #atoms communicated per step for force: 2 x 1974.8<br>> >> av. #atoms communicated per step for LINCS: 2 x 15.2<br>> >><br>> >> Average load imbalance: 500.0 %<br>> >> Part of the total run time spent waiting due to load imbalance: <br>> >> 4246403072.0 %<br>> >> Steps where the load balancing was limited by -rdd, -rcon and/or <br>> >> -dds: X 9 %<br>> >><br>> >> NOTE: 4246403072.0 % performance was lost due to load imbalance<br>> >> in the domain decomposition.<br>> >><br>> >> R E A L C Y C L E A N D T I M E A C C O U N T I N G<br>> >><br>> >> Computing: Nodes Number G-Cycles Seconds %<br>> >> -----------------------------------------------------------------------<br>> >> Write traj. 6 1001 18443320139.164 42130.9 100.0<br>> >> Update 6 1000001 18442922984.491 42130.0 100.0<br>> >> Rest 6 9223372036.855 21069.4 50.0<br>> >> -----------------------------------------------------------------------<br>> >> Total 6 18446422611.669 42138.0 100.0<br>> >> -----------------------------------------------------------------------<br>> >><br>> >> NOTE: 305 % of the run time was spent communicating energies,<br>> >> you might want to use the -nosum option of mdrun<br>> >><br>> >> Parallel run - timing based on wallclock.<br>> >><br>> >> NODE (s) Real (s) (%)<br>> >> Time: 7023.000 7023.000 100.0<br>> >> 1h57:03<br>> >> (Mnbf/s) (GFlops) (ns/day) (hour/ns)<br>> >> Performance: 14.214 1.902 12.302 1.951<br>> >> Finished mdrun on node 0 Wed Jul 29 23:47:18 2009<br>> >><br>> >><br>> >><br>> >> Below is my .mdp file: I am using the PME but not having much of a <br>> >> feel for how to set the options under Spacing for the PME/PPPM FFT <br>> >> grid, I left these as the default values. Could this be where the <br>> >> trouble lies?<br>> >><br>> >> My cut-off cannot be larger than 0.9 as my unit cell is only 18.2A in <br>> >> one direction.<br>> >><br>> >> How do I choose values for PME/PPPM? Ie what kind of values to use for <br>> >> nx, ny and nz ?<br>> > <br>> > See manual section 3.17.5<br>> > <br>> >> I read that they should be divisible by npme to get the best <br>> >> performance. Is npme the pme_order in the .mdp file? If not where do I <br>> >> set this parameter?<br>> > <br>> > No, -npme is a command line parameter to mdrun. Roughly speaking, things <br>> > that have a material effect on the physics are specified in the .mdp <br>> > file, and things that either require external file(names) to be supplied <br>> > or which only affect the implementation of the physics are specified on <br>> > the command line.<br>> > <br>> > Mark<br>> > <br>> >> Much appreciated,<br>> >><br>> >> Jenny<br>> >><br>> >><br>> >><br>> >> ; VARIOUS PREPROCESSING OPTIONS<br>> >> ; Preprocessor information: use cpp syntax.<br>> >> ; e.g.: -I/home/joe/doe -I/home/mary/hoe<br>> >> include = -I../top<br>> >> ; e.g.: -DI_Want_Cookies -DMe_Too<br>> >> define =<br>> >><br>> >> ; RUN CONTROL PARAMETERS<br>> >> integrator = md<br>> >> ; Start time and timestep in ps<br>> >> tinit = 0<br>> >> dt = 0.001<br>> >> nsteps = 1000000<br>> >> ; For exact run continuation or redoing part of a run<br>> >> ; Part index is updated automatically on checkpointing (keeps files <br>> >> separate)<br>> >> simulation_part = 1<br>> >> init_step = 0<br>> >> ; mode for center of mass motion removal<br>> >> comm-mode = linear<br>> >> ; number of steps for center of mass motion removal<br>> >> nstcomm = 1<br>> >> ; group(s) for center of mass motion removal<br>> >> comm-grps =<br>> >><br>> >> ; LANGEVIN DYNAMICS OPTIONS<br>> >> ; Friction coefficient (amu/ps) and random seed<br>> >> bd-fric = 0<br>> >> ld-seed = 1993<br>> >><br>> >> ; ENERGY MINIMIZATION OPTIONS<br>> >> ; Force tolerance and initial step-size<br>> >> emtol =<br>> >> emstep =<br>> >> ; Max number of iterations in relax_shells<br>> >> niter =<br>> >> ; Step size (ps^2) for minimization of flexible constraints<br>> >> fcstep =<br>> >> ; Frequency of steepest descents steps when doing CG<br>> >> nstcgsteep =<br>> >> nbfgscorr =<br>> >><br>> >> ; TEST PARTICLE INSERTION OPTIONS<br>> >> rtpi =<br>> >><br>> >> ; OUTPUT CONTROL OPTIONS<br>> >> ; Output frequency for coords (x), velocities (v) and forces (f)<br>> >> nstxout = 1000<br>> >> nstvout = 1000<br>> >> nstfout = 0<br>> >> ; Output frequency for energies to log file and energy file<br>> >> nstlog = 1000<br>> >> nstenergy = 1000<br>> >> ; Output frequency and precision for xtc file<br>> >> nstxtcout = 1000<br>> >> xtc-precision = 1000<br>> >> ; This selects the subset of atoms for the xtc file. You can<br>> >> ; select multiple groups. By default all atoms will be written.<br>> >> xtc-grps =<br>> >> ; Selection of energy groups<br>> >> energygrps =<br>> >><br>> >> ; NEIGHBORSEARCHING PARAMETERS<br>> >> ; nblist update frequency<br>> >> nstlist =<br>> >> ; ns algorithm (simple or grid)<br>> >> ns_type = grid<br>> >> ; Periodic boundary conditions: xyz, no, xy<br>> >> pbc = xyz<br>> >> periodic_molecules = yes<br>> >> ; nblist cut-off<br>> >> rlist = 0.9<br>> >><br>> >> ; OPTIONS FOR ELECTROSTATICS AND VDW<br>> >> ; Method for doing electrostatics<br>> >> coulombtype = PME<br>> >> rcoulomb-switch = 0<br>> >> rcoulomb = 0.9<br>> >> ; Relative dielectric constant for the medium and the reaction field<br>> >> epsilon_r =<br>> >> epsilon_rf =<br>> >><br>> >> ; Method for doing Van der Waals<br>> >> vdw-type = Cut-off<br>> >> ; cut-off lengths<br>> >> rvdw-switch = 0<br>> >> rvdw = 0.9<br>> >> ; Apply long range dispersion corrections for Energy and Pressure<br>> >> DispCorr = No<br>> >> ; Extension of the potential lookup tables beyond the cut-off<br>> >> table-extension =<br>> >> ; Seperate tables between energy group pairs<br>> >> energygrp_table =<br>> >><br>> >><br>> >> ; Spacing for the PME/PPPM FFT grid<br>> >> fourierspacing = 0.12<br>> >> ; FFT grid size, when a value is 0 fourierspacing will be used<br>> >> fourier_nx = 0<br>> >> fourier_ny = 0<br>> >> fourier_nz = 0<br>> >> ; EWALD/PME/PPPM parameters<br>> >> pme_order =<br>> >> ewald_rtol = 1e-05<br>> >> ewald_geometry = 3d<br>> >> epsilon_surface = 0<br>> >> optimize_fft = yes<br>> >><br>> >><br>> >><br>> >> ; OPTIONS FOR WEAK COUPLING ALGORITHMS<br>> >> ; Temperature coupling<br>> >> tcoupl = nose-hoover<br>> >> ; Groups to couple separately<br>> >> tc-grps = System<br>> >> ; Time constant (ps) and reference temperature (K)<br>> >> tau_t = 0.1<br>> >> ref_t = 150<br>> >><br>> >> ; Pressure coupling<br>> >> Pcoupl = No<br>> >> Pcoupltype =<br>> >> ; Time constant (ps), compressibility (1/bar) and reference P (bar)<br>> >> tau-p =<br>> >> compressibility =<br>> >> ref-p =<br>> >> ; Scaling of reference coordinates, No, All or COM<br>> >> refcoord_scaling = no<br>> >> ; Random seed for Andersen thermostat<br>> >> andersen_seed =<br>> >><br>> >> ; GENERATE VELOCITIES FOR STARTUP RUN<br>> >> gen_vel = yes<br>> >> gen_temp = 150<br>> >> gen_seed = 173529<br>> >><br>> >> ; OPTIONS FOR BONDS<br>> >> constraints = none<br>> >> ; Type of constraint algorithm<br>> >> constraint-algorithm = Lincs<br>> >> ; Do not constrain the start configuration<br>> >> continuation = no<br>> >> ; Use successive overrelaxation to reduce the number of shake iterations<br>> >> Shake-SOR = no<br>> >> ; Relative tolerance of shake<br>> >> shake-tol = 0.0001<br>> >> ; Highest order in the expansion of the constraint coupling matrix<br>> >> lincs-order = 4<br>> >> ; Number of iterations in the final step of LINCS. 1 is fine for<br>> >> ; normal simulations, but use 2 to conserve energy in NVE runs.<br>> >> ; For energy minimization with constraints it should be 4 to 8.<br>> >> lincs-iter = 1<br>> >> ; Lincs will write a warning to the stderr if in one step a bond<br>> >> ; rotates over more degrees than<br>> >> lincs-warnangle = 30<br>> >> ; Convert harmonic bonds to morse potentials<br>> >> morse = no<br>> >><br>> >> ; ENERGY GROUP EXCLUSIONS<br>> >> ; Pairs of energy groups for which all non-bonded interactions are <br>> >> excluded<br>> >> energygrp_excl =<br>> >><br>> >> ; WALLS<br>> >> ; Number of walls, type, atom types, densities and box-z scale factor <br>> >> for Ewald<br>> >> nwall = 0<br>> >> wall_type = 9-3<br>> >> wall_r_linpot = -1<br>> >> wall_atomtype =<br>> >> wall_density =<br>> >> wall_ewald_zfac = 3<br>> >><br>> >><br>> >> ; Non-equilibrium MD stuff<br>> >> acc-grps =<br>> >> accelerate =<br>> >> freezegrps = SI_O<br>> >> freezedim = Y Y Y<br>> >> cos-acceleration = 0<br>> >> deform =<br>> >><br>> >><br>> >><br>> >><br>> >><br>> > _______________________________________________<br>> > gmx-users mailing list gmx-users@gromacs.org<br>> > http://lists.gromacs.org/mailman/listinfo/gmx-users<br>> > Please search the archive at http://www.gromacs.org/search before posting!<br>> > Please don't post (un)subscribe requests to the list. Use the www <br>> > interface or send it to gmx-users-request@gromacs.org.<br>> > Can't post? Read http://www.gromacs.org/mailing_lists/users.php<br>> <br>> <br>> -- <br>> David van der Spoel, Ph.D., Professor of Biology<br>> Molec. Biophys. group, Dept. of Cell & Molec. Biol., Uppsala University.<br>> Box 596, 75124 Uppsala, Sweden. Phone: +46184714205. Fax: +4618511755.<br>> spoel@xray.bmc.uu.se spoel@gromacs.org http://folding.bmc.uu.se<br>> _______________________________________________<br>> gmx-users mailing list gmx-users@gromacs.org<br>> http://lists.gromacs.org/mailman/listinfo/gmx-users<br>> Please search the archive at http://www.gromacs.org/search before posting!<br>> Please don't post (un)subscribe requests to the list. Use the <br>> www interface or send it to gmx-users-request@gromacs.org.<br>> Can't post? Read http://www.gromacs.org/mailing_lists/users.php<br><br /><hr />See all the ways you can stay connected <a href='http://www.microsoft.com/windows/windowslive/default.aspx' target='_new'>to friends and family</a></body>
</html>