<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#ffffff" text="#000000">how many particles is your system? if the number per domain is too low there is not much you can do about the load imbalance...but it did report only an overall 3.2% overhead for this so...<br>
<br>
you can modify the PP/PME ratio during mdrun by manually specifying the domain decomposition yourself.<br>
so for example since you are off by 33%, try to specify the dd count so you end up with 3x the nodes for PME than you did before...<br>
example: set -dd 10 10 10 to use 1000 PP nodes the rest will be PME nodes; you can use a nonsquare matrix just try to minimze the condition number<br>
-gcom effectively overrides nstcalenergy, as it tells each node how many steps to run before synchronizing. Usually, mdrun will let you know about excessive wait times for synch but we do not see it here with your system (must be running some really high end infiniband!)<br>
-- <br>
Sent from my Android phone with K-9 Mail. Please excuse my brevity.<br><br><div class="gmail_quote">Bruno Monnet <bruno.monnet@hp.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<font face="Helvetica, Arial, sans-serif">Hi,<br>
</font><br>
I'm not really a Gromacs user, but I'm currently benchmarking
Gromacs 4.5.4 on a large cluster. It seems that my communication
(PME) is really high and gromacs keeps complaining for more PME
nodes :<br>
<br>
<blockquote><tt> Average load imbalance: 113.6 %</tt><br>
<tt> Part of the total run time spent waiting due to load
imbalance: 3.3 %</tt><br>
<tt> Steps where the load balancing was limited by -rdd, -rcon
and/or -dds: X 9 % Y 9 % Z 9 %</tt><br>
<tt> Average PME mesh/force load: 3.288</tt><br>
<tt> Part of the total run time spent waiting due to PP/PME
imbalance: 32.6 %</tt><br>
<br>
<tt>NOTE: 32.6 % performance was lost because the PME nodes</tt><br>
<tt> had more work to do than the PP nodes.</tt><br>
<tt> You might want to increase the number of PME nodes</tt><br>
<tt> or increase the cut-off and the grid spacing.</tt><br>
</blockquote>
<br>
I can't modify the original dataset as I only have the TPR file. I
switched from dlb yes -> dlb auto since it seems to have trouble
with more than 6000 / 8000 cores.<br>
<br>
I tried to add " -gcom " parameter. This speedup the computation.
This parameter is not really explained in the Gromacs documentation.
Could you give me some advice on how I could use it ?<br>
<br>
Best regards,<br>
Bruno Monnet<br>
<br>
<div class="moz-signature">
</div>
</blockquote></div></body>
</html>