<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7036.0">
<TITLE>Parallel Use of gromacs - parallel run is 20 times SLOWER than single node run</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<P><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">I ran a parallel (mpi) compiled version of gromacs using the following command line:</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">$ mpirun -np 5 mdrun_mpi -s topol.tpr -np 5 -v</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">At the end of the file md0.log I found:</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"> <FONT COLOR="#FF0000" SIZE=2 FACE="Arial">M E G A - F L O P S A C C O U N T I N G</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"> <FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Parallel run - timing based on wallclock.</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> RF=Reaction-Field FE=Free Energy SCFE=Soft-Core/Free Energy</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> T=Tabulated W3=SPC/TIP3p W4=TIP4p (single or pairs)</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> NF=No Forces</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Computing: M-Number M-Flops % of Flops</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">-----------------------------------------------------------------------</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Coulomb + LJ [W4-W4] 876.631638 234060.647346 88.0</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Outer nonbonded loop 692.459088 6924.590880 2.6</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> NS-Pairs 457.344228 9604.228788 3.6</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Reset In Box 13.782888 124.045992 0.0</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Shift-X 137.773776 826.642656 0.3</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> CG-CoM 3.445722 99.925938 0.0</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Sum Forces 206.660664 206.660664 0.1</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Virial 70.237023 1264.266414 0.5</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Update 68.886888 2135.493528 0.8</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Stop-CM 68.880000 688.800000 0.3</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> P-Coupling 68.886888 413.321328 0.2</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Calc-Ekin 68.893776 1860.131952 0.7</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Constraint-V 68.886888 413.321328 0.2</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Constraint-Vir 51.675498 1240.211952 0.5</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Settle 17.225166 5563.728618 2.1</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Virtual Site 3 17.221722 637.203714 0.2</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">-----------------------------------------------------------------------</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Total 266063.221098 100.0</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">-----------------------------------------------------------------------</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> NODE (s) Real (s) (%)</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Time: 3344.000 3344.000 100.0</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> 55:44</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> (Mnbf/s) (MFlops) (ns/day) (hour/ns)</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Performance: 0.262 79.564 0.517 46.444</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Detailed load balancing info in percentage of average</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Type NODE: 0 1 2 3 4 Scaling</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">-------------------------------------------</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Coulomb + LJ [W4-W4]:118 94 101 104 80 84%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Outer nonbonded loop: 97 98 98 103 102 96%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> NS-Pairs:116 94 101 104 82 85%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Reset In Box: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Shift-X: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> CG-CoM: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Sum Forces: 99 100 99 99 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Virial: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Update: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Stop-CM: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> P-Coupling: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Calc-Ekin: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Constraint-V: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Constraint-Vir: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Settle: 99 100 99 100 99 99%</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Virtual Site 3: 99 100 99 100 99 99%</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Total Force:118 94 101 104 81 84%</FONT></SPAN>
</P>
<BR>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Total Shake: 99 100 99 100 99 99%</FONT></SPAN>
</P>
<BR>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Total Scaling: 85% of max performance</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Finished mdrun on node 0 Sat Jul 14 23:32:32 2007</FONT></SPAN>
</P>
<BR>
<P><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">Now, I tried the same calculation on one node and found the following at the end of the file md.log:</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"> <FONT COLOR="#FF0000" SIZE=2 FACE="Arial">M E G A - F L O P S A C C O U N T I N G</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> RF=Reaction-Field FE=Free Energy SCFE=Soft-Core/Free Energy</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> T=Tabulated W3=SPC/TIP3p W4=TIP4p (single or pairs)</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> NF=No Forces</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Computing: M-Number M-Flops % of Flops</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">-----------------------------------------------------------------------</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Coulomb + LJ [W4-W4] 875.182588 233673.750996 88.0</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Outer nonbonded loop 688.853376 6888.533760 2.6</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> NS-Pairs 456.997574 9596.949054 3.6</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Reset In Box 13.782888 124.045992 0.0</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Shift-X 137.773776 826.642656 0.3</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> CG-CoM 3.445722 99.925938 0.0</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Virial 69.156915 1244.824470 0.5</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Update 68.886888 2135.493528 0.8</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Stop-CM 68.880000 688.800000 0.3</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> P-Coupling 68.886888 413.321328 0.2</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Calc-Ekin 68.893776 1860.131952 0.7</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Constraint-V 68.886888 413.321328 0.2</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Constraint-Vir 51.675498 1240.211952 0.5</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Settle 17.225166 5563.728618 2.1</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Virtual Site 3 17.221722 637.203714 0.2</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">-----------------------------------------------------------------------</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Total 265406.885286 100.0</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">-----------------------------------------------------------------------</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> NODE (s) Real (s) (%)</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> Time: 165.870 167.000 99.3</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> 2:45</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial"> (Mnbf/s) (GFlops) (ns/day) (hour/ns)</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Performance: 5.276 1.600 10.418 2.304</FONT></SPAN>
<BR><SPAN LANG="en-us"><FONT COLOR="#FF0000" SIZE=2 FACE="Arial">Finished mdrun on node 0 Thu Jul 12 15:17:49 2007</FONT></SPAN>
</P>
<BR>
<P><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">While I didn't expect to find pure linear scaling with gromacs. However, I didn't expect to find a massive INCREASE in computational effort across my 5 node, gigabit ethernet cluster.</FONT></SPAN></P>
<P><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">Anybody understand why this happened?</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">Thanks.</FONT></SPAN>
</P>
<P><SPAN LANG="en-us"><FONT SIZE=2 FACE="Arial">Jim Kress</FONT></SPAN>
</P>
</BODY>
</HTML>