<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
Even though it's generally slower, in some cases you can push the
hard scaling limit 2-3 times because in hybrid mode the cell is
decomposed in fewer domains for the same number of processors. It's
also faster on clusters with slow networks.<br>
<br>
Rossen<br>
<br>
<div class="moz-cite-prefix">On 4/25/13 9:47 AM, Erik Marklund
wrote:<br>
</div>
<blockquote
cite="mid:FBE6F27E-1FDB-4D26-B3C6-4867DB12A138@xray.bmc.uu.se"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
Hi,
<div><br>
</div>
<div>Please remind me why we allow for mixed OpenMP+MPI even
though it is always slower. It ought to be more complicated to
maintain code that allows such mixing.</div>
<div><br>
</div>
<div>Best,</div>
<div>Erik</div>
<div><br>
</div>
<div>
<div>
<div>On 25 Apr 2013, at 09:43, "<a moz-do-not-send="true"
href="mailto:hess@kth.se">hess@kth.se</a>" <<a
moz-do-not-send="true" href="mailto:hess@kth.se">hess@kth.se</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite"><span style="font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; font-family: Arial; ">Hi<br>
<br>
Yes, that is expected.<br>
Combined MPI+ OpenMP is always slower than either of the
two, except close to the scaling limit.<br>
Two OpenMP threads give the least overhead, especially
with hyperthreading. Although turning of hyperthreading is
then probably faster.<br>
<br>
Cheers,<br>
<br>
Berk<br>
<br>
<br>
----- Reply message -----<br>
From: "Jochen Hub" <<a moz-do-not-send="true"
href="mailto:jhub@gwdg.de">jhub@gwdg.de</a>><br>
To: "Discussion list for GROMACS development" <<a
moz-do-not-send="true"
href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a>><br>
Subject: [gmx-developers] Oversubscribing on 4.62 with MPI
/ OpenMP<br>
Date: Thu, Apr 25, 2013 09:37<br>
<br>
</span><br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">Am 4/24/13 9:53 PM, schrieb
Mark Abraham:</span><br style="font-family: Helvetica;
font-size: medium; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal;
line-height: normal; orphans: 2; text-align: -webkit-auto;
text-indent: 0px; text-transform: none; white-space:
normal; widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">> I suspect -np 2 is not
starting a process on each node like I suspect</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">> you think it should,
because all the symptoms are consistent with that.</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">> Possibly the Host field in
the .log file output is diagnostic here.</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">> Check how your your MPI
configuration works.</span><br style="font-family:
Helvetica; font-size: medium; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2; text-align:
-webkit-auto; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px; ">
<br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">I fixed the issue with the mpi
call. I make sure, that only one MPI<span
class="Apple-converted-space"> </span></span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">process is started per node
(mpiexec -n 2 -npernode=1 or -bynode) . The<span
class="Apple-converted-space"> </span></span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">oversubscription warning does
not appear, so everything seems fine.</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">However, the performance is
quite poor with MPI/OpenMP. Example:</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">(100 kAtoms, PME, Verlet,
cutoffs at 1nm nstlist=10)</span><br style="font-family:
Helvetica; font-size: medium; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2; text-align:
-webkit-auto; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px; ">
<br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">16 MPI processes: 6.8 ns/day</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">2 MPI processes, 8 OpenMP
threads pre MPI process: 4.46 ns/day</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">4 MPI / 4 OpenMP each does not
improve things.</span><br style="font-family: Helvetica;
font-size: medium; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal;
line-height: normal; orphans: 2; text-align: -webkit-auto;
text-indent: 0px; text-transform: none; white-space:
normal; widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px; ">
<br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">I use an icc13, and I tried
different MPI implementations (Mvapich 1.8,<span
class="Apple-converted-space"> </span></span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">openmpi 1.33)</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">Is that expected?</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">Many thanks,</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; display: inline
!important; float: none; ">Jochen</span><br
style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
<br style="font-family: Helvetica; font-size: medium;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; ">
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
</body>
</html>