<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
You can simply print the kinetic energy and scaling factor you
calculate and compare a serial against a parallel run and you will
see where things go off.<br>
<br>
But calculating kinetic energies and scaling is not trivial in MD.
With the leap-frog integrator the reported kinetic energy is the
average of that at t-0.5*delta_t and t+0.5*delta_t. And then the
result will also depend when during the step you scale the
velocities (before/after which half-step Ekin calculation).<br>
<br>
Collecting the state is a very expensive operation that you should
avoid. Better is to calculate the rank contribution to Ekin and do
a MPI_Allreduce on that. But since mdrun already does this for
energy reporting and temperature scaling, why not simply use the
Ekin mdrun calculates already? I would suggest to look at one of
the temperature coupling algorithms, e.g. v-rescale, and reuse
most of that machinery.<br>
<br>
Cheers,<br>
<br>
Berk<br>
<br>
On 03/19/2015 09:01 AM, Bernhardt, Nathan A. wrote:<br>
</div>
<blockquote cite="mid:FB2E3680-B205-44DE-BD2E-58EE16A1F292@ou.edu"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
Hello everybody,
<div><br>
</div>
<div>This is my first time to post a message so feel free to
comment accordingly. With that said however, let me get right to
it.</div>
<div><br>
</div>
<div>i am performing md simulations (GROMACS 4.6.5) using the leap
frog integrator. At this point i am not using LINCS but hope to
be able to later. Lets just assume that at some point in the
simulation i need to scale velocities such that the total energy
after scaling is equal to some target value. I can do this
easily so long as a single processor is used in the simulation.
Unfortunately, if i attempt to use parallel processing (domain
decomp) my equation for calculating the scaling factor seems to
no longer work. This should not be the case and leads me to
believe i may have misunderstood how gromacs is working.</div>
<div><br>
</div>
<div>I have tried two ways of doing this when domain decomp is
used. the first uses dd_partition_system() and is as follows.</div>
<div><br>
</div>
<div> <span style="font-family: Menlo; font-size: 11px; color:
rgb(206, 121, 36);">if</span><span style="font-family: Menlo;
font-size: 11px;">(DOMAINDECOMP(cr))</span></div>
<div><span style="font-family: Menlo; font-size: 11px;">{</span></div>
<div><span style="font-family: Menlo; font-size: 11px;"><span
class="Apple-tab-span" style="white-space:pre"></span></span><span
style="font-family: Menlo; font-size: 11px;">dd_collect_state(cr->dd,
state, state_global);</span></div>
<div><font face="Menlo"><span style="font-size: 11px;">}</span></font></div>
<div><font face="Menlo"><span style="font-size: 11px;"><br>
</span></font></div>
<div><font face="Menlo"><span style="font-size: 11px;">if(group_rank
== 0)</span></font></div>
<div><font face="Menlo"><span style="font-size: 11px;">{</span></font></div>
<div><font face="Menlo"><span class="Apple-tab-span"
style="font-size: 11px; white-space: pre;"></span><span
style="font-size: 11px;">calc_scaling_factor(state_global,
energy_data); //this function calculates the scaling factor.
i believe it is correct. </span></font></div>
<div><font face="Menlo"><span style="font-size: 11px;">}</span></font></div>
<div><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><span
style="color: rgb(206, 121, 36);">for</span>(i=<span
style="color: rgb(195, 55, 32);">0</span>; i<nr_atoms; i++)</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">{</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><span
class="Apple-tab-span" style="white-space:pre"></span>state_global->v[i][<span
style="color: rgb(195, 55, 32);">0</span>] =
scaling_factor*state_global->v[i][<span style="color:
rgb(195, 55, 32);">0</span>];</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><span
class="Apple-tab-span" style="white-space:pre"></span>state_global->v[i][<font
color="#c33720">1</font>] =
scaling_factor*state_global->v[i][<font color="#c33720">1</font>];</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><span
class="Apple-tab-span" style="white-space:pre"></span>state_global->v[i][<font
color="#c33720">2</font>] =
scaling_factor*state_global->v[i][<font color="#c33720">2</font>];</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">}</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">if(DOMAINDECOMP(cr))</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">{</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><span
class="Apple-tab-span" style="white-space:pre"></span>dd_partition_system(fplog,
step, cr, TRUE,
<span style="color: rgb(195, 55, 32);">1</span>,</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">
<span class="Apple-tab-span" style="white-space:pre">
</span>state_global, top_global, ir,</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">
state, &f, mdatoms, top, fr,</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">
vsite, shellfc, constr,</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">
nrnb, wcycle, FALSE);<br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">}</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">I
will admit i do not fully understand what
all dd_partition_system() does and the parameters it takes in.
Thus, i tried to avoid using it by simply having each node scale
the velocities for its home atoms. Below is sample code from
that attempt.</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">
<div style="font-family: Helvetica; font-size: 12px;"> <span
style="font-family: Menlo; font-size: 11px; color: rgb(206,
121, 36);">if</span><span style="font-family: Menlo;
font-size: 11px;">(DOMAINDECOMP(cr))</span></div>
<div style="font-family: Helvetica; font-size: 12px;"><span
style="font-family: Menlo; font-size: 11px;">{</span></div>
<div style="font-family: Helvetica; font-size: 12px;"><span
style="font-family: Menlo; font-size: 11px;"><span
class="Apple-tab-span" style="white-space: pre;"></span></span><span
style="font-family: Menlo; font-size: 11px;">dd_collect_state(cr->dd,
state, state_global);</span></div>
<div style="font-family: Helvetica; font-size: 12px;"><font
face="Menlo"><span style="font-size: 11px;">}</span></font></div>
<div style="font-family: Helvetica; font-size: 12px;"><font
face="Menlo"><span style="font-size: 11px;"><br>
</span></font></div>
<div style="font-family: Helvetica; font-size: 12px;"><font
face="Menlo"><span style="font-size: 11px;">if(group_rank ==
0)</span></font></div>
<div style="font-family: Helvetica; font-size: 12px;"><font
face="Menlo"><span style="font-size: 11px;">{</span></font></div>
<div style="font-family: Helvetica; font-size: 12px;"><font
face="Menlo"><span class="Apple-tab-span" style="font-size:
11px; white-space: pre;"></span><span style="font-size:
11px;">calc_scaling_factor(state_global, energy_data);
//this function calculates the scaling factor. i believe
it is correct. </span></font></div>
<div style="font-family: Helvetica; font-size: 12px;"><font
face="Menlo"><span style="font-size: 11px;">}</span></font></div>
<div><font face="Menlo"><span style="font-size: 11px;"><br>
</span></font></div>
<div><font face="Menlo"><span style="font-size: 11px;">broadcast_scaling_factor(scaling_factor);
//This sends the scaling factor to all nodes so they can
scale their home atoms velocities accordingly. </span></font></div>
<div><font face="Menlo"><span style="font-size: 11px;"><br>
</span></font></div>
<div>
<div style="margin: 0px;">if(DOMAINDECOMP(cr))</div>
<div style="margin: 0px;">{</div>
</div>
<div style="margin: 0px;"><span class="Apple-tab-span"
style="white-space:pre"></span><span style="color: rgb(206,
121, 36);">for</span>(i=<span style="color: rgb(195, 55,
32);">0</span>; i<mdatoms[<span style="color: rgb(195,
55, 32);">0</span>].homenr; i++)</div>
<div style="margin: 0px;"><span class="Apple-tab-span"
style="white-space:pre"></span>{</div>
<div style="margin: 0px;"><span class="Apple-tab-span"
style="white-space:pre"></span>state->v[i][<span
style="color: rgb(195, 55, 32);">0</span>] =
scaling_factor*state->v[i][<span style="color: rgb(195, 55,
32);">0</span>];</div>
<div style="margin: 0px;"><span class="Apple-tab-span"
style="white-space:pre"></span>state->v[i][<font
color="#c33720">1</font>] = scaling_factor*state->v[i][<font
color="#c33720">1</font>];</div>
<div style="margin: 0px;"><span class="Apple-tab-span"
style="white-space:pre"></span>state->v[i][<font
color="#c33720">2</font>] = scaling_factor*state->v[i][<font
color="#c33720">2</font>];</div>
<div style="margin: 0px;"><span class="Apple-tab-span"
style="white-space:pre"></span>}</div>
<div style="margin: 0px;">}</div>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">Neither
of these methods have proved successful. In both cases the total
energy after scaling is the same and off slightly from the
target value by about 10kj/mol. In the second method I have
tested that broadcasting of the scaling factor is successful.
Since both methods require a global state to be collected and
used for calculating the scaling factor i see no reason why the
function that calculates this factor would work for the single
processor case and not the domain decomp one. This leads me to
believe that either the dd_collect_state() function is not doing
exactly what i think it is, the dd_partition_system() may be
working in ways i do not understand or that state->v[][] may
not be the array I need to modify. Any help is appreciated. </div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">Thanks</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;">Nathan
Bernhardt</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;"><br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
</body>
</html>