<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">Hi,<br>
<br>
I think Mark's suggestion is very good. This avoids all special
communication for Drude oscillators. This only requires adding
Drude connections to the update group setup.<br>
<br>
Is what you are printing based on the global atom number? If not
that explains a lot.<br>
<br>
Cheers,<br>
<br>
Berk<br>
<br>
On 2020-06-09 20:26, Mark Abraham wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAMNuMAS0FKChaf5LBdzehJEmEpXtd61P0HR7T3Q+iQ1GNbbWbA@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
<div>Perhaps not the solution you're looking for, but since
2019, DD has been based on the notion of a domain being a
compact collection of update groups (which are indivisible
units like -CH2-) rather than a strict geometric criterion.
That was done so that h-bond only constraints need not
communicate, but is probably also a good choice for
Drude+parent. You should still be able to validate the
single-domain cases with your old code based on a long-ago
version.</div>
<div><br>
</div>
<div>Mark<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 9 Jun 2020 at 19:14,
Justin Lemkul <<a href="mailto:jalemkul@vt.edu"
moz-do-not-send="true">jalemkul@vt.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">...<br>
Hi All,<br>
<br>
I'm trying (once again) to get back into figuring out the
lingering bugs <br>
with the Drude implementation when using domain decomposition.
Since I <br>
last asked for help, I have gotten coordinate and velocity
communication <br>
working properly. Now, I'm stuck on forces. To quickly recap
the issue, <br>
it is possible that Drudes and their parent atoms get
separated in <br>
different domains. This requires communication of coordinates,
<br>
velocities, and forces via treatment as "special atoms" like
is the case <br>
with virtual sites. As such, my implementation largely follows
what <br>
happens for the virtual sites (communicate after any update).<br>
<br>
I have been tracing the forces at every step of do_force -
basically <br>
printing out the force on a Drude that I know is in a
different domain <br>
from its parent atom. I use the OpenMP output as reference. I
can <br>
reproduce the OpenMP forces with domain decomposition but no <br>
communication (e.g. gmx mdrun -ntmpi 2 -npme 1 -deffnm md -nb
cpu), <br>
based on Berk's suggestion from a long time ago. So the issue
I'm having <br>
must be coming from communicating somewhere, but I can't nail
it down. <br>
Here is an example of the output I'm looking at.<br>
<br>
First, from OpenMP (my reference, the correct output):<br>
<br>
=== Step 0 ===<br>
DO FORCE: top f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after do_nb_verlet #1 f[54] = 0.000000 0.000000
0.000000<br>
DO FORCE: after do_nb_verlet #2 f[54] = 0.000000 0.000000
0.000000<br>
DO FORCE: after nbnxn_atomdata_add_nbat_f_to_f f[54] =
1271.383667 <br>
-3106.622803 2148.540283<br>
DO FORCE: after nbnxn_atomdata_add_nbat_fshift_to_fshift f[54]
= <br>
1271.383667 -3106.622803 2148.540283<br>
DO FORCE: after do_force_lowlevel f[54] = 82.651733 130.833740
82.218506<br>
DO FORCE: b4 move_f f[54] = 82.651733 130.833740 82.218506<br>
DO FORCE: after move_f f[54] = 82.651733 130.833740 82.218506<br>
DO FORCE: after GPU use/emulate f[54] = 82.651733 130.833740
82.218506<br>
DO FORCE: after vsite_spread f[54] = 82.651733 130.833740
82.218506<br>
DO FORCE: b4 post f[54] = 82.651733 130.833740 82.218506<br>
DO FORCE: end f[54] = 58.264297 16.147758 43.956337<br>
=== Step 1 ===<br>
DO FORCE: top f[54] = 58.264297 16.147758 43.956337<br>
DO FORCE: after do_nb_verlet #1 f[54] = 0.000000 0.000000
0.000000<br>
DO FORCE: after do_nb_verlet #2 f[54] = 0.000000 0.000000
0.000000<br>
DO FORCE: after nbnxn_atomdata_add_nbat_f_to_f f[54] =
1205.647705 <br>
-3128.451904 2138.944580<br>
DO FORCE: after nbnxn_atomdata_add_nbat_fshift_to_fshift f[54]
= <br>
1205.647705 -3128.451904 2138.944580<br>
DO FORCE: after do_force_lowlevel f[54] = 200.794189
-175.644287 -279.924072<br>
DO FORCE: b4 move_f f[54] = 200.794189 -175.644287 -279.924072<br>
DO FORCE: after move_f f[54] = 200.794189 -175.644287
-279.924072<br>
DO FORCE: after GPU use/emulate f[54] = 200.794189 -175.644287
-279.924072<br>
DO FORCE: after vsite_spread f[54] = 200.794189 -175.644287
-279.924072<br>
DO FORCE: b4 post f[54] = 200.794189 -175.644287 -279.924072<br>
DO FORCE: end f[54] = 162.370026 -306.717041 -321.102356<br>
<br>
<br>
Now, my implementation with domain decomposition:<br>
<br>
=== Step 0 ===<br>
DO FORCE: top f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after do_nb_verlet #1 f[54] = 0.000000 0.000000
0.000000<br>
DO FORCE: after do_nb_verlet #2 f[54] = 0.000000 0.000000
0.000000<br>
DO FORCE: after nbnxn_atomdata_add_nbat_f_to_f f[54] =
338.912842 <br>
-2940.618164 2357.080078<br>
DO FORCE: after do_force_lowlevel f[54] = 1899.546387
-1663.452881 <br>
1703.655273<br>
DO FORCE: b4 move_f f[54] = 1899.546387 -1663.452881
1703.655273<br>
DO FORCE: after move_f f[54] = 82.647949 130.835449 82.213165<br>
DO FORCE: after GPU use/emulate f[54] = 82.647949 130.835449
82.213165<br>
DO FORCE: after vsite_spread f[54] = 82.647949 130.835449
82.213165<br>
DO FORCE: b4 post f[54] = 82.647949 130.835449 82.213165<br>
DO FORCE: end f[54] = 58.260483 16.149330 43.951458<br>
=== Step 1 ===<br>
DO FORCE: top f[54] = 58.260483 16.149330 43.951458<br>
DO FORCE: after do_nb_verlet #1 f[54] = 0.000000 0.000000
0.000000<br>
DO FORCE: after do_nb_verlet #2 f[54] = 0.000000 0.000000
0.000000<br>
DO FORCE: after nbnxn_atomdata_add_nbat_f_to_f f[54] =
265.444092 <br>
-2965.024170 2346.120117<br>
DO FORCE: after do_force_lowlevel f[54] = 1834.273926
-1685.225830 <br>
1654.119141<br>
DO FORCE: b4 move_f f[54] = 1834.273926 -1685.225830
1654.119141<br>
DO FORCE: after move_f f[54] = 258.300781 -122.286865
-219.277039<br>
DO FORCE: after GPU use/emulate f[54] = 258.300781 -122.286865
-219.277039<br>
DO FORCE: after vsite_spread f[54] = 258.300781 -122.286865
-219.277039<br>
DO FORCE: b4 post f[54] = 258.300781 -122.286865 -219.277039<br>
DO FORCE: end f[54] = 229.446487 -248.274734 -255.144485<br>
<br>
From this output, I can see that communication works in step
0 and <br>
between steps 0 and 1, since the force is correctly
propagated. I also <br>
do not know to what extent I can expect forces to match before
the <br>
"move_f" step (which is where I communicate non-local Drude
forces and <br>
follows the existing "dd_move_f" in do_force_cutsVERLET). But
the forces <br>
should certainly be the same after communicating so they are
correctly <br>
input to post_process_forces.<br>
<br>
Can anyone suggest how the code paths might differ between
these two <br>
steps? I've debugged every step along the way that I can
figure out and <br>
all I can come up with is that the forces end up different. I
know that <br>
may be a big request without seeing the code, but I'm simply
determining <br>
non-local Drudes the same way we do with vsites, and
communicating their <br>
forces with the existing dd_move_f_specat function that vsites
also use.<br>
<br>
Any help would be greatly appreciated. I've been stuck on this
forever <br>
and it is clear that our user community really wants this
feature. I can <br>
give them OpenMP easily, but that's rather restrictive...<br>
<br>
-Justin<br>
<br>
-- <br>
==================================================<br>
<br>
Justin A. Lemkul, Ph.D.<br>
Assistant Professor<br>
Office: 301 Fralin Hall<br>
Lab: 303 Engel Hall<br>
<br>
Virginia Tech Department of Biochemistry<br>
340 West Campus Dr.<br>
Blacksburg, VA 24061<br>
<br>
<a href="mailto:jalemkul@vt.edu" target="_blank"
moz-do-not-send="true">jalemkul@vt.edu</a> | (540) 231-3129<br>
<a href="http://www.thelemkullab.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">http://www.thelemkullab.com</a><br>
<br>
==================================================<br>
<br>
-- <br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a
href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a>
before posting!<br>
<br>
* Can't post? Read <a
href="http://www.gromacs.org/Support/Mailing_Lists"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a
href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>
or send a mail to <a
href="mailto:gmx-developers-request@gromacs.org"
target="_blank" moz-do-not-send="true">gmx-developers-request@gromacs.org</a>.<br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
</blockquote>
<br>
</body>
</html>