<div dir="ltr"><div>Hi,</div><div><br></div><div>Perhaps not the solution you're looking for, but since 2019, DD has been based on the notion of a domain being a compact collection of update groups (which are indivisible units like -CH2-) rather than a strict geometric criterion. That was done so that h-bond only constraints need not communicate, but is probably also a good choice for Drude+parent. You should still be able to validate the single-domain cases with your old code based on a long-ago version.</div><div><br></div><div>Mark<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 9 Jun 2020 at 19:14, Justin Lemkul <<a href="mailto:jalemkul@vt.edu">jalemkul@vt.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Hi All,<br>
<br>
I'm trying (once again) to get back into figuring out the lingering bugs <br>
with the Drude implementation when using domain decomposition. Since I <br>
last asked for help, I have gotten coordinate and velocity communication <br>
working properly. Now, I'm stuck on forces. To quickly recap the issue, <br>
it is possible that Drudes and their parent atoms get separated in <br>
different domains. This requires communication of coordinates, <br>
velocities, and forces via treatment as "special atoms" like is the case <br>
with virtual sites. As such, my implementation largely follows what <br>
happens for the virtual sites (communicate after any update).<br>
<br>
I have been tracing the forces at every step of do_force - basically <br>
printing out the force on a Drude that I know is in a different domain <br>
from its parent atom. I use the OpenMP output as reference. I can <br>
reproduce the OpenMP forces with domain decomposition but no <br>
communication (e.g. gmx mdrun -ntmpi 2 -npme 1 -deffnm md -nb cpu), <br>
based on Berk's suggestion from a long time ago. So the issue I'm having <br>
must be coming from communicating somewhere, but I can't nail it down. <br>
Here is an example of the output I'm looking at.<br>
<br>
First, from OpenMP (my reference, the correct output):<br>
<br>
=== Step 0 ===<br>
DO FORCE: top f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after do_nb_verlet #1 f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after do_nb_verlet #2 f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after nbnxn_atomdata_add_nbat_f_to_f f[54] = 1271.383667 <br>
-3106.622803 2148.540283<br>
DO FORCE: after nbnxn_atomdata_add_nbat_fshift_to_fshift f[54] = <br>
1271.383667 -3106.622803 2148.540283<br>
DO FORCE: after do_force_lowlevel f[54] = 82.651733 130.833740 82.218506<br>
DO FORCE: b4 move_f f[54] = 82.651733 130.833740 82.218506<br>
DO FORCE: after move_f f[54] = 82.651733 130.833740 82.218506<br>
DO FORCE: after GPU use/emulate f[54] = 82.651733 130.833740 82.218506<br>
DO FORCE: after vsite_spread f[54] = 82.651733 130.833740 82.218506<br>
DO FORCE: b4 post f[54] = 82.651733 130.833740 82.218506<br>
DO FORCE: end f[54] = 58.264297 16.147758 43.956337<br>
=== Step 1 ===<br>
DO FORCE: top f[54] = 58.264297 16.147758 43.956337<br>
DO FORCE: after do_nb_verlet #1 f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after do_nb_verlet #2 f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after nbnxn_atomdata_add_nbat_f_to_f f[54] = 1205.647705 <br>
-3128.451904 2138.944580<br>
DO FORCE: after nbnxn_atomdata_add_nbat_fshift_to_fshift f[54] = <br>
1205.647705 -3128.451904 2138.944580<br>
DO FORCE: after do_force_lowlevel f[54] = 200.794189 -175.644287 -279.924072<br>
DO FORCE: b4 move_f f[54] = 200.794189 -175.644287 -279.924072<br>
DO FORCE: after move_f f[54] = 200.794189 -175.644287 -279.924072<br>
DO FORCE: after GPU use/emulate f[54] = 200.794189 -175.644287 -279.924072<br>
DO FORCE: after vsite_spread f[54] = 200.794189 -175.644287 -279.924072<br>
DO FORCE: b4 post f[54] = 200.794189 -175.644287 -279.924072<br>
DO FORCE: end f[54] = 162.370026 -306.717041 -321.102356<br>
<br>
<br>
Now, my implementation with domain decomposition:<br>
<br>
=== Step 0 ===<br>
DO FORCE: top f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after do_nb_verlet #1 f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after do_nb_verlet #2 f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after nbnxn_atomdata_add_nbat_f_to_f f[54] = 338.912842 <br>
-2940.618164 2357.080078<br>
DO FORCE: after do_force_lowlevel f[54] = 1899.546387 -1663.452881 <br>
1703.655273<br>
DO FORCE: b4 move_f f[54] = 1899.546387 -1663.452881 1703.655273<br>
DO FORCE: after move_f f[54] = 82.647949 130.835449 82.213165<br>
DO FORCE: after GPU use/emulate f[54] = 82.647949 130.835449 82.213165<br>
DO FORCE: after vsite_spread f[54] = 82.647949 130.835449 82.213165<br>
DO FORCE: b4 post f[54] = 82.647949 130.835449 82.213165<br>
DO FORCE: end f[54] = 58.260483 16.149330 43.951458<br>
=== Step 1 ===<br>
DO FORCE: top f[54] = 58.260483 16.149330 43.951458<br>
DO FORCE: after do_nb_verlet #1 f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after do_nb_verlet #2 f[54] = 0.000000 0.000000 0.000000<br>
DO FORCE: after nbnxn_atomdata_add_nbat_f_to_f f[54] = 265.444092 <br>
-2965.024170 2346.120117<br>
DO FORCE: after do_force_lowlevel f[54] = 1834.273926 -1685.225830 <br>
1654.119141<br>
DO FORCE: b4 move_f f[54] = 1834.273926 -1685.225830 1654.119141<br>
DO FORCE: after move_f f[54] = 258.300781 -122.286865 -219.277039<br>
DO FORCE: after GPU use/emulate f[54] = 258.300781 -122.286865 -219.277039<br>
DO FORCE: after vsite_spread f[54] = 258.300781 -122.286865 -219.277039<br>
DO FORCE: b4 post f[54] = 258.300781 -122.286865 -219.277039<br>
DO FORCE: end f[54] = 229.446487 -248.274734 -255.144485<br>
<br>
From this output, I can see that communication works in step 0 and <br>
between steps 0 and 1, since the force is correctly propagated. I also <br>
do not know to what extent I can expect forces to match before the <br>
"move_f" step (which is where I communicate non-local Drude forces and <br>
follows the existing "dd_move_f" in do_force_cutsVERLET). But the forces <br>
should certainly be the same after communicating so they are correctly <br>
input to post_process_forces.<br>
<br>
Can anyone suggest how the code paths might differ between these two <br>
steps? I've debugged every step along the way that I can figure out and <br>
all I can come up with is that the forces end up different. I know that <br>
may be a big request without seeing the code, but I'm simply determining <br>
non-local Drudes the same way we do with vsites, and communicating their <br>
forces with the existing dd_move_f_specat function that vsites also use.<br>
<br>
Any help would be greatly appreciated. I've been stuck on this forever <br>
and it is clear that our user community really wants this feature. I can <br>
give them OpenMP easily, but that's rather restrictive...<br>
<br>
-Justin<br>
<br>
-- <br>
==================================================<br>
<br>
Justin A. Lemkul, Ph.D.<br>
Assistant Professor<br>
Office: 301 Fralin Hall<br>
Lab: 303 Engel Hall<br>
<br>
Virginia Tech Department of Biochemistry<br>
340 West Campus Dr.<br>
Blacksburg, VA 24061<br>
<br>
<a href="mailto:jalemkul@vt.edu" target="_blank">jalemkul@vt.edu</a> | (540) 231-3129<br>
<a href="http://www.thelemkullab.com" rel="noreferrer" target="_blank">http://www.thelemkullab.com</a><br>
<br>
==================================================<br>
<br>
-- <br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
</blockquote></div>