<div dir="ltr">Hi,<div><br></div><div>please don't use gmx-developers for user questions. Feel free to use it if you want to fix the problem, and have questions about implementation details.</div><div><br></div><div><div class="gmail_extra">
Please provide more details: How large is your system? How much memory does a node have? On how many nodes do you try to run? How many mpi-ranks do you have per node?</div><div class="gmail_extra"><br></div><div class="gmail_extra">
Roland<br><br><div class="gmail_quote">
On Thu, Aug 21, 2014 at 12:21 PM, Yunlong Liu <span dir="ltr"><<a href="mailto:yliu120@jh.edu" target="_blank">yliu120@jh.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hi Gromacs Developers,<br>
<br>
I found something about the dynamic loading balance really interesting. I am running my simulation on Stampede supercomputer, which has nodes with 16-physical core ( really 16 Intel Xeon cores on one node ) and an NVIDIA Tesla K20m GPU associated.
<br>
<br>
When I am using only the CPUs, I turned on dynamic loading balance by -dlb yes. And it seems to work really good, and the loading imbalance is only 1~2%. This really helps improve the performance by 5~7%。But when I am running my code on GPU-CPU hybrid ( GPU
node, 16-cpu and 1 GPU), the dynamic loading balance kicked in since the imbalance goes up to ~50% instantly after loading. Then the the system reports a fail-to-allocate-memory error:<br>
<br>
<font color="#3366ff">NOTE: Turning on dynamic load balancing<br>
<br>
<br>
-------------------------------------------------------<br>
Program mdrun_mpi, VERSION 5.0<br>
Source code file: /home1/03002/yliu120/build/gromacs-5.0/src/gromacs/utility/smalloc.c, line: 226<br>
<br>
Fatal error:<br>
Not enough memory. Failed to realloc 1020720 bytes for dest->a, dest->a=d5800030<br>
(called from file /home1/03002/yliu120/build/gromacs-5.0/src/gromacs/mdlib/domdec_top.c, line 1061)<br>
For more information and tips for troubleshooting, please check the GROMACS<br>
website at <a href="http://www.gromacs.org/Documentation/Errors" target="_blank">
http://www.gromacs.org/Documentation/Errors</a><br>
-------------------------------------------------------<br>
: Cannot allocate memory<br>
Error on rank 0, will try to stop all ranks<br>
Halting parallel program mdrun_mpi on CPU 0 out of 4<br>
<br>
gcq#274: "I Feel a Great Disturbance in the Force" (The Emperor Strikes Back)<br>
<br>
[cli_0]: aborting job:<br>
application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0<br>
[c442-702.stampede.tacc.utexas.edu:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 6. MPI process died?<br>
[c442-702.stampede.tacc.utexas.edu:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?<br>
[c442-702.stampede.tacc.utexas.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 112839) exited with status 255<br>
TACC: MPI job exited with code: 1<br>
<br>
TACC: Shutdown complete. Exiting.</font><br>
<br>
So I manually turned off the dynamic loading balance by -dlb no. The simulation goes through with the very high loading imbalance, like:<br>
<br>
<font color="#3366ff">DD step 139999 load imb.: force 51.3%<br>
<br>
Step Time Lambda<br>
140000 280.00000 0.00000<br>
<br>
Energies (kJ/mol)<br>
U-B Proper Dih. Improper Dih. CMAP Dih. LJ-14<br>
4.88709e+04 1.21990e+04 2.99128e+03 -1.46719e+03 1.98569e+04<br>
Coulomb-14 LJ (SR) Disper. corr. Coulomb (SR) Coul. recip.<br>
2.54663e+05 4.05141e+05 -3.16020e+04 -3.75610e+06 2.24819e+04<br>
Potential Kinetic En. Total Energy Temperature Pres. DC (bar)<br>
-3.02297e+06 6.15217e+05 -2.40775e+06 3.09312e+02 -2.17704e+02<br>
Pressure (bar) Constr. rmsd<br>
-3.39003e+01 3.10750e-05<br>
<br>
DD step 149999 load imb.: force 60.8%<br>
<br>
Step Time Lambda<br>
150000 300.00000 0.00000<br>
<br>
Energies (kJ/mol)<br>
U-B Proper Dih. Improper Dih. CMAP Dih. LJ-14<br>
4.96380e+04 1.21010e+04 2.99986e+03 -1.51918e+03 1.97542e+04<br>
Coulomb-14 LJ (SR) Disper. corr. Coulomb (SR) Coul. recip.<br>
2.54305e+05 4.06024e+05 -3.15801e+04 -3.75534e+06 2.24001e+04<br>
Potential Kinetic En. Total Energy Temperature Pres. DC (bar)<br>
-3.02121e+06 6.17009e+05 -2.40420e+06 3.10213e+02 -2.17403e+02<br>
Pressure (bar) Constr. rmsd<br>
-1.40623e+00 3.16495e-05</font><br>
<div><br>
I think this high loading imbalance will affect more than 20% of the performance but at least it will let the simulation on. Therefore, the problem I would like to report is that when running simulation with GPU-CPU hybrid with very few GPU, the dynamic loading
balance will cause domain decomposition problems ( fail-to-allocate-memory ). I don't know whether there is any solution to this problem currently or anything could be improved?<span><font color="#888888"><br>
<br>
Yunlong<br>
<br>
<br>
<br>
<br>
-- <br>
<p><font color="blue">========================================<br>
Yunlong Liu, PhD Candidate<br>
Computational Biology and Biophysics<br>
Department of Biophysics and Biophysical Chemistry<br>
School of Medicine, The Johns Hopkins University<br>
Email: <a href="mailto:yliu120@jhmi.edu" target="_blank">yliu120@jhmi.edu</a><br>
Address: 725 N Wolfe St, WBSB RM 601, 21205<br>
========================================<br>
</font></p>
</font></span></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br><a href="tel:865-241-1537" value="+18652411537" target="_blank">865-241-1537</a>, ORNL PO BOX 2008 MS6309
</div></div></div>