<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi,<br>
<br>
Those timers report the time the CPU is waiting for results to
arrive from the local and non-local non-bonded calculations on the
GPU. When the CPU has few or no forces to compute, this wait time
can be a large part of the total run time.<br>
<br>
Cheers,<br>
<br>
Berk<br>
<br>
On 2020-04-14 10:37 , 张驭洲 wrote:<br>
</div>
<blockquote
cite="mid:2613d5c7.808f.17177d626f5.Coremail.zhangyuzhou15@mails.ucas.edu.cn"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p> Hello GROMACS developers,
</p>
<p> <br>
</p>
<p> I'm using GROMACS 2020.1 on a node with 2 Intel(R) Xeon(R)
Gold 6142 CPUs and 4 NVIDIA Tesla V100-PCIE-32GB GPUs.
</p>
<p> With the command line as follows:
</p>
<p> gmx mdrun -s p16.tpr -o p16.trr -c p16_out.gro -e p16.edr
-g p16.log -pin on -ntmpi 4 -ntomp 6 -nb gpu -bonded gpu -pme
gpu -npme 1
</p>
<p> I got the following performance results:
</p>
<p> </p>
<p> R E A L C Y C L E A N D T I M E A C C O U N T I N
G </p>
<p> On 3 MPI ranks doing PP, each using 6 OpenMP threads, and<br>
on 1 MPI rank doing PME, using 6 OpenMP threads </p>
<p> Computing: Num Num Call Wall time
Giga-Cycles<br>
Ranks Threads Count (s) total
sum %<br>
-----------------------------------------------------------------------------<br>
Domain decomp. 3 6 2001 15.290
715.584 6.4<br>
DD comm. load 3 6 245 0.008
0.377 0.0<br>
DD comm. bounds 3 6 48 0.003
0.151 0.0<br>
Send X to PME 3 6 200001 9.756
456.559 4.1<br>
Neighbor search 3 6 2001 12.184
570.190 5.1<br>
Launch GPU ops. 3 6 400002 17.929
839.075 7.5<br>
Force 3 6 200001 3.912
183.082 1.6<br>
Wait + Comm. F 3 6 40001 4.229
197.913 1.8<br>
PME mesh * 1 6 200001 16.733
261.027 2.3<br>
PME wait for PP * 162.467
2534.449 22.7<br>
Wait + Recv. PME F 3 6 200001 18.827
881.091 7.9<br>
Wait PME GPU gather 3 6 200001 2.896
135.522 1.2<br>
Wait Bonded GPU 3 6 2001 0.003
0.122 0.0<br>
Wait GPU NB nonloc. 3 6 200001 15.328
717.330 6.4<br>
Wait GPU NB local 3 6 200001 0.175
8.169 0.1<br>
Wait GPU state copy 3 6 160000 26.204
1226.327 11.0<br>
NB X/F buffer ops. 3 6 798003 7.023
328.655 2.9<br>
Write traj. 3 6 21 0.182
8.540 0.1<br>
Update 3 6 200001 6.685
312.856 2.8<br>
Comm. energies 3 6 40001 6.684
312.796 2.8<br>
Rest 31.899
1492.851 13.3<br>
-----------------------------------------------------------------------------<br>
Total 179.216
11182.921 100.0<br>
-----------------------------------------------------------------------------<br>
(*) Note that with separate PME ranks, the walltime column
actually sums to<br>
twice the total reported, but the cycle count total and %
are correct.<br>
-----------------------------------------------------------------------------
</p>
<p> Core t (s) Wall t (s) (%)<br>
Time: 4301.031 179.216 2399.9<br>
(ns/day) (hour/ns)<br>
Performance: 96.421 0.249 </p>
<p> <br>
</p>
<p> Using two nodes and the following command: </p>
<p> gmx_mpi mdrun -s p16.tpr -o p16.trr -c p16_out.gro -e
p16.edr -g p16.log -ntomp 6 -nb gpu -bonded gpu -pme gpu -npme 1
</p>
<p> I got these results: </p>
<p> <br>
</p>
<p> </p>
<p> R E A L C Y C L E A N D T I M E A C C O U N T I N
G </p>
<p> On 6 MPI ranks doing PP, each using 6 OpenMP threads, and<br>
on 1 MPI rank doing PME, using 6 OpenMP threads </p>
<p> Computing: Num Num Call Wall time
Giga-Cycles<br>
Ranks Threads Count (s) total
sum %<br>
-----------------------------------------------------------------------------<br>
Domain decomp. 6 6 2001 8.477
793.447 3.7<br>
DD comm. load 6 6 256 0.005
0.449 0.0<br>
DD comm. bounds 6 6 60 0.002
0.216 0.0<br>
Send X to PME 6 6 200001 32.588
3050.168 14.1<br>
Neighbor search 6 6 2001 6.639
621.393 2.9<br>
Launch GPU ops. 6 6 400002 14.686
1374.563 6.4<br>
Comm. coord. 6 6 198000 36.691
3434.263 15.9<br>
Force 6 6 200001 2.913
272.694 1.3<br>
Wait + Comm. F 6 6 200001 32.024
2997.400 13.9<br>
PME mesh * 1 6 200001 77.479
1208.657 5.6<br>
PME wait for PP * 119.009
1856.517 8.6<br>
Wait + Recv. PME F 6 6 200001 14.328
1341.122 6.2<br>
Wait PME GPU gather 6 6 200001 11.115
1040.397 4.8<br>
Wait Bonded GPU 6 6 2001 0.003
0.279 0.0<br>
Wait GPU NB nonloc. 6 6 200001 27.604
2583.729 11.9<br>
Wait GPU NB local 6 6 200001 0.548
51.333 0.2<br>
NB X/F buffer ops. 6 6 796002 11.095
1038.515 4.8<br>
Write traj. 6 6 21 0.105
9.851 0.0<br>
Update 6 6 200001 3.498
327.440 1.5<br>
Comm. energies 6 6 40001 2.947
275.863 1.3<br>
-----------------------------------------------------------------------------<br>
Total 198.094
21631.660 100.0<br>
-----------------------------------------------------------------------------<br>
(*) Note that with separate PME ranks, the walltime column
actually sums to<br>
twice the total reported, but the cycle count total and %
are correct.<br>
-----------------------------------------------------------------------------
</p>
<p> Core t (s) Wall t (s) (%)<br>
Time: 8319.867 198.094 4200.0<br>
(ns/day) (hour/ns)<br>
Performance: 87.232 0.275 </p>
<p> <br>
</p>
<p> I'm curious about the "Wait GPU NB nonloc" and "Wait GPU NB
local" part, which you can see in both cases, the wall time of
wait GPU NB local is very short but that of nonloc is pretty
long, and the wall time of Force in both cases is much shorter
than that of Wait GPU NB nonloc. Could you please give an
explanation of the these timing terms? And I'd appreciate it
very much if you can give some suggestions of reducing the time
consumption of that waiting! </p>
<p> <br>
</p>
<p> Sincerely, </p>
<p> Zhang </p>
<p> <br>
</p>
<p> <br>
</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
</body>
</html>