<html>
<head>
<style>
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
FONT-SIZE: 10pt;
FONT-FAMILY:Tahoma
}
</style>
</head>
<body class='hmmessage'><div style="text-align: left;"><div style="text-align: left;">Hi,<br><br>Your PME nodes seem to be running one order of magnitude slower than they should.<br>This could be explained by a memory usage problem, which is indicated by the out<br>of memory error.<br><br>I am running systems on 8 cores for 24 hours and the memory usage stays constant<br>after the first few steps.<br>I have no clue what the problem could be.<br>I am also looking into a dynamic load balancing problem which only seems to happen<br>on the Cray XT4 and for which I, up till now, also have no clue what could cause this.<br><br>What compiler (and version) are your using?<br><br>Berk<br></div></div><br><br><hr id="stopSpelling">> Date: Tue, 30 Sep 2008 18:22:44 +0200<br>> From: st01397@student.uib.no<br>> To: gmx-users@gromacs.org<br>> Subject: RE: [gmx-users] Possible bug in parallelization, PME or load-balancing on Gromacs 4.0_rc1 ??<br>> CC: gmx3@hotmail.com<br>> <br>> I have some (hopefully) clarifying commments to my previous post now:<br>> <br>> First to answer your question regarding pme.c. My compilation was done <br>> from v. 1.125<br>> ------------<br>> Line 1037-<br>> if ((kx>0) || (ky>0)) {<br>> kzstart = 0;<br>> } else {<br>> kzstart = 1;<br>> p0++;<br>> }<br>> ------<br>> As you can see the p0++; line is there.<br>> <br>> Now here are some additional points:<br>> <br>> On Mon, 29 Sep 2008, Bjørn Steen Sæthre wrote:<br>> <br>> > The only Error message I can find is the rather cryptic::<br>> ><br>> > NOTE: Turning on dynamic load balancing<br>> ><br>> > _pmii_daemon(SIGCHLD): PE 4 exit signal Killed<br>> > [NID 1412]Apid 159787: initiated application termination<br>> ><br>> > There are no error's apart from that.<br>> <br>> > Furthermore I can now report that this error is endemic in all my sims<br>> > using harmonic position restraints in GROMACS 4.0_beta1 and GMX<br>> > 4.0_rc1.<br>> ><br>> > About core dumps. I will talk to our HPC staff, and get back to you with<br>> > something more substantial I hope.<br>> >><br>> <br>> OK, I have gotten some info from our HPC staff, they checked another job of<br>> mine which crashed in the exact same fashion, with the exact same starting<br>> run-topology and node configuration.<br>> They found some more info in the admin's log:<br>> <br>> > Hi,<br>> > this job got an OOM (out of memory), which is only recorded in the<br>> > system logs, not available directly to users:<br>> <br>> > [2008-09-29 17:18:18][c11-0c0s1n0]Out of memory: Killed process 8888<br>> > (parmdrun).<br>> <br>> I can also add that I have been able to stabilize the engine, by altering the<br>> cut-offs and lowering the total PME-load of the run, at the expense of far<br>> greater computational inefficiency.<br>> <br>> That is I went from unstable < to stable > as in the following diff on <br>> the mdp-files:<br>> -----------------------------<br>> 21c21<br>> < rlist = 0.9<br>> ---<br>> > rlist = 1.0<br>> 24c24<br>> < rcoulomb = 0.9<br>> ---<br>> > rcoulomb = 1.0<br>> 26c26<br>> < rvdw = 0.9<br>> ---<br>> > rvdw = 1.0<br>> 28,30c28,31<br>> < fourier_nx = 60<br>> < fourier_ny = 40<br>> < fourier_nz = 40<br>> ---<br>> > fourier_nx = 48<br>> > fourier_ny = 32<br>> > fourier_nz = 32<br>> 35c36<br>> ------------------------------<br>> That is, the PME-workload went from 1/2 of nodes to 1/3 of them since I was<br>> using exactly the same startup configuration ---------------------<br>> <br>> This however, while enhancing stability, the output rate slowed down<br>> appreciably. And as shown in the log output, the reason is clear:<br>> ------------------------------------------------------------<br>> Making 2D domain decomposition 8 x 4 x 1<br>> starting mdrun 'Propane-hydrate prism (2x2x3 UC)'<br>> 2000000 steps, 4000.0 ps.<br>> Step 726095: Run time exceeded 3.960 hours, will terminate the run<br>> <br>> Step 726100: Run time exceeded 3.960 hours, will terminate the run<br>> <br>> Average load imbalance: 26.7 %<br>> Part of the total run time spent waiting due to load imbalance: 1.5 %<br>> Average PME mesh/force load: 9.369<br>> Part of the total run time spent waiting due to PP/PME imbalance: 57.5 %<br>> <br>> NOTE: 57.5 % performance was lost because the PME nodes<br>> had more work to do than the PP nodes.<br>> You might want to increase the number of PME nodes<br>> or increase the cut-off and the grid spacing.<br>> <br>> <br>> Parallel run - timing based on wallclock.<br>> <br>> NODE (s) Real (s) (%)<br>> Time: 5703.000 5703.000 100.0<br>> 1h35:03<br>> (Mnbf/s) (GFlops) (ns/day) (hour/ns)<br>> Performance: 29.593 8.566 60.600 0.396<br>> <br>> gcq#0: Thanx for Using GROMACS - Have a Nice Day<br>> -----------------------------------------------<br>> <br>> <br>> One thing more is odd here though.<br>> In the startup script I allocated 4 hours, and set -maxh 4:<br>> <br>> -----------------------------------------------<br>> #PBS -l walltime=4:00:00,mppwidth=48,mppnppn=4<br>> cd /work/bjornss/pmf/structII/hydrate_annealing/heatup_400K_2nd<br>> source $HOME/gmx_latest_290908/bin/GMXRC<br>> aprun -n 48 parmdrun -s topol.tpr -maxh 4 -npme 16<br>> exit $?<br>> -----------------------<br>> <br>> why the wallclock inconsistency (ie. wallclock is 1:35:03 which does not<br>> correspond to the note of 3.96 hours exceeded.)<br>> <br>> <br>> <br>> I hope this is helpful in resolving the issue brought up originally. (Might<br>> there be a possible memory leak somewhere?)<br>> <br>> Regards<br>> Bjørn<br>> <br>> <br>> PhD-student<br>> Insitute of Physics & Tech.- University of Bergen<br>> Allegt. 55,<br>> 5007 Bergen<br>> Norway<br>> <br>> Tel(office): +47 55582869<br>> Cell: +47 99253386<br>> _______________________________________________<br>> gmx-users mailing list gmx-users@gromacs.org<br>> http://www.gromacs.org/mailman/listinfo/gmx-users<br>> Please search the archive at http://www.gromacs.org/search before posting!<br>> Please don't post (un)subscribe requests to the list. Use the <br>> www interface or send it to gmx-users-request@gromacs.org.<br>> Can't post? Read http://www.gromacs.org/mailing_lists/users.php<br><br /><hr />Express yourself instantly with MSN Messenger! <a href='http://clk.atdmt.com/AVE/go/onm00200471ave/direct/01/' target='_new'>MSN Messenger</a></body>
</html>