<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
Thanks, Mark. I've post it on the bugzilla. I'm glad to try the new assembly loops on BG/P as well.<br>Lanyuan<br><br><hr id="stopSpelling">From: mark.abraham@anu.edu.au<br>To: gmx-users@gromacs.org<br>Date: Wed, 9 Jun 2010 07:48:26 +1000<br>Subject: Re: RE: [gmx-users] PME problem on BG/P cluster<br><br>----- Original Message -----<br>From: LuLanyuan <lulanyuan@msn.com><br>Date: Wednesday, June 9, 2010 2:10<br>Subject: RE: [gmx-users] PME problem on BG/P cluster<br>To: gmx-users@gromacs.org<br><br><span> <style>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Verdana;}
</style> <table><tbody><tr><td class="ecxhmmessage"><p><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font> Dear Mark,<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>When I set <span>NOASSEMBLYLOOPS to 1, the simulation could be finished without any problem. So I guess it's related to the assembly loops for BG.</span><BR><br>OK, that's useful data. I understand that these loops are believed to work on BG/P. Knowing the code, I can think of no reason why the problem should be BG/P-specific. Please open a http://bugzilla.gromacs.org report, CC my email address, and attach your .tpr, and I'll try it on my BG/L.<br><br>I actually have well-tested ready-to-release updates to these kernels that perform up to about 10% better on PME on BG/L. Depending what we find above, and if you're interested, I could let you try them on BG/P, since I
don't have the ability to test on BG/P.<br><br>Mark<br><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>From: mark.abraham@anu.edu.au<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>To: gmx-users@gromacs.org<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>Date: Sat, 5 Jun 2010 03:31:33 +1000<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>Subject: Re: [gmx-users] PME problem on BG/P cluster<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal
; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>----- Original Message -----<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>From: LuLanyuan <lulanyuan@msn.com><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>Date: Saturday, June 5, 2010 2:01<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>Subject: [gmx-users] PME problem on BG/P cluster<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>To: gmx-users@gromacs.org<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240
); font-size: 14px;">> </font><br><span><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font> <style>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Verdana;}
</style> <table><tbody><tr><td class="ecxecxhmmessage"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> Hello,<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>I got a weird problem by running Gromacs 4.0.7 on a BG/P machine ("Intrepid" at Argonne national lab).<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>The simulated system is a box of SPC water with 648,000 atoms and all MD simulations w
ere performed on 256 CPU cores with MPI. The compiling environment was Linux with IBM compiler and libs.<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>I first compile the code with flags suggested on the Wiki, such as:<br><font style="font-style: normal; font-weight: normal; font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>./configure --prefix=$PREFIX \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> &nbs
p; --host=ppc \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> --build=ppc64 \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> --disable-software-sqrt \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 2
48, 240); font-size: 14px;">> </font>> </font> & nbsp; --enable-ppc-sqrt=1 \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> &n bsp; --enable-ppc-altivec \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>The wiki doesn't suggest this, and it'
s for another architecture entirely. I don't know if it's causing a problem, but you should see what kernels the .log file reports it is t
rying to use.<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> --enable-bluegene \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> &n bsp;  
; --disable-fortran \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> --enable-mpi \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> --with-fft=fftpack \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(24
5, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>This is a waste of a good BlueGene :-) Use FFTW, which has been optimized for BlueGene. [Edit: ah I see why you tried this]<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> --wit hout-x \<br><font style="font-style: normal; font-weight: normal; background-col
or: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>&nbs p; CC="mpixlc_r" \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> CFLAGS="-O3 -qarc h=450d -qtune=450" \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> &nbs
p; MPICC="mpixlc_r"<br><font style="font-style: normal; font-weight: nor
mal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> CXX="mpixlcxx_r"<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> CXXFLAGS="-O3 -qarch=450 -qtune=450"<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>  
; F77="mpixlf77_r"<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> FFLAGS="-O3 -qarch=450 -qtune=450"<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> LIBS="-lmass"<br><font style="font-style: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-w
eight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>Here I used fftpack to ensure that the problem is not due to the fftw lib. I got the water system running will with Cut-off for electrostatics. However, the systems always crashed after a few ( ~100) steps if I used PME. The same system with same PME option runs fine on other non-blue gene clusters I tested. <br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>The error message I got was sth like <br><font style="font-style: normal; font-weight:
normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>t = 0.100 ps: Water molecule starting at atom 403468 can not be settled.<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240);"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>Check for bad contacts and/or reduce the timestep.<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>Wrote pdb files with previous and current coordinates.<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-sty
le: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font><br><font style="font-sty
le: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>and<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>2 particles communicated to PME node 63 are more than a cell length out of the domain decomposition cell of their charge group<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240);"><font style="font-style: normal; font-weight:
normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> < /font>From .log file, the kinetic energy is increasing and turned to be "nan". So the system is exploding.<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>I found if I turned off the blue gene optimizatio
ns during configure, the water system could be run without problem. For example, I used<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> --enable-software-sqrt \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> - -disable-ppc-sqrt \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: no
rmal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font> --disable-bluegene \<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>and everything else was the same. <br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>I suspect there was an issue regarding the blue gene specific code and PME.<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;"><font style="font-style: normal;
font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>> </font>Could anyone give any comments?<br><f
ont style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>First, try without --enable-ppc-altivec in case that's confusing things. Then try setting the enviroment variable NOASSEMBLYLOOPS to 1 before mdrun to see whether the issue really is s pecific to the BlueGene-kernel. (Consult mpirun documentation about how to set env vars suitably) You might also try compiling with lower optimization levels, to see if its a compiler/optimization issue. Depending what you find above, there are other things to try.<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font
-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>Mark<br><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font><br></font><BR></td></tr></tbody></table></span><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>                                            <br><hr><font style="font-style: normal; font-weight: normal; background-color: rgb(245, 248, 240); font-size: 14px;">> </font>聊天+搜索+邮箱 想要轻松出游,手机MSN帮你搞定! <a href="http://3g.msn.cn/">立刻下载!</a></td></tr></tbody></table> </p></span>> -- <br>> gmx-users mailing list gmx-users@gromacs.org<br>&g
t; http://lists.gromacs.org/mailman/listinfo/gmx-users<br>> Please search the archive at http://www.gromacs.org/search <br>> before posting!<br>> Please don't post (un)subscribe requests to the list. Use the <br>> www interface or send it to gmx-users-request@gromacs.org.<br>> Can't post? Read http://www.gromacs.org/mailing_lists/users.php                                            <br /><hr />更多热辣资讯尽在新版MSN首页! <a href='http://cn.msn.com/' target='_new'>立刻访问!</a></body>
</html>