<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
Hi,<br><br>One issue could be MPI memory usage.<br>I have noticed that many MPI implementations use an amount of memory<br>per process that is quadratic (!) in the number of processes involved.<br>This can quickly get out of hand. But 28 GB is a lot of memory.<br><br>One thing that might help slightly is to not use double precision,<br>which is almost never required. This will also make your simulations<br>a factor 1.4 faster.<br><br>Berk<br><br>> Date: Tue, 2 Feb 2010 18:55:37 +0100<br>> From: breuerss@uni-koeln.de<br>> To: gmx-users@gromacs.org<br>> Subject: [gmx-users] Replica Exchange MD on more than 64 processors<br>> <br>> Dear list<br>> <br>> I recently came up with a problem concerning a replica exchange simulation. The simulation is run<br>> with gromacs-mpi in Version 4.0.7 compiled with following flags<br>> --enable-threads --enable-mpi --with-fft=mkl -enable-double,<br>> <br>> intel compiler version 11.0<br>> mvapich version 1.1.0<br>> mkl version 10.1<br>> <br>> The program is working fine in this cluster evironment consisting of 32 nodes with 8 processors and<br>> 32GB each. I've already run several simulations using the MPI feature.<br>> <br>> It seems that I stuck in a similar problem that was already announced on this list by bharat v.<br>> adkar in december 2009 without an eventual solution:<br>> <br>> http://www.mail-archive.com/gmx-users@gromacs.org/msg27175.html<br>> <br>> I am doing a replica exchange simulation on a simulation box with 5000 molecules (81 atoms each) and<br>> 4 different temperatures. The simulation runs nicely with 64 processors (8 nodes) but stops with an<br>> error message on 128 processors (16 nodes).<br>> <br>> Taking the following four points into account:<br>> <br>> 1. every cluster node has at least 28GB memory in a usable way available<br>> 2. the system I am working with should only use<br>> 5000*81*900B=347.614MB (according to the FAQ)<br>> 3. even if every replica (4) is run on the same node the memory usage<br>> should be less than 2GB<br>> 4. the simulation works fine with 64 processors<br>> <br>> it seems to me the following error<br>> <br>> -------------------------------------------------------<br>> Program mdrun, VERSION 4.0.7<br>> Source code file: smalloc.c, line: 179<br>> <br>> Fatal error:<br>> Not enough memory. Failed to realloc 790760 bytes for nlist->jjnr, nlist->jjnr=0xae70b7b0<br>> (called from file ns.c, line 503)<br>> -------------------------------------------------------<br>> <br>> has to be caused by another issue than missing memory.<br>> <br>> I am wondering if there is anyone else who is still facing the same problem or has already found a<br>> solution for this issue.<br>> <br>> Kind regards<br>> <br>> Sebastian<br>> <br>> -- <br>> _____________________________________________________________________________<br>> <br>> Sebastian Breuers Tel: +49-221-470-4108<br>> EMail: breuerss@uni-koeln.de<br>> <br>> Universität zu Köln University of Cologne<br>> Department für Chemie Department Chemistry<br>> Organische Chemie university of Cologne<br>> <br>> Greinstr. 4 Greinstr. 4<br>> D-50939 Köln D-50939 Cologne, Federal Rep. of Germany<br>> _____________________________________________________________________________<br>> <br>> -- <br>> gmx-users mailing list gmx-users@gromacs.org<br>> http://lists.gromacs.org/mailman/listinfo/gmx-users<br>> Please search the archive at http://www.gromacs.org/search before posting!<br>> Please don't post (un)subscribe requests to the list. Use the <br>> www interface or send it to gmx-users-request@gromacs.org.<br>> Can't post? Read http://www.gromacs.org/mailing_lists/users.php<br> <br /><hr />New Windows 7: Simplify what you do everyday. <a href='http://windows.microsoft.com/shop' target='_new'>Find the right PC for you.</a></body>
</html>