Hi Mark and Alexey,<div><br></div><div>Thank you for taking the time to write the responses. Following is the info about the cluster</div><div><br></div><div><div>[choubey@hpc-login2 ~]$ uname -a</div><div>Linux hpc-login2 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux</div>
<div><br></div><div><div>[choubey@hpc2000 ~]$ mpiexec --version</div><div>Version 0.82, configure options: '--host=x86_64-redhat-linux-gnu' '--build=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/usr/com' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-pbs=/usr/lib64/torque' '--disable-p4-shmem' 'CFLAGS=-O2 -g' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux'</div>
<div><br></div><div><div>[choubey@hpc2000 ~]$ ifort --version</div><div>ifort (IFORT) 10.0 20070426</div><div>Copyright (C) 1985-2007 Intel Corporation. šAll rights reserved.</div><div><br></div><div><div>[choubey@hpc2000 ~]$ icc --version</div>
<div>icc (ICC) 10.0 20070426</div><div>Copyright (C) 1985-2007 Intel Corporation. šAll rights reserved.</div><div><br></div><div><br></div><div>Thanks for all the help.</div><div><br></div><div>Amit</div><div><br></div></div>
<div><br></div></div></div><div><br></div></div><div><br><br><div class="gmail_quote">2010/3/3 Alexey Shvetsov <span dir="ltr"><<a href="mailto:alexxyum@gmail.com">alexxyum@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi<br>
<br>
Looks like your system simply runs out of memory. So power cycling nodes isnt<br>
needed. If your cluster runs linux then it already has OOM Killer that will<br>
kill processes that runs out of memory. Also having swap on nodes is a good<br>
idea even with huge amount šof memory.<br>
Memory usage for mpi processes will strongly depend on mpi implentation<br>
because some of them are usualy caching slave process memory (like usualy do<br>
mvapich2)<br>
<br>
So can you provide info about youre cluster setup.<br>
OS version (including kernel version) šuname -a<br>
mpi version šmpirun --version or mpiexec --version<br>
also compiler version that was used for compiling gromacs<br>
<div class="im"><br>
On þÅÔ×ÅÒÇ 04 ÍÁÒÔÁ 2010 03:15:53 Amit Choubey wrote:<br>
> Hi Roland,<br>
><br>
> I was using 32 nodes with 8 cores, each with 16 Gb memory. The system was<br>
> about 154 M particles. This should be feasible according to the numbers.<br>
> Assuming that it takes 50bytes per atoms and 1.76 KB per atom per core then<br>
><br>
> Masternode -> (50*154 M + 8*1.06)bytes ~ 16GB (There is no leverage here)<br>
> All other nodes 8*1.06 ~ 8.5 GB<br>
><br>
> I am planning to try the same run on 64 nodes with 8 cores each again but<br>
> not until i am a little more confident. The problem is if gromacs crashes<br>
> due to memory it makes the nodes to hang and people have to recycle the<br>
> power supply.<br>
><br>
><br>
> Thank you,<br>
><br>
</div>--<br>
Best Regards,<br>
Alexey 'Alexxy' Shvetsov<br>
Petersburg Nuclear Physics Institute, Russia<br>
Department of Molecular and Radiation Biophysics<br>
Gentoo Team Ru<br>
Gentoo Linux Dev<br>
mailto:<a href="mailto:alexxyum@gmail.com">alexxyum@gmail.com</a><br>
mailto:<a href="mailto:alexxy@gentoo.org">alexxy@gentoo.org</a><br>
mailto:<a href="mailto:alexxy@omrb.pnpi.spb.ru">alexxy@omrb.pnpi.spb.ru</a><br>
<br>--<br>
gmx-users mailing list š š<a href="mailto:gmx-users@gromacs.org">gmx-users@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-users" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-users</a><br>
Please search the archive at <a href="http://www.gromacs.org/search" target="_blank">http://www.gromacs.org/search</a> before posting!<br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-users-request@gromacs.org">gmx-users-request@gromacs.org</a>.<br>
Can't post? Read <a href="http://www.gromacs.org/mailing_lists/users.php" target="_blank">http://www.gromacs.org/mailing_lists/users.php</a><br></blockquote></div><br></div>