<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.16.1">
</HEAD>
<BODY>
Hello Gromacs users.<BR>
<BR>
In the CVS version, I experience that mdrun crashes instantly when run<BR>
in parallel across nodes (for any simulation system). The cluster<BR>
consists of 8 nodes with Intel 6600 Quad-Core processors. As long as a<BR>
job is run on a single node (using 1,2 or 4 CPU:s) everything works<BR>
fine but when trying to run on several nodes mdrun crashes directly<BR>
with the following error message (no output or log files are written to disk):<BR>
<BR>
> Getting Loaded...<BR>
> Reading file topol.tpr, VERSION 3.3.99_development_20071104 (single precision)<BR>
> Loaded with Money<BR>
><BR>
> [warhol8:29695] *** An error occurred in MPI_Allreduce<BR>
> [warhol8:29695] *** on communicator MPI_COMM_WORLD<BR>
> [warhol8:29695] *** MPI_ERR_COMM: invalid communicator<BR>
> [warhol8:29695] *** MPI_ERRORS_ARE_FATAL (goodbye)<BR>
<BR>
For the 1024 DPPC benchmark system the following two commands were<BR>
used to start the simulation (default names on input files):<BR>
<BR>
> /opt/gromacs/cvs/bin/grompp<BR>
> /opt/openmp/1.2.4/bin/mpirun --hostfile hostfile /opt/gromacs/cvs/bin/mdrun_mpi -v -dd 2 2 2<BR>
<BR>
where hostfile contains two specific nodes with 4 slots each.<BR>
<BR>
The OS is Ubuntu 7.10 x86_64 on all nodes. mdrun_mpi is compiled with<BR>
OpenMPI 1.2.4 but I have also tried with LAM/MPI 7.1.2 and it crashes<BR>
in the same manner with an identical error message. Furthermore I have<BR>
tried a static compilation on another cluster (Intel Xeon EM64T<BR>
Processors) and copied the files to our cluster with the same<BR>
result. I have searched the web for this error and there are some<BR>
suggestions that this may be related to 64 bit architecture, see e.g.<BR>
<BR>
<A HREF="http://www.open-mpi.org/community/lists/users/2006/04/0978.php">http://www.open-mpi.org/community/lists/users/2006/04/0978.php</A><BR>
<BR>
The MPI installation on the cluster works for the 3.3.2 version of<BR>
Gromacs and also for some simple test programs for MPI such as nodes<BR>
writing out their name and rank.<BR>
<BR>
Does anyone have any ideas on the origins of these crashes and/or<BR>
suggestions on how to resolve them?<BR>
<BR>
Regards<BR>
Erik Brandt<BR>
<BR>
Ph.D. Student<BR>
Theoretical Physics, KTH, Stockholm, Sweden<BR>
<BR>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
-- <BR>
Erik Brandt <<A HREF="mailto:erikb@theophys.kth.se">erikb@theophys.kth.se</A>><BR>
KTH
</TD>
</TR>
</TABLE>
</BODY>
</HTML>