What MVAPICH version are you using?<div><br></div><div>Are you using a TPR file you know is running fine on some other machine?</div><div><br></div><div>Does the 4.5.2 version they installed run correct? If so what is the configure line they used?</div>
<div><br></div><div>Roland<br><br><div class="gmail_quote">On Thu, Dec 9, 2010 at 5:14 PM, J. Nathan Scott <span dir="ltr"><<a href="mailto:scottjn@chemistry.montana.edu">scottjn@chemistry.montana.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Hello gmx users! I realize this may be a touch off topic, but I am<br>
hoping that someone out there can offer some advice on how to build<br>
Gromacs for parallel use on a Teragrid site. Our group is currently<br>
using Abe on Teragrid, and unfortunately the latest version of Gromacs<br>
compiled for public use on Abe is 4.0.2. Apparently installation of<br>
4.5.3 is at least on the to-do list for Abe, but we would very much<br>
like to use 4.5.3 now if we can get this issue figured it out.<br>
<br>
I have built a parallel version of mdrun using Abe installed versions<br>
of fftw3 and mvapich2 using the following commands:<br>
<br>
setenv CPPFLAGS "-I/usr/apps/math/fftw/fftw-3.1.2/gcc/include/<br>
-I/usr/apps/mpi/marmot_mvapich2_intel/include"<br>
setenv LDFLAGS "-L/usr/apps/math/fftw/fftw-3.1.2/gcc/lib<br>
-L/usr/apps/mpi/marmot_mvapich2_intel/lib"<br>
./configure --enable-mpi --enable-float --prefix=/u/ac/jnscott/gromacs<br>
--program-suffix=_mpi<br>
make -j 8 mdrun && make install-mdrun<br>
<br>
My PBS script file looks like the following:<br>
<br>
-------------------------------<br>
#!/bin/csh<br>
#PBS -l nodes=2:ppn=8<br>
#PBS -V<br>
#PBS -o pbs_nvt.out<br>
#PBS -e pbs_nvt.err<br>
#PBS -l walltime=2:00:00<br>
#PBS -N gmx<br>
cd /u/ac/jnscott/1stn/1stn_wt/oplsaa_spce<br>
mvapich2-start-mpd<br>
setenv NP `wc -l ${PBS_NODEFILE} | cut -d'/' -f1`<br>
setenv MV2_SRQ_SIZE 4000<br>
mpirun -np ${NP} mdrun_mpi -s nvt.tpr -o nvt.trr -x nvt.xtc -cpo<br>
nvt.cpt -c nvt.gro -e nvt.edr -g nvt.log -dlb yes<br>
-------------------------------<br>
<br>
Unfortunately my runs always fail in the same manner. The log file<br>
simply ends, as you can see below. It appears that Gromacs is picking<br>
up the correct number of nodes specified in the PBS script, but then<br>
something causes it to quit abruptly with no error message.<br>
<br>
-------------------------------<br>
<snip><br>
Initializing Domain Decomposition on 16 nodes<br>
Dynamic load balancing: yes<br>
Will sort the charge groups at every domain (re)decomposition<br>
Initial maximum inter charge-group distances:<br>
two-body bonded interactions: 0.526 nm, LJ-14, atoms 1735 1744<br>
multi-body bonded interactions: 0.526 nm, Ryckaert-Bell., atoms 1735 1744<br>
Minimum cell size due to bonded interactions: 0.578 nm<br>
Maximum distance for 5 constraints, at 120 deg. angles, all-trans: 0.820 nm<br>
Estimated maximum distance required for P-LINCS: 0.820 nm<br>
This distance will limit the DD cell size, you can override this with -rcon<br>
Guess for relative PME load: 0.27<br>
Will use 10 particle-particle and 6 PME only nodes<br>
This is a guess, check the performance at the end of the log file<br>
Using 6 separate PME nodes<br>
Scaling the initial minimum size with 1/0.8 (option -dds) = 1.25<br>
Optimizing the DD grid for 10 cells with a minimum initial size of 1.025 nm<br>
The maximum allowed number of cells is: X 5 Y 5 Z 4<br>
Domain decomposition grid 2 x 5 x 1, separate PME nodes 6<br>
PME domain decomposition: 2 x 3 x 1<br>
Interleaving PP and PME nodes<br>
This is a particle-particle only node<br>
<br>
Domain decomposition nodeid 0, coordinates 0 0 0<br>
<br>
Using two step summing over 2 groups of on average 5.0 processes<br>
<br>
Table routines are used for coulomb: TRUE<br>
Table routines are used for vdw: FALSE<br>
Will do PME sum in reciprocal space.<br>
<br>
<snip><br>
<br>
Will do ordinary reciprocal space Ewald sum.<br>
Using a Gaussian width (1/beta) of 0.320163 nm for Ewald<br>
Cut-off's: NS: 1 Coulomb: 1 LJ: 1<br>
Long Range LJ corr.: <C6> 3.3589e-04<br>
System total charge: 0.000<br>
Generated table with 1000 data points for Ewald.<br>
Tabscale = 500 points/nm<br>
Generated table with 1000 data points for LJ6.<br>
Tabscale = 500 points/nm<br>
Generated table with 1000 data points for LJ12.<br>
Tabscale = 500 points/nm<br>
Generated table with 1000 data points for 1-4 COUL.<br>
Tabscale = 500 points/nm<br>
Generated table with 1000 data points for 1-4 LJ6.<br>
Tabscale = 500 points/nm<br>
Generated table with 1000 data points for 1-4 LJ12.<br>
Tabscale = 500 points/nm<br>
<br>
Enabling SPC-like water optimization for 6952 molecules.<br>
<br>
Configuring nonbonded kernels...<br>
Configuring standard C nonbonded kernels...<br>
Testing x86_64 SSE2 support... present.<br>
<br>
Removing pbc first time<br>
<br>
Initializing Parallel LINear Constraint Solver<br>
<br>
<snip><br>
Linking all bonded interactions to atoms<br>
There are 9778 inter charge-group exclusions,<br>
will use an extra communication step for exclusion forces for PME<br>
<br>
The maximum number of communication pulses is: X 1 Y 2<br>
The minimum size for domain decomposition cells is 0.827 nm<br>
The requested allowed shrink of DD cells (option -dds) is: 0.80<br>
The allowed shrink of domain decomposition cells is: X 0.35 Y 0.73<br>
The maximum allowed distance for charge groups involved in interactions is:<br>
non-bonded interactions 1.000 nm<br>
two-body bonded interactions (-rdd) 1.000 nm<br>
multi-body bonded interactions (-rdd) 0.827 nm<br>
atoms separated by up to 5 constraints (-rcon) 0.827 nm<br>
<br>
<br>
Making 2D domain decomposition grid 2 x 5 x 1, home cell index 0 0 0<br>
<br>
Center of mass motion removal mode is Linear<br>
We have the following groups for center of mass motion removal:<br>
0: rest<br>
<br>
++++ PLEASE READ AND CITE THE FOLLOWING REFERENCE ++++<br>
G. Bussi, D. Donadio and M. Parrinello<br>
Canonical sampling through velocity rescaling<br>
J. Chem. Phys. 126 (2007) pp. 014101<br>
-------- -------- --- Thank You --- -------- --------<br>
-----------------------------------------------------------<br>
<br>
My PBS error file is not of much help either I fear, an example of<br>
such a file is pasted below:<br>
<br>
-----------------------------------<br>
stty: standard input: Invalid argument<br>
stty: standard input: Invalid argument<br>
NNODES=16, MYRANK=0, HOSTNAME=abe0828<br>
NNODES=16, MYRANK=2, HOSTNAME=abe0828<br>
NNODES=16, MYRANK=12, HOSTNAME=abe0828<br>
NNODES=16, MYRANK=4, HOSTNAME=abe0828<br>
NNODES=16, MYRANK=10, HOSTNAME=abe0828<br>
NNODES=16, MYRANK=8, HOSTNAME=abe0828<br>
NNODES=16, MYRANK=6, HOSTNAME=abe0828<br>
NNODES=16, MYRANK=14, HOSTNAME=abe0828<br>
NODEID=0 argc=17<br>
NODEID=2 argc=17<br>
NODEID=4 argc=17<br>
NODEID=10 argc=17<br>
NODEID=12 argc=17<br>
NODEID=6 argc=17<br>
NODEID=14 argc=17<br>
NODEID=8 argc=17<br>
NNODES=16, MYRANK=5, HOSTNAME=abe0825<br>
NNODES=16, MYRANK=13, HOSTNAME=abe0825<br>
:-) G R O M A C S (-:<br>
<br>
NNODES=16, MYRANK=9, HOSTNAME=abe0825<br>
NNODES=16, MYRANK=11, HOSTNAME=abe0825<br>
Great Red Oystrich Makes All Chemists Sane<br>
<br>
:-) VERSION 4.5.3 (-:<br>
<br>
<snip><br>
Back Off! I just backed up nvt.log to ./#nvt.log.2#<br>
Reading file nvt.tpr, VERSION 4.5.3 (single precision)<br>
<br>
Will use 10 particle-particle and 6 PME only nodes<br>
This is a guess, check the performance at the end of the log file<br>
Making 2D domain decomposition 2 x 5 x 1<br>
<br>
Back Off! I just backed up nvt.edr to ./#nvt.edr.2#<br>
----------------------------------------------<br>
<br>
The non-Torque section of the PBS log file is below:<br>
<br>
-----------------------------------------------<br>
Warning: no access to tty (Bad file descriptor).<br>
Thus no job control in this shell.<br>
running mpdallexit on abe0828<br>
LAUNCHED mpd on abe0828 via<br>
RUNNING: mpd on abe0828<br>
LAUNCHED mpd on abe0825 via abe0828<br>
RUNNING: mpd on abe0825<br>
abe0828_43972 (10.1.67.66)<br>
abe0825_37571 (10.1.67.63)<br>
rank 1 in job 1 abe0828_43972 caused collective abort of all ranks<br>
exit status of rank 1: killed by signal 9<br>
rank 0 in job 1 abe0828_43972 caused collective abort of all ranks<br>
exit status of rank 0: killed by signal 9<br>
-------------------------------------------------<br>
<br>
I would should also note that both .edr and .trr files are created in<br>
the working directory, but both files are 0 bytes.<br>
<br>
Like I said, I realize this question is perhaps a bit off the topic of<br>
Gromacs exclusively, but I hope that someone can offer some tips or<br>
spot any obvious problems with my method that I have not noticed and<br>
would sincerely appreciate any help you can offer a novice.<br>
<br>
Best Wishes,<br>
-Nathan<br>
<br>
<br>
----------<br>
J. Nathan Scott, Ph.D.<br>
Postdoctoral Fellow<br>
Department of Chemistry and Biochemistry<br>
Montana State University<br>
<font color="#888888">--<br>
gmx-users mailing list <a href="mailto:gmx-users@gromacs.org">gmx-users@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-users" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-users</a><br>
Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/Search" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/Search</a> before posting!<br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-users-request@gromacs.org">gmx-users-request@gromacs.org</a>.<br>
Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309<br>
</div>