<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi,<br>
<br>
This issue is triggered when restrarting from a checkpoint file
that is not from a step % nstlist == 0.<br>
A one-line fix was merged into release-5.1 in March:<br>
<a class="moz-txt-link-freetext" href="https://gerrit.gromacs.org/#/c/5721/">https://gerrit.gromacs.org/#/c/5721/</a><br>
<br>
You can apply this fix, or run with -notunepme. Or you can run
with -notunepme and press Crtl-C once to get a new checkpoint file
with step % nstlist == 0 and continue from that.<br>
<br>
Cheers,<br>
<br>
Berk<br>
<br>
On 05/29/2016 01:17 AM, Yorquant Wang wrote:<br>
</div>
<blockquote
cite="mid:CAEZVKDO=D+4i_2bCvuv-+=pc=nK53mAUWZ_4HJ30kjs-mTGrAA@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px">Dear All,<br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px"><br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px">I recently met a error:"Software
inconsistency error: pme_loadbal_do called at an interval
!= nstlist" when I try to restart one of jobs.</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px"><font
face="Calibri,Arial,Helvetica,sans-serif">I used GMX
5.1.2 on K80 GPU nodes with haswell CPU.</font><br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px"><br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px">Is there anyone know how to solve
this error?<br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px">Thank you very much!<br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px"><br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px">cheers<br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font
face="Calibri,Arial,Helvetica,sans-serif" size="2"><span
style="font-size:16px">Yukun Wang<br>
</span></font></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">below
is the output log files:<br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
:-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
GROMACS is written by:</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Emile Apol Rossen Apostolov Herman J.C. Berendsen
Par Bjelkmar</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Aldert
van Buuren Rudi van Drunen Anton Feenstra Sebastian
Fritsch</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Gerrit Groenhof Christoph Junghans Anca Hamuraru
Vincent Hindriksen</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Dimitrios
Karkoulis Peter Kasson Jiri Kraus Carsten
Kutzner</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Per Larsson Justin A. Lemkul Magnus Lundborg Pieter
Meulenhoff</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Erik Marklund Teemu Murtola Szilard Pall
Sander Pronk</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Roland Schulz Alexey Shvetsov Michael Shirts
Alfons Sijbers</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Peter Tieleman Teemu Virolainen Christian Wennberg
Maarten Wolf</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
and the project leaders:</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Mark Abraham, Berk Hess, Erik Lindahl, and David van der
Spoel</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Copyright
(c) 1991-2000, University of Groningen, The Netherlands.</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Copyright
(c) 2001-2015, The GROMACS development team at</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Uppsala
University, Stockholm University and</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">the
Royal Institute of Technology, Sweden.</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">check
out <a moz-do-not-send="true" href="http://www.gromacs.org">http://www.gromacs.org</a>
for more information.</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">GROMACS
is free software; you can redistribute it and/or modify it</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">under
the terms of the GNU Lesser General Public License</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">as
published by the Free Software Foundation; either version 2.1</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">of
the License, or (at your option) any later version.</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">GROMACS:
gmx mdrun, VERSION 5.1.2</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Executable:
/lustre/usr/gromacs/5.1-icc15-impi5-gpu/bin/gmx_mpi</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Data
prefix: /lustre/usr/gromacs/5.1-icc15-impi5-gpu</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Command
line:</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
gmx_mpi mdrun -deffnm 200DPPC-16-WT_COO-_150mV_2 -append -cpi
-v -resethway -noconfout</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">No
previous checkpoint file present, assuming this is a new run.</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Number
of logical cores detected (16) does not match the number
reported by OpenMP (8).</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Consider
setting the launch configuration manually!</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Running
on 1 node with total 16 cores, 16 logical cores, 2 compatible
GPUs</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Hardware
detected on host gpu35 (the node of MPI rank 0):</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
CPU info:</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Vendor: GenuineIntel</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brand: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
SIMD instructions most likely to fit this hardware: AVX_256</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
SIMD instructions selected at GROMACS compile time: AVX_256</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
GPU info:</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Number of GPUs detected: 2</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
#0: NVIDIA Tesla K20m, compute cap.: 3.5, ECC: yes, stat:
compatible</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
#1: NVIDIA Tesla K20m, compute cap.: 3.5, ECC: yes, stat:
compatible</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Reading
file 200DPPC-16-WT_COO-_150mV_2.tpr, VERSION 5.0.4 (single
precision)</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Note:
file tpx version 100, software tpx version 103</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Changing
nstlist from 25 to 40, rlist from 1.038 to 1.08</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">The
number of OpenMP threads was set by environment variable
OMP_NUM_THREADS to 8</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Using
2 MPI processes</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Using
8 OpenMP threads per MPI process</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">On
host gpu35 2 compatible GPUs are present, with IDs 0,1</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">On
host gpu35 2 GPUs auto-selected for this run.</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Mapping
of GPU IDs to the 2 PP ranks in this node: 0,1</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">NOTE:
GROMACS was configured without NVML support hence it can not
exploit</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
application clocks of the detected Tesla K20m GPU to
improve performance.</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Recompile with the NVML library (compatible with the
driver used) or set application clocks manually.</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Non-default
thread affinity set probably by the OpenMP library,</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">disabling
internal thread affinity</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">WARNING:
This run will generate roughly 38667 Mb of data</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">NOTE:
DLB will not turn on during the first phase of PME tuning</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">starting
mdrun '200dppc_membrane_16Maculatin-WT-COO-'</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">10000000000
steps, 20000000.0 ps (continuing from step 268830925, 537661.8
ps).</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">-------------------------------------------------------</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Program
gmx mdrun, VERSION 5.1.2</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Source
code file:
/home/rpm/rpmbuild/BUILD/gromacs-5.1.2/src/gromacs/ewald/pme-load-balancing.cpp,
line: 947</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Software
inconsistency error:</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">pme_loadbal_do
called at an interval != nstlist</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">For
more information and tips for troubleshooting, please check
the GROMACS</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">website
at <a moz-do-not-send="true"
href="http://www.gromacs.org/Documentation/Errors">http://www.gromacs.org/Documentation/Errors</a></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">-------------------------------------------------------</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">-------------------------------------------------------</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Program
gmx mdrun, VERSION 5.1.2</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Source
code file:
/home/rpm/rpmbuild/BUILD/gromacs-5.1.2/src/gromacs/ewald/pme-load-balancing.cpp,
line: 947</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Software
inconsistency error:</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">pme_loadbal_do
called at an interval != nstlist</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">For
more information and tips for troubleshooting, please check
the GROMACS</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">website
at <a moz-do-not-send="true"
href="http://www.gromacs.org/Documentation/Errors">http://www.gromacs.org/Documentation/Errors</a></div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">-------------------------------------------------------</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Halting
parallel program gmx mdrun on rank 1 out of 2</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Halting
parallel program gmx mdrun on rank 0 out of 2</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">application
called MPI_Abort(MPI_COMM_WORLD, 1) - process 1</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">application
called MPI_Abort(MPI_COMM_WORLD, 1) - process 0</div>
<div
style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">srun:
error: gpu35: tasks 0-1: Exited with exit code 1<br>
</div>
<div><br>
</div>
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div><span><font color="#888888">Yukun Wang </font></span></div>
<div><span><font color="#888888">Postdoc<br>
Institute for NanoBioTechnology at The Johns Hopkins
University<br>
Cell phone: +1 (443) 509 2191<br>
Baltimore, MD USA</font></span></div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
</blockquote>
<br>
</body>
</html>