<div dir="ltr"><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px">Dear All,<br></span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px"><br></span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px">I recently met a error:"Software inconsistency error: pme_loadbal_do called at an interval != nstlist" when I try to restart one of jobs.</span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px"><font face="Calibri,Arial,Helvetica,sans-serif">I used GMX 5.1.2 on K80 GPU nodes with haswell CPU.</font><br></span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px"><br></span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px">Is there anyone know how to solve this error?<br></span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px">Thank you very much!<br></span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px"><br></span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px">cheers<br></span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px">Yukun Wang<br></span></font></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">below is the output log files:<br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-:</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> GROMACS is written by:</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par Bjelkmar</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian Fritsch</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent Hindriksen</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten Kutzner</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter Meulenhoff</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Erik Marklund Teemu Murtola Szilard Pall Sander Pronk</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Roland Schulz Alexey Shvetsov Michael Shirts Alfons Sijbers</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Peter Tieleman Teemu Virolainen Christian Wennberg Maarten Wolf</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> and the project leaders:</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Copyright (c) 1991-2000, University of Groningen, The Netherlands.</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Copyright (c) 2001-2015, The GROMACS development team at</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Uppsala University, Stockholm University and</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">the Royal Institute of Technology, Sweden.</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">check out <a href="http://www.gromacs.org">http://www.gromacs.org</a> for more information.</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">GROMACS is free software; you can redistribute it and/or modify it</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">under the terms of the GNU Lesser General Public License</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">as published by the Free Software Foundation; either version 2.1</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">of the License, or (at your option) any later version.</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">GROMACS: gmx mdrun, VERSION 5.1.2</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Executable: /lustre/usr/gromacs/5.1-icc15-impi5-gpu/bin/gmx_mpi</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Data prefix: /lustre/usr/gromacs/5.1-icc15-impi5-gpu</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Command line:</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> gmx_mpi mdrun -deffnm 200DPPC-16-WT_COO-_150mV_2 -append -cpi -v -resethway -noconfout</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">No previous checkpoint file present, assuming this is a new run.</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Number of logical cores detected (16) does not match the number reported by OpenMP (8).</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Consider setting the launch configuration manually!</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Running on 1 node with total 16 cores, 16 logical cores, 2 compatible GPUs</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Hardware detected on host gpu35 (the node of MPI rank 0):</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> CPU info:</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Vendor: GenuineIntel</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Brand: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> SIMD instructions most likely to fit this hardware: AVX_256</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> SIMD instructions selected at GROMACS compile time: AVX_256</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> GPU info:</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Number of GPUs detected: 2</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> #0: NVIDIA Tesla K20m, compute cap.: 3.5, ECC: yes, stat: compatible</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> #1: NVIDIA Tesla K20m, compute cap.: 3.5, ECC: yes, stat: compatible</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Reading file 200DPPC-16-WT_COO-_150mV_2.tpr, VERSION 5.0.4 (single precision)</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Note: file tpx version 100, software tpx version 103</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Changing nstlist from 25 to 40, rlist from 1.038 to 1.08</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">The number of OpenMP threads was set by environment variable OMP_NUM_THREADS to 8</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Using 2 MPI processes</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Using 8 OpenMP threads per MPI process</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">On host gpu35 2 compatible GPUs are present, with IDs 0,1</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">On host gpu35 2 GPUs auto-selected for this run.</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Mapping of GPU IDs to the 2 PP ranks in this node: 0,1</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">NOTE: GROMACS was configured without NVML support hence it can not exploit</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> application clocks of the detected Tesla K20m GPU to improve performance.</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"> Recompile with the NVML library (compatible with the driver used) or set application clocks manually.</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Non-default thread affinity set probably by the OpenMP library,</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">disabling internal thread affinity</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">WARNING: This run will generate roughly 38667 Mb of data</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">NOTE: DLB will not turn on during the first phase of PME tuning</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">starting mdrun '200dppc_membrane_16Maculatin-WT-COO-'</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">10000000000 steps, 20000000.0 ps (continuing from step 268830925, 537661.8 ps).</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">-------------------------------------------------------</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Program gmx mdrun, VERSION 5.1.2</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Source code file: /home/rpm/rpmbuild/BUILD/gromacs-5.1.2/src/gromacs/ewald/pme-load-balancing.cpp, line: 947</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Software inconsistency error:</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">pme_loadbal_do called at an interval != nstlist</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">For more information and tips for troubleshooting, please check the GROMACS</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">website at <a href="http://www.gromacs.org/Documentation/Errors">http://www.gromacs.org/Documentation/Errors</a></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">-------------------------------------------------------</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">-------------------------------------------------------</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Program gmx mdrun, VERSION 5.1.2</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Source code file: /home/rpm/rpmbuild/BUILD/gromacs-5.1.2/src/gromacs/ewald/pme-load-balancing.cpp, line: 947</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Software inconsistency error:</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">pme_loadbal_do called at an interval != nstlist</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">For more information and tips for troubleshooting, please check the GROMACS</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">website at <a href="http://www.gromacs.org/Documentation/Errors">http://www.gromacs.org/Documentation/Errors</a></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">-------------------------------------------------------</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Halting parallel program gmx mdrun on rank 1 out of 2</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Halting parallel program gmx mdrun on rank 0 out of 2</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0</div><div style="color:rgb(0,0,0);font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">srun: error: gpu35: tasks 0-1: Exited with exit code 1<br></div><div><br></div><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><span><font color="#888888">Yukun Wang </font></span></div><div><span><font color="#888888">Postdoc<br>Institute for NanoBioTechnology at The Johns Hopkins University<br>Cell phone: +1 (443) 509 2191<br>Baltimore, MD USA</font></span></div></div></div>
</div>