<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p><br>
</p>
<div dir="ltr" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<div id="divRplyFwdMsg" dir="ltr"></div>
<div>
<p>Hi folks,<br>
</p>
<p><br>
</p>
<p>I am a member of the Scientific Computing group at Oak Ridge National Lab's National Center for Computational Sciences (NCCS). We are preparing high performance codes for public use on our new Summit system by testing on the prototype SummitDev:<br>
</p>
<p><br>
</p>
<p><span style="font-family:Arial,"Trebuchet MS",Verdana; font-size:12.6px; background-color:rgb(255,255,255)">The Summitdev system is an early access system that is one generation removed from OLCF’s next big supercomputer, Summit. The system contains three
racks, each with 18 IBM POWER8 S822LC nodes for a total of 54 nodes. Each IBM S822LC node has 2 IBM POWER8 CPUs and 4 NVIDIA Tesla P100 GPUs. The POWER8 CPUs have 32 8GB DDR4 memory (256 GB). Each POWER8 node has 10 cores with 8 HW threads. The GPUs are connected
by NVLink 1.0 at 80GB/s and each GPU has 16GB HBM2 memory. The nodes are connected in a full fat-tree via EDR InfiniBand. The racks are liquid cooled with a heat exchanger rate. Summitdev has access to Spider 2, the OLCF’s center-wide Lustre parallel file
system.</span><br>
</p>
<p><br>
</p>
<p>I have built gromacs 5.1.4., seemingly successfully, using GPU, MPI, SIMD options with CMake directives (I didn't get the alvtivec option completely right, but that should be ok). The only things that were not found during the cmake were the NVML library
(we do not have the Deployment package installed), LAPACK (although it did find essl after I set the BLAS directive), and some includes like io.h.<br>
</p>
<p><br>
</p>
<p>I cannot use the make check because on summitdev I have to launch gmx_mpi with mpirun, and I think the make check does not call gmx that way. I have manually tested the build tools (pdb2gmx, editconf, grommp, solvate), and they work fine. <br>
</p>
<p><br>
</p>
<p>However, I cannot get mdrun to work even on one node on a small box of water. After playing with the number of processes so that the openMP threads message stopped crying, I get the following:<br>
</p>
<p><br>
</p>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">>>cudaMallocHost of size 1024128 bytes failed: all CUDA-capable devices are busy or unavailable</span>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255); margin-top:0px; margin-bottom:0px">
<br>
Here is the complete context for this error:<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255); margin-top:0px; margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255); margin-top:0px; margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255); margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px">****************************************************************************************************</span></font><br>
</div>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
<div><span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)"><br>
</span></div>
bash-4.2$ mpirun -np 80 /ccs/home/adaa/gromacs/bin/gmx_mpi mdrun -v -deffnm water_em</span>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)">
<br>
:-) GROMACS - gmx mdrun, VERSION 5.1.4 (-:<br>
GROMACS is written by:<br>
Emile Apol Rossen Apostolov Herman J.C. Berendsen Par Bjelkmar <br>
Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian Fritsch <br>
Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent Hindriksen<br>
Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten Kutzner <br>
Per Larsson Justin A. Lemkul Magnus Lundborg Pieter Meulenhoff <br>
Erik Marklund Teemu Murtola Szilard Pall Sander Pronk <br>
Roland Schulz Alexey Shvetsov Michael Shirts Alfons Sijbers <br>
Peter Tieleman Teemu Virolainen Christian Wennberg Maarten Wolf <br>
and the project leaders:<br>
Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel<br>
Copyright (c) 1991-2000, University of Groningen, The Netherlands.<br>
Copyright (c) 2001-2015, The GROMACS development team at<br>
Uppsala University, Stockholm University and<br>
the Royal Institute of Technology, Sweden.<br>
check out http://www.gromacs.org for more information.<br>
GROMACS is free software; you can redistribute it and/or modify it<br>
under the terms of the GNU Lesser General Public License<br>
as published by the Free Software Foundation; either version 2.1<br>
of the License, or (at your option) any later version.<br>
GROMACS: gmx mdrun, VERSION 5.1.4<br>
Executable: /ccs/home/adaa/gromacs/bin/gmx_mpi<br>
Data prefix: /ccs/home/adaa/gromacs<br>
Command line:<br>
gmx_mpi mdrun -v -deffnm water_em<br>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
<div style="margin-top:0px; margin-bottom:0px"><br>
</div>
Back Off! I just backed up water_em.log to ./#water_em.log.15#<br>
Number of logical cores detected (160) does not match the number reported by OpenMP (80).<br>
Consider setting the launch configuration manually!<br>
Running on 4 nodes with total 640 logical cores, 16 compatible GPUs<br>
Logical cores per node: 160<br>
Compatible GPUs per node: 4<br>
All nodes have identical type(s) of GPUs<br>
Hardware detected on host summitdev-r0c1n04 (the node of MPI rank 0):<br>
CPU info:<br>
Vendor: IBM<br>
Brand: POWER8NVL (raw), altivec supported<br>
SIMD instructions most likely to fit this hardware: IBM_VSX<br>
SIMD instructions selected at GROMACS compile time: IBM_VMX<br>
GPU info:<br>
Number of GPUs detected: 4<br>
#0: NVIDIA Tesla P100-SXM2-16GB, compute cap.: 6.0, ECC: yes, stat: compatible<br>
#1: NVIDIA Tesla P100-SXM2-16GB, compute cap.: 6.0, ECC: yes, stat: compatible<br>
#2: NVIDIA Tesla P100-SXM2-16GB, compute cap.: 6.0, ECC: yes, stat: compatible<br>
#3: NVIDIA Tesla P100-SXM2-16GB, compute cap.: 6.0, ECC: yes, stat: compatible<br>
Compiled SIMD instructions: IBM_VMX, GROMACS could use IBM_VSX on this machine, which is better<br>
Reading file water_em.tpr, VERSION 5.1.4 (single precision)<br>
Using 80 MPI processes<br>
Using 8 OpenMP threads per MPI process<br>
On host summitdev-r0c1n04 4 compatible GPUs are present, with IDs 0,1,2,3<br>
On host summitdev-r0c1n04 4 GPUs auto-selected for this run.<br>
Mapping of GPU IDs to the 20 PP ranks in this node: 0,0,0,0,0,1,1,1,1,1,2,2,2,2,2,3,3,3,3,3<br>
<br>
NOTE: GROMACS was configured without NVML support hence it can not exploit<br>
application clocks of the detected Tesla P100-SXM2-16GB GPU to improve performance.<br>
Recompile with the NVML library (compatible with the driver used) or set application clocks manually.<br>
<br>
-------------------------------------------------------<br>
Program gmx mdrun, VERSION 5.1.4<br>
Source code file: /ccs/home/adaa/gromacs-5.1.4/src/gromacs/gmxlib/cuda_tools/pmalloc_cuda.cu, line: 70<br>
Fatal error:<br>
cudaMallocHost of size 1024128 bytes failed: all CUDA-capable devices are busy or unavailable<br>
</div>
<p><br>
</p>
<p><br>
</p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255); margin-top:0px; margin-bottom:0px">
<font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px">****************************************************************************************************</span></font><br>
</div>
<div><font face="Calibri,Arial,Helvetica,sans-serif" size="2"><span style="font-size:16px"><br>
</span></font></div>
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:16px; background-color:rgb(255,255,255)"></span>Thanks for any help you can offer.
<div><br>
<p><br>
</p>
<p>Best,<br>
</p>
<p><br>
</p>
<p>Ada<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div id="Signature">
<div name="divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<p style="font-size:16px"><span style="font-size:10pt">Ada Sedova</span></p>
<p style="font-size:16px"><span style="font-size:10pt">Postdoctoral Research Associate</span></p>
<p style="font-size:16px"><span style="font-size:10pt">Scientific Computing Group, NCCS</span><br>
</p>
<p style="font-size:16px"><span style="font-size:10pt">Oak Ridge National Laboratory, Oak Ridge, TN</span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>