On Mon, Dec 3, 2012 at 9:38 PM, Shirts, Michael (mrs5pt) <span dir="ltr"><<a href="mailto:mrs5pt@eservices.virginia.edu" target="_blank">mrs5pt@eservices.virginia.edu</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Just to chime in, the illegal instruction error is definitely a bad one.<br>
However, a FAQ that clearly flags this error and that the solution is fixing<br>
those two compiler flags as the likely fix is probably enough for now.<br></blockquote><div><br></div><div>Note that modifying compiler flags won't help because in these cases you'd typically need to "lower" the CPU acceleration. This sis currently not possible without re-configuring.</div>
<div><br></div><div>--<br>Szilárd<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
Best,<br>
~~~~~~~~~~~~<br>
Michael Shirts<br>
Assistant Professor<br>
Department of Chemical Engineering<br>
University of Virginia<br>
<a href="mailto:michael.shirts@virginia.edu">michael.shirts@virginia.edu</a><br>
(434)-243-1821<br>
<br>
<br>
</div>> From: Roland Schulz <<a href="mailto:roland@utk.edu">roland@utk.edu</a>><br>
> Reply-To: Discussion list for GROMACS development <<a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a>><br>
> Date: Mon, 3 Dec 2012 13:18:27 -0500<br>
> To: Discussion list for GROMACS development <<a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a>><br>
<div class=""><div class="h5">> Subject: Re: [gmx-developers] Gromacs trying to use OpenMP instead of<br>
> thread-mpi despite trying to convince it?<br>
><br>
> On Mon, Dec 3, 2012 at 1:02 PM, Szilárd Páll <<a href="mailto:szilard.pall@cbr.su.se">szilard.pall@cbr.su.se</a>> wrote:<br>
><br>
>><br>
>> On Mon, Dec 3, 2012 at 5:12 AM, Roland Schulz <<a href="mailto:roland@utk.edu">roland@utk.edu</a>> wrote:<br>
>><br>
>>><br>
>>><br>
>>><br>
>>> On Sun, Dec 2, 2012 at 12:58 PM, Shirts, Michael (mrs5pt) <<br>
>>> <a href="mailto:mrs5pt@eservices.virginia.edu">mrs5pt@eservices.virginia.edu</a>> wrote:<br>
>>><br>
>>>> So, more progress, but no simulations running yet.<br>
>>>><br>
>>>> mdrun -nt 8 -ntmpi 8 gives the same error as before (I actually tried<br>
>>>> that<br>
>>>> before, and forgot to include it in my error report)<br>
>>>><br>
>>>> mdrun -ntmpi 8 -ntomp 1 gives the error<br>
>>>> Fatal error:<br>
>>>> OMP_NUM_THREADS (8) and the number of threads requested on the command<br>
>>>> line<br>
>>>> (1) have different values<br>
>>>> For more information and tips for troubleshooting, please check the<br>
>>>> GROMACS<br>
>>>> website at <a href="http://www.gromacs.org/Documentation/Errors" target="_blank">http://www.gromacs.org/Documentation/Errors</a><br>
>>>><br>
>>><br>
>>> We probably should print a notice that OMP_NUM_THREADS is set.<br>
>>> Otherwise this is really confusing if OMP_NUM_THREADS isn't set by the user<br>
>>> but by the system.<br>
>>><br>
>><br>
>> There is a note printed whenever the number of OpenMP threads is set by<br>
>> OMP_NUM_THREADS instead of -ntomp.<br>
>><br>
> I think if even core developers don't understand/find a note, then that's a<br>
> pretty clear sign that it will be confusing to the average user ;-)<br>
><br>
><br>
>><br>
>><br>
>>><br>
>>><br>
>>>> Fatal error:<br>
>>>> OMP_NUM_THREADS is invalid: '0'<br>
>>>><br>
>>><br>
>>> This is also for ntomp. Also there you wand to use 1 not 0 to disable<br>
>>> OpenMP (1 because it is total number of threads and thus 1 means serial).<br>
>>><br>
>>>><br>
>>>> /var/spool/PBS/mom_priv/jobs/<a href="http://2053253.lc5.itc.virginia.edu.SC" target="_blank">2053253.lc5.itc.virginia.edu.SC</a>: line 22:<br>
>>>> 12201<br>
>>>> Illegal instruction /h3/n1/shirtsgroup/gromac<br>
>>>> s_46/install/bin/mdrun_d -ntomp 1 -ntmpi -8 -deffnm<br>
>>>> /bigtmp/mrs5pt/eth.vrescale.50<br>
>>>><br>
>>><br>
>>> I suppose the CPU on the compute node is different from the build host.<br>
>>> You need to change GMX_CPU_ACCELERATION to the one correct for the compute<br>
>>> node. It could also help to set GMX_DISTRIBUTABLE_BUILD (both are cmake<br>
>>> options).<br>
>>><br>
>><br>
>> GMX_DISTRIBUTABLE_BUILD only does one thing, it turns off rdtscp. Is the<br>
>> intention to provide more features for this option? If not, I don't see the<br>
>> point in not calling it GMX_DISABLE_RDTSCP.<br>
>><br>
> I called it that for two reasons:<br>
> - Users don't know whether they want to disabled RDTSCP but they might know<br>
> whether they want to have a distributable build. In other words, in this<br>
> case the ultimate goal ("make it work on a different CPU") makes more sense<br>
> to the user then how this is achieved.<br>
> - It can be extended in the furture.<br>
><br>
> But I think we should consider to disable rdtscp by default. Unless we add<br>
> a runtime detection. The advantage is to small to cause problems. And as<br>
> far as I know the only other option which can cause illegal instruction is<br>
> GMX_CPU_ACCELERATION. But that one is pretty obvious to the user (it is a<br>
> non-advanced cmake option and I think people are much more likely to have<br>
> heard of SSE then rdtscp).<br>
><br>
> Roland<br>
><br>
><br>
><br>
>><br>
>> Cheers,<br>
>> --<br>
>> Szilárd<br>
>><br>
>><br>
>>><br>
>>> Roland<br>
>>><br>
>>><br>
>>>><br>
>>>> Best,<br>
>>>><br>
>>>> ~~~~~~~~~~~~<br>
>>>> Michael Shirts<br>
>>>> Assistant Professor<br>
>>>> Department of Chemical Engineering<br>
>>>> University of Virginia<br>
>>>> <a href="mailto:michael.shirts@virginia.edu">michael.shirts@virginia.edu</a><br>
>>>> (434)-243-1821<br>
>>>><br>
>>>><br>
>>>>> From: Berk Hess <<a href="mailto:hess@kth.se">hess@kth.se</a>><br>
>>>>> Date: Sun, 2 Dec 2012 09:34:11 +0100<br>
>>>>> To: "<a href="mailto:michael.shirts@virginia.edu">michael.shirts@virginia.edu</a>" <<a href="mailto:michael.shirts@virginia.edu">michael.shirts@virginia.edu</a>>,<br>
>>>> Discussion<br>
>>>>> list for GROMACS development <<a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a>><br>
>>>>> Subject: Re: [gmx-developers] Gromacs trying to use OpenMP instead of<br>
>>>>> thread-mpi despite trying to convince it?<br>
>>>>><br>
>>>><br>
>>>>> Hi,<br>
>>>>><br>
>>>>> You queuing system probably doesn't set OMP_NUM_THREADS then<br>
>>>>> and I assume this machine has at least 16 (HT) cores.<br>
>>>>> mdrun -ntmpi 8 -ntomp 1<br>
>>>>> will do what you want, or:<br>
>>>>> mdrun -nt 8 -ntmpi 8<br>
>>>>><br>
>>>>> Cheers,<br>
>>>>><br>
>>>>> Berk<br>
>>>>><br>
>>>>> On 12/02/2012 08:28 AM, Shirts, Michael (mrs5pt) wrote:<br>
>>>>>> Quick question:<br>
>>>>>><br>
>>>>>> Compiling the most recent code in release-4-6, I compile without<br>
>>>> OpenMP<br>
>>>>>> (because using group rather than verlet cutoffs), and using any of the<br>
>>>>>> below:<br>
>>>>>><br>
>>>>>> mdrun_d -ntmpi 8 -deffnm ethrun<br>
>>>>>> or<br>
>>>>>> mdrun_d -nt 8 -deffnm ethrun<br>
>>>>>> or<br>
>>>>>> mdrun_d -deffnm ethrun<br>
>>>>>> or<br>
>>>>>> mdrun_d -ntomp 0 -deffnm ethrun<br>
>>>>>> or<br>
>>>>>> mdrun_d -ntomp 0 -ntmpi8 -deffnm ethrun<br>
>>>>>><br>
>>>>>> I get:<br>
>>>>>> Fatal error:<br>
>>>>>> OpenMP threads are requested, but Gromacs was compiled without OpenMP<br>
>>>>>> support<br>
>>>>>> For more information and tips for troubleshooting, please check the<br>
>>>> GROMACS<br>
>>>>>> website at <a href="http://www.gromacs.org/Documentation/Errors" target="_blank">http://www.gromacs.org/Documentation/Errors</a><br>
>>>>>><br>
>>>>>> Even though I'm presumably requesting thread-mpi. Worked fine with<br>
>>>> -nt<br>
>>>>>> previously (before the new -nt options introduced a few months back).<br>
>>>>>><br>
>>>>>> Any suggestions or something I'm doing wrong? Perhaps gromacs is<br>
>>>>>> interpreting the cluster environment as requesting OpenMP somehow?<br>
>>>> FWIW, the<br>
>>>>>> PBS script request line is "#PBS -l select=1:mpiprocs=8:ncpus=8".<br>
>>>>>><br>
>>>>>> Apologies if I missed this answers somewhere out there already.<br>
>>>>>><br>
>>>>>> Thanks,<br>
>>>>>> ~~~~~~~~~~~~<br>
>>>>>> Michael Shirts<br>
>>>>>> Assistant Professor<br>
>>>>>> Department of Chemical Engineering<br>
>>>>>> University of Virginia<br>
>>>>>> <a href="mailto:michael.shirts@virginia.edu">michael.shirts@virginia.edu</a><br>
>>>>>> (434)-243-1821<br>
>>>>>><br>
>>>>><br>
>>>><br>
>>>> --<br>
>>>> gmx-developers mailing list<br>
>>>> <a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a><br>
>>>> <a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
>>>> Please don't post (un)subscribe requests to the list. Use the<br>
>>>> www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>>><br>
>>><br>
>>><br>
>>> --<br>
>>> ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>
>>> 865-241-1537, ORNL PO BOX 2008 MS6309<br>
>>><br>
>>> --<br>
>>> gmx-developers mailing list<br>
>>> <a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a><br>
>>> <a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
>>> Please don't post (un)subscribe requests to the list. Use the<br>
>>> www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
>>><br>
>><br>
>><br>
><br>
><br>
> --<br>
> ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>
> 865-241-1537, ORNL PO BOX 2008 MS6309<br>
> --<br>
> gmx-developers mailing list<br>
> <a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a><br>
> <a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
> Please don't post (un)subscribe requests to the list. Use the<br>
> www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
<br>
--<br>
gmx-developers mailing list<br>
<a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
</div></div></blockquote></div><br></div>