Hi,<div><br></div><div>I think I found out what's going on. </div><div>It is a combination of gcc and libc. If expf is used, the time is wasted in feholdexcept, fesetenv, fesetround. </div><div><div>Because libc doesn't has a x86-64 optimized version for expf. See:</div>
<div><a href="http://www.mailinglistarchive.com/html/fortran@gcc.gnu.org/2006-09/msg01281.html">http://www.mailinglistarchive.com/html/fortran@gcc.gnu.org/2006-09/msg01281.html</a></div><div><br></div><div>It only shows up in 4.2.x and 4.3.x because other gcc version don't replace exp with expf.</div>
<div>
<br></div><div>Thus one could either file a bug against gcc 4.3 (4.2 isn't maintained anymore) to not do the replacement. But I doubt this is high enough priority to get in. Thus it makes more sense to file against libc to fix expf. I don't understand why they didn't include the optimized expf which was submitted (see the links in the above linked message).</div>
<div><br></div><div>To make a work-around in GROMACS I suggest to submit the patch from the earlier (the temp double variable). Probably best with #ifdef to have the workaround only for 4.2.x and 4.3.x, to make sure it doesn't affect any other compiler negatively. Should I commit that?</div>
<div><br></div><div>BTW: One can save 5% in solve PME by -ffast-math (gcc 4.4.2). Is it save to use -ffast-math when compiling pme.c? If so should we make this default in the configure/Makefile?</div><div><br></div><div>
Roland</div>
<div><br><br><div class="gmail_quote">On Sat, May 15, 2010 at 5:40 PM, Roland Schulz <span dir="ltr"><<a href="mailto:roland@utk.edu" target="_blank">roland@utk.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<div><br></div><div>Two more things:</div><div>I can't reproduce Berk's measurement that 4.3.2 is fine. PME solve is as slow for me compiled with 4.3.2 then with 4.3.4 (both 8.9s instead of 2.9s for 4.4.2 for 20^3 grid on 8 threads). And this agrees with the bug report I linked that the replacement is 4.2.x and 4.3.x are doing this replacement. I'm running this on Xeon X5550.</div>
<div><br></div><div>I can't figure out so far whether the difference is in the function call overhead or in the exp/expf function itself. The data is in (4 packed floats) xmm registers and thus the call to exp/expf require a cvtpd2ps/movss. That the expf is fast in a test program suggests it is the data conversion. The profiling with hpctoolkit suggests it is the expf function itself. I'm not sure whether my test program is not capturing the problem correctly or whether I misread the hpctoolkit output. Or whether it is some interplay of both (e.g. save/restore of registers).</div>
<div><br></div><div>If someone could run some profiling to verify my hpctoolkit results to test what is the reason, this would help. As far as I know it has to be a sampling based profiling: E.g. Apple Shark but not e.g. gprof/tau.</div>
<div><br></div><div><font color="#888888">Roland</font><div><div></div><div><br><br><div class="gmail_quote">On Sat, May 15, 2010 at 4:30 AM, Alexey Shvetsov <span dir="ltr"><<a href="mailto:alexxyum@gmail.com" target="_blank">alexxyum@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all!<br><br>I made some tests with current git master and gcc-4.4.3 and gcc-4.5.0<br>Seems gromacs compiled with gcc-4.5.0 is a little bit faster then gcc-4.4.3 (~2-5%)<br>I can do this tests with profiling if needed<br>
<br><div class="gmail_quote">2010/5/15 Roland Schulz <span dir="ltr"><<a href="mailto:roland@utk.edu" target="_blank">roland@utk.edu</a>></span><div><div></div><div><br><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
Hi,<div><br></div><div>it is caused by gcc using expf instead of exp.</div><div><br></div><div>By changing pme.c in the two places where exp is called this way:</div><div><div>- tmp1[kx] = exp(tmp1[kx]);</div>
<div>+ double t = tmp1[kx];</div><div>+ tmp1[kx] = exp(t);</div><div> </div><div>gcc 4.3.4 is as fast as gcc 4.4.2. </div><div><br></div><div>See also <a href="http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35202" target="_blank">http://gcc.gnu.org/bugzilla/show_bug.cgi?id=35202</a></div>
<div>(The bug is not related to performance but it discusses the replacement from exp->expf)</div><div><br></div><div>But what I don't understand at all is why expf is so much slower (expf 7-8x as slow in pme.c than exp (*)). A small test program which computes exp on an array, compiled with gcc 4.3.4 is actually faster than with 4.4.2 (30%). Thus for the test-program the expf seams to be faster. Why expf is so slow in pme.c is odd.</div>
<div><br></div><div>*) Got that measurement form HPCToolkit. Shows performance line-by-line. Thus it was easy to pin-point the cause.</div><div><br></div><div>Roland</div><div> </div></div><div><div><div>
</div><div><br><div class="gmail_quote">
On Fri, May 14, 2010 at 10:43 AM, Berk Hess <span dir="ltr"><<a href="mailto:hess@cbr.su.se" target="_blank">hess@cbr.su.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
Hi,<br>
<br>
The PME code in git master is a lot faster than the current 4.0.7<br>
release code.<br>
However, there seems to be a bug in gcc version 4.3.3 and 4.3.4 (4.3.2<br>
and 4.4.1 are fine)<br>
that make the pme_solve part 5 times (!) slower than with proper<br>
compiler versions.<br>
This is between 10% and 20% of the total mdrun performance.<br>
We are currently trying to figure out what triggers this issue and we<br>
are sending<br>
a bug report to the gcc mailing list.<br>
<br>
I someone has a hint, please reply.<br>
<br>
Berk<br>
<font color="#888888"><br>
--<br>
gmx-developers mailing list<br>
<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
</font></blockquote></div><br><br clear="all"><br></div></div>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309<br>
</div>
<br>--<br>
gmx-developers mailing list<br>
<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br></blockquote></div></div></div><br><br clear="all"><br>-- <br>Best Regards,<br>Alexey 'Alexxy' Shvetsov<br>
Petersburg Nuclear Physics Institute, Russia<br>Department of Molecular and Radiation Biophysics<br>Gentoo Team Ru<br>Gentoo Linux Dev<br>mailto:<a href="mailto:alexxyum@gmail.com" target="_blank">alexxyum@gmail.com</a><br>
mailto:<a href="mailto:alexxy@gentoo.org" target="_blank">alexxy@gentoo.org</a><br>
mailto:<a href="mailto:alexxy@omrb.pnpi.spb.ru" target="_blank">alexxy@omrb.pnpi.spb.ru</a><br>
<br>--<br>
gmx-developers mailing list<br>
<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a><br>
<a href="http://lists.gromacs.org/mailman/listinfo/gmx-developers" target="_blank">http://lists.gromacs.org/mailman/listinfo/gmx-developers</a><br>
Please don't post (un)subscribe requests to the list. Use the<br>
www interface or send it to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br></blockquote></div><br><br clear="all"><br>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>
865-241-1537, ORNL PO BOX 2008 MS6309<br>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br>ORNL/UT Center for Molecular Biophysics <a href="http://cmb.ornl.gov" target="_blank">cmb.ornl.gov</a><br>865-241-1537, ORNL PO BOX 2008 MS6309<br>
</div>
</div>