<div dir='auto'>Hi,<div dir="auto"><br></div><div dir="auto">An i9-7940X has slightly more than 1 TFLOPS peak in double precision.</div><div dir="auto"><br></div><div dir="auto">Cheers,</div><div dir="auto"><br></div><div dir="auto">Berk</div><div class="gmail_extra" dir="auto"><br><div class="gmail_quote">On Jul 24, 2019 19:19, James <james@ryley.com> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Thanks for all the information. Unless I am misunderstanding the numbers, yes, compared to single precision, the GeForce cards are awful (about 1/30th of single precision performance). That being said, because single precision speed is absurdly high, the GeForce 1080 Ti still provides 330-350 GFLOPS in double. In comparison, the i9-7940X provides about 60 GFLOPS in double according to the benchmarks I've seen (<a href="https://ranker.sisoftware.co.uk/top_device.php?q=c2ffc9f1d7bad7eaccbe83b395fcc1f0d6be83b690e8d5e4c2a7c2ffcfe1d4f281bc8c&l=en">https://ranker.sisoftware.co.uk/top_device.php?q=c2ffc9f1d7bad7eaccbe83b395fcc1f0d6be83b690e8d5e4c2a7c2ffcfe1d4f281bc8c&l=en</a>). So, assuming the processor can keep the GPU fed, even the "awful" performance of the GeForce 1080 Ti would be like adding ~5 CPUs.</div><div><br></div><div>If you were to move to the Volta-based cards, their double precision performance is in the 6 - 7.4 TFLOP range, or 100X faster than a high-end i9 (and presumably way beyond what you could get from a dual or quad Xeon system). Granted, the Volta cards are expensive, but if you want a whole cluster in a box, it seems like a pretty good deal. (Again, if the processor can keep the GPU fed -- I have no idea if you could actually realize anywhere near a 100X speedup).</div><div><br></div><div>Either of these scenarios seems pretty good to me -- unless the processor cannot keep the GPU fed.</div><div><br></div><div>Assuming that a substantial double precision speedup could be realized, I'd be interested in knowing if anyone on the list would like to implement this on a contract basis. If so, feel free to contact me directly.</div><div><br clear="all"><div><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr">Sincerely,<br>
James <br></div></div></div></div><br></div></div><br><div class="elided-text"><div dir="ltr">On Wed, Jul 24, 2019 at 7:17 AM <<a href="mailto:gromacs.org_gmx-developers-request@maillist.sys.kth.se">gromacs.org_gmx-developers-request@maillist.sys.kth.se</a>> wrote:<br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex">Send gromacs.org_gmx-developers mailing list submissions to<br>
<a href="mailto:gromacs.org_gmx-developers@maillist.sys.kth.se">gromacs.org_gmx-developers@maillist.sys.kth.se</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a><br>
<br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:gromacs.org_gmx-developers-request@maillist.sys.kth.se">gromacs.org_gmx-developers-request@maillist.sys.kth.se</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:gromacs.org_gmx-developers-owner@maillist.sys.kth.se">gromacs.org_gmx-developers-owner@maillist.sys.kth.se</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of gromacs.org_gmx-developers digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: Double precision on GPU's (Szil?rd P?ll)<br>
2. Re: Double precision on GPU's (Berk Hess)<br>
3. Re: Double precision on GPU's (Erik Lindahl)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Wed, 24 Jul 2019 15:20:21 +0200<br>
From: Szil?rd P?ll <<a href="mailto:pall.szilard@gmail.com">pall.szilard@gmail.com</a>><br>
To: Discussion list for GROMACS development<br>
<<a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a>><br>
Subject: Re: [gmx-developers] Double precision on GPU's<br>
Message-ID:<br>
<CANnYEw7nm=36HCY29EXSRFZyYz8bcMZWM5L=<a href="mailto:pNd06PNehgKknw@mail.gmail.com">pNd06PNehgKknw@mail.gmail.com</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi,<br>
<br>
Indeed, a straight port without much performance optimization may not be a<br>
lot of effort, but integrating an additional kernel flavor into the<br>
existing codebase will mean added complexity which will probably require<br>
some refactoring and preliminary work to accommodate the new set of kernels<br>
without code duplication and avoiding introducing complexity or performance<br>
overhead in the current kernels.<br>
<br>
However also note that most NVIDIA consumer cards -- which are very widely<br>
used by our users -- have a 32x lower DP throughput than SP which is far<br>
more than what what most people would find acceptable, I'd say.<br>
<br>
--<br>
Szil?rd<br>
<br>
<br>
On Mon, Jul 22, 2019 at 10:28 PM Berk Hess <<a href="mailto:hess@kth.se">hess@kth.se</a>> wrote:<br>
<br>
> Hi,<br>
><br>
> IIRC all Nvidia Tesla cards have always had double precision, at half the<br>
> throughput of single precision. But there are very few cases where double<br>
> precision is needed. Energy drift in single precision is never an issue,<br>
> unless you really can not use a thermostat.<br>
><br>
> But having said that, making the GPU code, either CUDA or OpenCL work in<br>
> double precision is probably not much effort. But making it work<br>
> efficiently requires optimizing several algorithmic parameters and maybe<br>
> changing the arrangement of some data in the different GPU memory levels.<br>
><br>
> Cheers,<br>
><br>
> Berk<br>
><br>
> On 7/22/19 10:10 PM, James wrote:<br>
><br>
> Hi,<br>
><br>
> My apologies if this question has been previously discussed. I just joined<br>
> the list and all I know is that from reading the docs and release comments,<br>
> writ ing code for double precision on GPU's is not a priority.<br>
><br>
> However, I believe all recent upper-end Nvidia cards have native double<br>
> precision (which was not true several generations ago). So, you don't have<br>
> to have a real "scientific computing" GPU to take advantage of this -- most<br>
> people probably already have the hardware. Still, I understand that most<br>
> people do not need/want to run double precision. But, some do (and you have<br>
> to if you are concerned with conservation of energy -- the energy drift in<br>
> single precision is substantial).<br>
><br>
> So, I would like to ask what the level of effort to do this is believed to<br>
> be? Would it require a lot of new code, or would it be porting the single<br>
> precision code to double precision?<br>
><br>
> Sincerely,<br>
> James<br>
><br>
><br>
> --<br>
> Gromacs Developers mailing list<br>
><br>
> * Please search the archive at<br>
> <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before<br>
> posting!<br>
><br>
> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists">http://www.gromacs.org/Support/Mailing_Lists</a><br>
><br>
> * For (un)subscribe requests visit<br>
> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a><br>
> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20190724/6b8b282e/attachment-0001.html">http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20190724/6b8b282e/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 24 Jul 2019 15:25:53 +0200<br>
From: Berk Hess <<a href="mailto:hess@kth.se">hess@kth.se</a>><br>
To: <a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a><br>
Subject: Re: [gmx-developers] Double precision on GPU's<br>
Message-ID: <<a href="mailto:e49ccd11-2dc5-1d52-efe7-e39299fcf436@kth.se">e49ccd11-2dc5-1d52-efe7-e39299fcf436@kth.se</a>><br>
Content-Type: text/plain; charset="windows-1252"; Format="flowed"<br>
<br>
Hi,<br>
<br>
The energy conservation check can still be done with a thermostat, as <br>
GROMACS keeps track of the amount of energy the thermostat adds to or <br>
removes from the system.<br>
<br>
I don't understand what you mean exactly with "energy balance". Either <br>
you are interested in energy dissipation between different parts of the <br>
system, in which case you often can not use a thermostat, or you are not <br>
and then you can, and probably should, use a thermostat (and still keep <br>
track of energy conservation).<br>
<br>
Cheers,<br>
<br>
Berk<br>
<br>
On 2019-07-23 21:19 , James wrote:<br>
> Hi Berk,<br>
><br>
> Thank you for the information. I prefer not to use thermostats, at <br>
> least when trying to get quantitative values, because (and I may be <br>
> misunderstanding thermostats, but since changing the temperature <br>
> changes the energy, I assume this is true) then I cannot use "energy <br>
> in = energy out" as a sanity check. I'm more interested in energy <br>
> balance than in maintaining a given temperature.<br>
><br>
> If I were to be able to get the double precision GPU stuff coded, <br>
> would the team be willing to maintain it? Or, since I don't know the <br>
> code, perhaps a better question is: Does someone who is already <br>
> familiar with the relevant code have time to do this as a side project <br>
> (with compensation)? If anyone is interested, please feel free to <br>
> contact me off list.<br>
><br>
> Thanks,<br>
> James<br>
><br>
> =======================================<br>
> Hi,<br>
><br>
> IIRC all Nvidia Tesla cards have always had double precision, at half<br>
> the throughput of single precision. But there are very few cases where<br>
> double precision is needed. Energy drift in single precision is never an<br>
> issue, unless you really can not use a thermostat.<br>
><br>
> But having said that, making the GPU code, either CUDA or OpenCL work in<br>
> double precision is probably not much effort. But making it work<br>
> efficiently requires optimizing several algorithmic parameters and maybe<br>
> changing the arrangement of some data in the different GPU memory levels.<br>
><br>
> Cheers,<br>
><br>
> Berk<br>
><br>
> On 7/22/19 10:10 PM, James wrote:<br>
> >/Hi, />//>/My apologies if this question has been previously discussed. I just />/joined the list and all I know is that from reading the docs and />/release comments, writing code for double precision on GPU's is not a />/priority. />//>/However, I believe all recent upper-end Nvidia cards have native />/double precision (which was not true several generations ago). So, you />/don't have to have a real "scientific computing" GPU to take advantage />/of this -- most people probably already have the hardware. Still, I />/understand that most people do not need/want to run double precision. />/But, some do (and you have to if you are concerned with conservation />/of energy -- the energy drift in single precision is substantial). />//>/So, I would like to ask what the level of effort to do this is />/believed to be? Would it require a lot of new code, or would it be />/porting the single precision code to double precision? />//>/Sincerely, />/James /><br>
><br>
><br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20190724/54a798c7/attachment-0001.html">http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20190724/54a798c7/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Wed, 24 Jul 2019 16:17:24 +0200<br>
From: Erik Lindahl <<a href="mailto:erik.lindahl@gmail.com">erik.lindahl@gmail.com</a>><br>
To: <a href="mailto:gmx-developers@gromacs.org">gmx-developers@gromacs.org</a><br>
Subject: Re: [gmx-developers] Double precision on GPU's<br>
Message-ID: <<a href="mailto:CF6EF0AC-6C50-4CC7-95EB-049E997C610B@gmail.com">CF6EF0AC-6C50-4CC7-95EB-049E997C610B@gmail.com</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Hi,<br>
<br>
Just to add some minor things to what Berk already said:<br>
<br>
1. With current nvidia cards, for the Tesla line the double precision floprate is 50% of single. However, for the consumer cards it's only 1/8-1/12th. In practice the latter is so low that it's often better to run on the CPU instead. This is the main reason we haven't bothered with it yet - most academic labs use GeForce :-)<br>
<br>
2. You are likely well aware of this, but for users in general another problem with skipping thermostats for complex biophysical systems is that the potential energy often goes down as the system relaxes - which will make the temperature go up in an NVE ensemble.<br>
<br>
3. Having said that, I think it should be a fairly straightforward minor project for someone. It's probably mostly a matter of changing variable types so they compile to double if GMX_DOUBLE is set.<br>
<br>
Cheers,<br>
<br>
Erik<br>
<br>
Erik Lindahl <<a href="mailto:erik.lindahl@scilifelab.se">erik.lindahl@scilifelab.se</a>><br>
Professor of Biophysics<br>
Science for Life Laboratory<br>
Stockholm University & KTH<br>
Office (SciLifeLab): +46 8 524 81567<br>
Cell (Sweden): +46 73 4618050<br>
Cell (US): +1 (650) 924 7674 <br>
<br>
<br>
<br>
> On 24 Jul 2019, at 15:25, Berk Hess <<a href="mailto:hess@kth.se">hess@kth.se</a>> wrote:<br>
> <br>
> Hi,<br>
> <br>
> The energy conservation check can still be done with a thermostat, as GROMACS keeps track of the amount of energy the thermostat adds to or removes from the system.<br>
> <br>
> I don't understand what you mean exactly with "energy balance". Either you are interested in energy dissipation between different parts of the system, in which case you often can not use a thermostat, or you are not and then you can, and probably should, use a thermostat (and still keep track of energy conservation).<br>
> <br>
> Cheers,<br>
> <br>
> Berk<br>
> <br>
>> On 2019-07-23 21:19 , James wrote:<br>
>> Hi Berk,<br>
>> <br>
>> Thank you for the information. I prefer not to use thermostats, at least when trying to get quantitative values, because (and I may be misunderstanding thermostats, but since changing the temperature changes the energy, I assume this is true) then I cannot use "energy in = energy out" as a sanity check. I'm more interested in energy balance than in maintaining a given temperature.<br>
>> <br>
>> If I were to be able to get the double precision GPU stuff coded, would the team be willing to maintain it? Or, since I don't know the code, perhaps a better question is: Does someone who is already familiar with the relevant code have time to do this as a side project (with compensation)? If anyone is interested, please feel free to contact me off list.<br>
>> <br>
>> Thanks,<br>
>> James<br>
>> <br>
>> =======================================<br>
>> Hi,<br>
>> <br>
>> IIRC all Nvidia Tesla cards have always had double precision, at half <br>
>> the throughput of single precision. But there are very few cases where <br>
>> double precision is needed. Energy drift in single precision is never an <br>
>> issue, unless you really can not use a thermostat.<br>
>> <br>
>> But having said that, making the GPU code, either CUDA or OpenCL work in <br>
>> double precision is probably not much effort. But making it work <br>
>> efficiently requires optimizing several algorithmic parameters and maybe <br>
>> changing the arrangement of some data in the different GPU memory levels.<br>
>> <br>
>> Cheers,<br>
>> <br>
>> Berk<br>
>> <br>
>> On 7/22/19 10:10 PM, James wrote:<br>
>> > Hi,<br>
>> ><br>
>> > My apologies if this question has been previously discussed. I just <br>
>> > joined the list and all I know is that from reading the docs and <br>
>> > release comments, writing code for double precision on GPU's is not a <br>
>> > priority.<br>
>> ><br>
>> > However, I believe all recent upper-end Nvidia cards have native <br>
>> > double precision (which was not true several generations ago). So, you <br>
>> > don't have to have a real "scientific computing" GPU to take advantage <br>
>> > of this -- most people probably already have the hardware. Still, I <br>
>> > understand that most people do not need/want to run double precision. <br>
>> > But, some do (and you have to if you are concerned with conservation <br>
>> > of energy -- the energy drift in single precision is substantial).<br>
>> ><br>
>> > So, I would like to ask what the level of effort to do this is <br>
>> > believed to be? Would it require a lot of new code, or would it be <br>
>> > porting the single precision code to double precision?<br>
>> ><br>
>> > Sincerely,<br>
>> > James<br>
>> ><br>
>> <br>
>> <br>
> <br>
> -- <br>
> Gromacs Developers mailing list<br>
> <br>
> * Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
> <br>
> * Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists">http://www.gromacs.org/Support/Mailing_Lists</a><br>
> <br>
> * For (un)subscribe requests visit<br>
> <a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20190724/98915b6c/attachment.html">http://maillist.sys.kth.se/pipermail/gromacs.org_gmx-developers/attachments/20190724/98915b6c/attachment.html</a>><br>
<br>
------------------------------<br>
<br>
-- <br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a>.<br>
<br>
End of gromacs.org_gmx-developers Digest, Vol 183, Issue 10<br>
***********************************************************<br>
</blockquote></div>
</blockquote></div><br></div></div>