<html>
  <head>
    <meta content="text/html; charset=windows-1252"
      http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">On 2016-03-23 14:18, Mark Abraham
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAMNuMAT4GftoMf8t5uzB5o9E+CxFG2HEdi-JvbCDqukjXooY6w@mail.gmail.com"
      type="cite">
      <meta http-equiv="Content-Type" content="text/html;
        charset=windows-1252">
      <div dir="ltr">Hi,<br>
        <br>
        <div class="gmail_quote">
          <div dir="ltr">On Wed, Mar 23, 2016 at 1:56 PM Berk Hess &lt;<a
              moz-do-not-send="true" href="mailto:hess@kth.se"><a class="moz-txt-link-abbreviated" href="mailto:hess@kth.se">hess@kth.se</a></a>&gt;
            wrote:<br>
          </div>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF">
              <div>Hi,<br>
                <br>
                It's impractical and bad for performance to go from AoS
                to SoA.<br>
              </div>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>Sure, but that's what we do every MD step on typical
            CPUs. We do DD communication on XYZXYZXYZXYZ and then
            transpose them to XXXXYYYYZZZZ to use in short-ranged
            kernels. If we did DD communication on XXXXX....YYYYY.....
            ZZZZZZ.... then we still have gather-scatter overhead, but
            we don't have transposition in the mix.</div>
          <div><br>
          </div>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF">
              <div> The question is now if we can fix this issue or if
                we should always try to use .data() in inner-loops</div>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>I think you're missing my point... we've always done a
            cast from rvec * to real * to pass to inner loops. :-) Using
            data() is a similar idea - hence my suggestion of
            as_real_array(theVectorOfRVecs).</div>
        </div>
      </div>
    </blockquote>
    No. The bonded, vsite and constraint code uses rvec and we'd like to
    keep this in most cases to use rvec operations.<br>
    <blockquote
cite="mid:CAMNuMAT4GftoMf8t5uzB5o9E+CxFG2HEdi-JvbCDqukjXooY6w@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_quote">
          <div><br>
          </div>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF">
              <div> or if we should completely avoid
                std::vector&lt;RVec&gt; in performance sensitive code
                and use good old C pointers with nasty manual
                allocation.<br>
              </div>
            </div>
          </blockquote>
          <div><br>
          </div>
          <div>Allocation is irrelevant to how one uses it. One must
            have a real * to be sure the compiler isn't confused. To get
            that from a vector, we use data(). Whether the allocation of
            the memory is managed by std::vector or snew is immaterial.</div>
        </div>
      </div>
    </blockquote>
    That's true. But we need a way of avoiding coders from using [] on
    std::vector&lt;RVec&gt; in performance sensitive code. Or we could
    consider not allowing that anywhere, which is at least possible (I
    just discussed this with Mark).<br>
    <br>
    Berk<br>
    <blockquote
cite="mid:CAMNuMAT4GftoMf8t5uzB5o9E+CxFG2HEdi-JvbCDqukjXooY6w@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_quote">
          <div><br>
          </div>
          <div>Mark</div>
          <div><br>
          </div>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div text="#000000" bgcolor="#FFFFFF">
              <div> Cheers,<br>
                <br>
                Berk</div>
            </div>
            <div text="#000000" bgcolor="#FFFFFF">
              <div><br>
                <br>
                On 2016-03-23 13:43, Mark Abraham wrote:<br>
              </div>
            </div>
            <div text="#000000" bgcolor="#FFFFFF">
              <blockquote type="cite">
                <div dir="ltr">Hi,
                  <div><br>
                  </div>
                  <div>RVec was intended as a convenience feature for
                    people writing high-level, rather than low-level
                    code. I would indeed not advocate using it in
                    performance-sensitive mdrun code without checking
                    for such performance effects.</div>
                  <div><br>
                  </div>
                  <div>I think the real issue is closer to that the use
                    of *rvec* generally confuses compilers because
                    annotations about pointers do indeed get lost, and
                    perhaps [3] in real [3][] is not as close to being
                    part of the type in the compiler implementation as
                    we might expect. (For a long time, we've used very
                    little rvec close to the short-ranged kernels. The
                    first thing we do with the rvec * args to the SIMD
                    bondeds is to cast them to real *. I think this is
                    just as hard to police as Berk's concern about not
                    using std::vector&lt;RVec&gt;::operator[] naively -
                    the problem isn't the "fault" of std::vector or its
                    use.)</div>
                  <div><br>
                  </div>
                  <div>The use of rvec * engenders some transposing that
                    might be avoided if we had SoA rather than AoS
                    layout in mdrun. A single rvec is likely to have all
                    three coordinates on the same cache line, but if one
                    is anyway often using nearby atoms the caching
                    behaviour of three separate real * arrays should be
                    rather similar. One still tends to have
                    gather/scatter overhead, however.</div>
                  <div><br>
                  </div>
                  <div>Mark</div>
                  <div><br>
                  </div>
                </div>
                <br>
                <div class="gmail_quote">
                  <div dir="ltr">On Wed, Mar 23, 2016 at 11:24 AM Berk
                    Hess &lt;<a moz-do-not-send="true"
                      href="mailto:hess@kth.se" target="_blank">hess@kth.se</a>&gt;

                    wrote:<br>
                  </div>
                  <blockquote class="gmail_quote" style="margin:0 0 0
                    .8ex;border-left:1px #ccc solid;padding-left:1ex">
                    <div text="#000000" bgcolor="#FFFFFF">
                      <div>No.<br>
                        using the RVec * pointer obtained through
                        .data() already fixes the issue. We don't even
                        need as_rvec_array().<br>
                        <br>
                        The real issue is that it's near impossible to
                        enforce this and detect vector access of RVecs
                        if we start using std::vector everywhere through
                        the code.<br>
                        <br>
                        Cheers,<br>
                        <br>
                        Berk</div>
                    </div>
                    <div text="#000000" bgcolor="#FFFFFF">
                      <div><br>
                        <br>
                        On 2016-03-23 11:19, Mark Abraham wrote:<br>
                      </div>
                    </div>
                    <div text="#000000" bgcolor="#FFFFFF">
                      <blockquote type="cite">
                        <div dir="ltr">Hi,
                          <div><br>
                          </div>
                          <div>If so, then we might want a helper
                            function like as_rvec_array, but
                            as_real_array instead.</div>
                          <div><br>
                          </div>
                          <div>Mark</div>
                        </div>
                        <br>
                        <div class="gmail_quote">
                          <div dir="ltr">On Wed, Mar 23, 2016 at 10:51
                            AM Berk Hess &lt;<a moz-do-not-send="true"
                              href="mailto:hess@kth.se" target="_blank">hess@kth.se</a>&gt;


                            wrote:<br>
                          </div>
                          <blockquote class="gmail_quote"
                            style="margin:0 0 0 .8ex;border-left:1px
                            #ccc solid;padding-left:1ex">
                            <div text="#000000" bgcolor="#FFFFFF">
                              <div>On 2016-03-23 10:50, Erik Lindahl
                                wrote:<br>
                              </div>
                              <blockquote type="cite"> Hi,
                                <div><br>
                                </div>
                                <div>I haven’t followed the discussion
                                  in detail, but a long time ago I
                                  remember having simliar issues in the
                                  kernels when using a list[] of rvec[]
                                  (in plain-old-c, no c++) instead of
                                  extracting the pointer and handling
                                  the multiply-by-3 manually. Could it
                                  be something similar here, e.g. that
                                  the compiler things it does not know
                                  enough about RVec rather than
                                  something going from with the outer
                                  list?</div>
                              </blockquote>
                            </div>
                            <div text="#000000" bgcolor="#FFFFFF"> That
                              would be my guess. The index used in the
                              same loop comes from a vector as well and
                              doesn't seem to affect performance.</div>
                            <div text="#000000" bgcolor="#FFFFFF"><br>
                              <br>
                              Berk</div>
                            <div text="#000000" bgcolor="#FFFFFF"><br>
                              <blockquote type="cite">
                                <div><br>
                                </div>
                                <div>Cheers,</div>
                                <div><br>
                                </div>
                                <div>Erik</div>
                                <div><br>
                                </div>
                                <div>
                                  <div>
                                    <blockquote type="cite">
                                      <div>On 23 Mar 2016, at 10:44,
                                        Berk Hess &lt;<a
                                          moz-do-not-send="true"
                                          href="mailto:hess@kth.se"
                                          target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:hess@kth.se">hess@kth.se</a></a>&gt;

                                        wrote:</div>
                                      <br>
                                      <div>
                                        <div text="#000000"
                                          bgcolor="#FFFFFF">
                                          <div>On 2016-03-23 10:42, Mark
                                            Abraham wrote:<br>
                                          </div>
                                          <blockquote type="cite">
                                            <div dir="ltr">Hi,<br>
                                              <br>
                                              <div class="gmail_quote">
                                                <div dir="ltr">On Wed,
                                                  Mar 23, 2016 at 9:44
                                                  AM Berk Hess &lt;<a
                                                    moz-do-not-send="true"
href="mailto:hess@kth.se" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:hess@kth.se">hess@kth.se</a></a>&gt; wrote:<br>
                                                </div>
                                                <blockquote
                                                  class="gmail_quote"
                                                  style="margin:0 0 0
                                                  .8ex;border-left:1px
                                                  #ccc
                                                  solid;padding-left:1ex">Hi,<br>
                                                  <br>
                                                  Luckily Szilard does
                                                  thorough testing and
                                                  noticed a performance<br>
                                                  degradation in change
                                                  set 25 of <a
                                                    moz-do-not-send="true"
href="https://gerrit.gromacs.org/#/c/5232/" target="_blank"><a class="moz-txt-link-freetext" href="https://gerrit.gromacs.org/#/c/5232/">https://gerrit.gromacs.org/#/c/5232/</a></a>
                                                  The<br>
                                                  only signifcant change
                                                  with respect to
                                                  previous sets is
                                                  replacing C<br>
                                                  pointers by
                                                  std::vector. I traced
                                                  the performance
                                                  difference back to a<br>
                                                  single loop, which
                                                  must have become
                                                  several factors slower
                                                  to explain<br>
                                                  the time difference. I
                                                  get the performance
                                                  back when replacing
                                                  the<br>
                                                  vector by a pointer
                                                  extracted with
                                                  .data(), see below. I
                                                  looked at the<br>
                                                  assembly code from gcc
                                                  5.3.1 and the vector
                                                  case generated 200
                                                  extra<br>
                                                  instructions, which
                                                  makes it difficult to
                                                  see what the actual
                                                  difference<br>
                                                  is. The pointer case
                                                  uses a lot of vmovss
                                                  and vaddss, which the
                                                  vector<br>
                                                  one does much less,
                                                  but this is only
                                                  serial SIMD
                                                  instruction. I thought<br>
                                                  that [] in vector
                                                  might does bounds
                                                  checking,</blockquote>
                                                <div><br>
                                                </div>
                                                <div>Definitely not in
                                                  release builds.</div>
                                                <div> </div>
                                                <blockquote
                                                  class="gmail_quote"
                                                  style="margin:0 0 0
                                                  .8ex;border-left:1px
                                                  #ccc
                                                  solid;padding-left:1ex">
                                                  but it seems it does
                                                  not.<br>
                                                  Can anyone explain why
                                                  the vector case can be
                                                  so slow?</blockquote>
                                                <blockquote
                                                  class="gmail_quote"
                                                  style="margin:0 0 0
                                                  .8ex;border-left:1px
                                                  #ccc
                                                  solid;padding-left:1ex">
                                                  If this is a general
                                                  issue (with RVec or
                                                  more?), we need to
                                                  always extra<br>
                                                  a pointer with .data()
                                                  for use in all
                                                  inner-loops. This is
                                                  pretty<br>
                                                  annoying and difficult
                                                  to enforce.<br>
                                                  <br>
                                                  Cheers,<br>
                                                  <br>
                                                  Berk<br>
                                                  <br>
                                                                       
                                                    const
                                                  std::vector&lt;RVec&gt;
                                                  f_foreign =<br>
                                                  idt_foreign-&gt;force<br>
                                                </blockquote>
                                                <div><br>
                                                </div>
                                                <div>This does a copy of
                                                  the vector, and
                                                  doesn't seem to be in
                                                  any version of this
                                                  patch in gerrit. Is
                                                  this what you meant to
                                                  write?</div>
                                              </div>
                                            </div>
                                          </blockquote>
                                          I tried this. But my original
                                          "vectorized" patch set took a
                                          pointer from idt_foreign and
                                          did not copy the vector, that
                                          gives the same, slow,
                                          performance.<br>
                                          <br>
                                          Berk<br>
                                          <blockquote type="cite">
                                            <div dir="ltr">
                                              <div class="gmail_quote">
                                                <div><br>
                                                </div>
                                                <div>Mark</div>
                                                <div><br>
                                                </div>
                                                <blockquote
                                                  class="gmail_quote"
                                                  style="margin:0 0 0
                                                  .8ex;border-left:1px
                                                  #ccc
                                                  solid;padding-left:1ex">
                                                  or<br>
                                                                       
                                                    const RVec         
                                                       *f_foreign   =<br>
idt_foreign-&gt;force.data();<br>
                                                  <br>
                                                                       
                                                     int natom =
                                                  atomList-&gt;atom.size();<br>
                                                                       
                                                     for (int i = 0; i
                                                  &lt; natom; i++)<br>
                                                                       
                                                     {<br>
                                                                       
                                                         int ind =
                                                  atomList-&gt;atom[i];<br>
                                                                       
                                                       
                                                   rvec_inc(f[ind],
                                                  f_foreign[ind]);<br>
                                                                       
                                                     }<br>
                                                  --<br>
                                                  Gromacs Developers
                                                  mailing list<br>
                                                  <br>
                                                  * Please search the
                                                  archive at <a
                                                    moz-do-not-send="true"
href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List"
                                                    target="_blank"><a class="moz-txt-link-freetext" href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a></a>
                                                  before posting!<br>
                                                  <br>
                                                  * Can't post? Read <a
moz-do-not-send="true"
                                                    href="http://www.gromacs.org/Support/Mailing_Lists"
                                                    target="_blank"><a class="moz-txt-link-freetext" href="http://www.gromacs.org/Support/Mailing_Lists">http://www.gromacs.org/Support/Mailing_Lists</a></a><br>
                                                  <br>
                                                  * For (un)subscribe
                                                  requests visit<br>
                                                  <a
                                                    moz-do-not-send="true"
href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers"
                                                    rel="noreferrer"
                                                    target="_blank"><a class="moz-txt-link-freetext" href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a></a>
                                                  or send a mail to <a
moz-do-not-send="true" href="mailto:gmx-developers-request@gromacs.org"
                                                    target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a></a>.<br>
                                                </blockquote>
                                              </div>
                                            </div>
                                            <br>
                                            <fieldset></fieldset>
                                            <br>
                                          </blockquote>
                                          <br>
                                        </div>
                                        -- <br>
                                        Gromacs Developers mailing list<br>
                                        <br>
                                        * Please search the archive at <a
                                          moz-do-not-send="true"
                                          href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List"
                                          target="_blank"><a class="moz-txt-link-freetext" href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a></a>
                                        before posting!<br>
                                        <br>
                                        * Can't post? Read <a
                                          moz-do-not-send="true"
                                          href="http://www.gromacs.org/Support/Mailing_Lists"
                                          target="_blank"><a class="moz-txt-link-freetext" href="http://www.gromacs.org/Support/Mailing_Lists">http://www.gromacs.org/Support/Mailing_Lists</a></a><br>
                                        <br>
                                        * For (un)subscribe requests
                                        visit<br>
                                        <a moz-do-not-send="true"
href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers"
                                          target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>
                                        or send a mail to <a
                                          moz-do-not-send="true"
                                          href="mailto:gmx-developers-request@gromacs.org"
                                          target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:gmx-developers-request@gromacs.org">gmx-developers-request@gromacs.org</a></a>.</div>
                                    </blockquote>
                                  </div>
                                  <br>
                                </div>
                                <br>
                                <fieldset></fieldset>
                                <br>
                              </blockquote>
                              <br>
                            </div>
                            --<br>
                            Gromacs Developers mailing list<br>
                            <br>
                            * Please search the archive at <a
                              moz-do-not-send="true"
                              href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List"
                              target="_blank"><a class="moz-txt-link-freetext" href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a></a>
                            before posting!<br>
                            <br>
                            * Can't post? Read <a
                              moz-do-not-send="true"
                              href="http://www.gromacs.org/Support/Mailing_Lists"
                              rel="noreferrer" target="_blank"><a class="moz-txt-link-freetext" href="http://www.gromacs.org/Support/Mailing_Lists">http://www.gromacs.org/Support/Mailing_Lists</a></a><br>
                            <br>
                            * For (un)subscribe requests visit<br>
                            <a moz-do-not-send="true"
href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers"
                              rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>
                            or send a mail to <a moz-do-not-send="true"
href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote>
                        </div>
                        <br>
                        <fieldset></fieldset>
                        <br>
                      </blockquote>
                      <br>
                    </div>
                    --<br>
                    Gromacs Developers mailing list<br>
                    <br>
                    * Please search the archive at <a
                      moz-do-not-send="true"
                      href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List"
                      rel="noreferrer" target="_blank"><a class="moz-txt-link-freetext" href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a></a>
                    before posting!<br>
                    <br>
                    * Can't post? Read <a moz-do-not-send="true"
                      href="http://www.gromacs.org/Support/Mailing_Lists"
                      rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
                    <br>
                    * For (un)subscribe requests visit<br>
                    <a moz-do-not-send="true"
href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers"
                      rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>
                    or send a mail to <a moz-do-not-send="true"
                      href="mailto:gmx-developers-request@gromacs.org"
                      target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote>
                </div>
                <br>
                <fieldset></fieldset>
                <br>
              </blockquote>
              <br>
            </div>
            --<br>
            Gromacs Developers mailing list<br>
            <br>
            * Please search the archive at <a moz-do-not-send="true"
              href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List"
              rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a>
            before posting!<br>
            <br>
            * Can't post? Read <a moz-do-not-send="true"
              href="http://www.gromacs.org/Support/Mailing_Lists"
              rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
            <br>
            * For (un)subscribe requests visit<br>
            <a moz-do-not-send="true"
href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers"
              rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a>
            or send a mail to <a moz-do-not-send="true"
              href="mailto:gmx-developers-request@gromacs.org"
              target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
    </blockquote>
    <br>
  </body>
</html>