<div dir="ltr">Hi,<div><br></div><div>Yes, agree we should disable keeping core files, now that they're not serving a direct purpose and occasionally cause problems.</div><div><br></div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Nov 9, 2017 at 10:32 AM Schulz, Roland <<a href="mailto:roland.schulz@intel.com">roland.schulz@intel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
This was added when we had intermittent crashes in Jenkins. We didn't find the cause without it because we couldn’t reproduce it.<br>
If we have this problem again in the future we could archive core files only if there is only a single one. If there is more than one than it shouldn't be a hard to reproduce crash.<br>
<br>
Roland<br>
<br>
-----Original Message-----<br>
From: <a href="mailto:gromacs.org_gmx-developers-bounces@maillist.sys.kth.se" target="_blank">gromacs.org_gmx-developers-bounces@maillist.sys.kth.se</a> [mailto:<a href="mailto:gromacs.org_gmx-developers-bounces@maillist.sys.kth.se" target="_blank">gromacs.org_gmx-developers-bounces@maillist.sys.kth.se</a>] On Behalf Of Szilárd Páll<br>
Sent: Thursday, November 9, 2017 9:20 AM<br>
To: Discussion list for GROMACS development <<a href="mailto:gmx-developers@gromacs.org" target="_blank">gmx-developers@gromacs.org</a>><br>
Subject: Re: [gmx-developers] jenkins killed by core files<br>
<br>
On Thu, Nov 9, 2017 at 5:59 PM, Szilárd Páll <<a href="mailto:pall.szilard@gmail.com" target="_blank">pall.szilard@gmail.com</a>> wrote:<br>
> Hi,<br>
><br>
> Earlier today (around 15:00 CET) a change resulted in ~1200 failed<br>
> tests and generated a volume of core files that swamped the jenkins<br>
> server. We spent far too much time tracking down the issue and<br>
> recovering from it and identified some critical issues in the setup<br>
> that we believe require changes.<br>
><br>
> Given that:<br>
> - we all are increasingly drafting and developing in gerrit, not even<br>
> testing locally,<br>
> - we don't have enough space for even the ~1200 cores files of a<br>
> single failed job could generate (let alone multiple iterations of a<br>
> buggy change),<br>
> - we're storing uncompressed cores files that we rarely if ever look<br>
> at we need to take action and prevent such time-consuming failures.<br>
><br>
> There are two options I see:<br>
> - I'll disable archiving core files (right away so the aforementioned<br>
> change won't bomb jenkins again ;)) -- after devs time saved by having<br>
> jenkins compile for them can now be spent on occasionally testing a<br>
> bit more locally (or on the build slaves if necessary) when hard to<br>
> track down bugs cause crashes;<br>
><br>
> - We have found a jenkis plugin that compresses artifacts; if this<br>
> reduces the size of archived data enough, we could try to deploy it<br>
> and re-enable core file archival. I have the suspicion that it won't<br>
> work due to a bug prevented us from using it to begin with, but I'll<br>
> have to check.<br>
<br>
Indeed, it is not compatible with the articaft copy plugin:<br>
<a href="https://issues.jenkins-ci.org/browse/JENKINS-22637" rel="noreferrer" target="_blank">https://issues.jenkins-ci.org/browse/JENKINS-22637</a><br>
<br>
Core file archival is now disabled for the master matrix, will do the same for 2016.<br>
<br>
><br>
> Cheers,<br>
> --<br>
> Szilárd<br>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.<br>
--<br>
Gromacs Developers mailing list<br>
<br>
* Please search the archive at <a href="http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists/GMX-developers_List</a> before posting!<br>
<br>
* Can't post? Read <a href="http://www.gromacs.org/Support/Mailing_Lists" rel="noreferrer" target="_blank">http://www.gromacs.org/Support/Mailing_Lists</a><br>
<br>
* For (un)subscribe requests visit<br>
<a href="https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers" rel="noreferrer" target="_blank">https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-developers</a> or send a mail to <a href="mailto:gmx-developers-request@gromacs.org" target="_blank">gmx-developers-request@gromacs.org</a>.</blockquote></div>