<div dir="ltr">Hi,<div><br></div><div>The situation isn't as simple as that:</div><div dir="ltr"><div><ul><li>There certainly are changes that fail because they have not been rebased, but those are easy to identify from the error messages that get posted back to Gerrit and say that the required build scripts cannot be found.</li><li>There also are a few changes that fail on purpose, since they are waiting for the C++11 change to move forward, and require some stuff from there (and the new matrix anyways would also fail with the C++11 change).</li><li>Additionally, many changes that have been rebased also have one or a few failing builds. Persistently retriggering the failing builds leads to the builds eventually succeeding, but this is not a sustainable situation. The situation isn't exactly new (we've had random failures earlier as well, with similar symptoms), but the volume of these is now much higher. This really looks like that either Jenkins itself (the software, or the hardware/virtualization layer it runs on) is flaky, or that our configuration creates way too much peak load somewhere, causing something to time out or otherwise fail (since typically the error messages are of the sort "Channel is already closed").</li><li>On top of that, there likely is at least one deadlock hiding somewhere in mdrun, since occasionally the mdrun integration tests and/or regression tests may hang for 15 minutes (before Jenkins kills them). It's just a hunch that this deadlock also triggers more easily when there is more load on the system.</li></ul><div>The new (temporary) setup with double jobs on nearly everything puts more load on Jenkins, so it might just have tipped the balance beyond a point where things were working ~OK (at least, when you didn't upload too many changes at the same time). But it would be nice to iron out these issues now, instead of just ignoring it in hopes that it goes back to manageable levels when we reduce the load.</div></div><div><br></div>Just my two cents,</div><div dir="ltr">Teemu</div><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr">On Sun, Sep 6, 2015 at 8:20 PM Mark Abraham <<a href="mailto:mark.j.abraham@gmail.com" target="_blank">mark.j.abraham@gmail.com</a>> wrote:</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<p dir="ltr">It's submitted to master already, so just rebase to HEAD as/when you want.</p>
<p dir="ltr">Mark</p><div class="gmail_quote"><div dir="ltr">On Sun, 6 Sep 2015 15:25 David van der Spoel <<a href="mailto:spoel@xray.bmc.uu.se" target="_blank">spoel@xray.bmc.uu.se</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 02/09/15 23:12, Mark Abraham wrote:<br>> With 5.1 off the table, we're implementing some much-needed updates to<br>
> the way we handle Jenkins verification of GROMACS.<br>
><br>
> Teemu's rewritten the scripts we use to implement the various kinds of<br>
> verification jobs, which will let us maintain and extend in much less<br>
> ad-hoc fashion. Some parts of those scripts will now live in the GROMACS<br>
> source repository, so that they can change in step with code changes.<br>
> We've already submitted that script to master, so when you rebase<br>
> patches over 0ce920a017, Jenkins will be able to use its new toys.<br>
<br>
So is this the reason that most patches fail right now?<br>
Where is this patch in gerrit? I can not seem to find it...<br></blockquote></div></blockquote></div></div></div>