Re: minutes of ESC call ...

classic Classic list List threaded Threaded
5 messages Options
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...

On 09/05/2019 16:46, Michael Meeks wrote:

> * Jenkins / CI update (Christian)
>      from:Thu May  2 13:58:26 2019
>      master linux rel  jobs: 129 ok: 129 ko:   0 fail ratio:  0.00 % break:   0 broken duration: 0.00%
>      master linux dbg  jobs:  86 ok:  83 ko:   3 fail ratio:  3.49 % break:   2 broken duration: 4.03%
>      master mac rel    jobs:  75 ok:  70 ko:   5 fail ratio:  6.67 % break:   4 broken duration: 5.67%
>      master mac dbg    jobs:  69 ok:  62 ko:   7 fail ratio: 10.14 % break:   4 broken duration:12.32%
>      master win rel    jobs:  86 ok:  63 ko:  23 fail ratio: 26.74 % break:  17 broken duration:23.09%
>      master win dbg    jobs:  90 ok:  75 ko:  15 fail ratio: 16.67 % break:  12 broken duration:12.80%
>      master win64 dbg  jobs:  88 ok:  67 ko:  21 fail ratio: 23.86 % break:  16 broken duration:25.91%
>      lo-5.3 mac        jobs:   0 ok:   0 ko:   0 fail ratio:  0.00 % break:   0 broken duration: 0.00%
>      lo-5.4 mac        jobs:   0 ok:   0 ko:   0 fail ratio:  0.00 % break:   0 broken duration: 0.00%
>      master gerrit lin jobs: 451 ok: 331 ko:  21 fail ratio:  4.66% time for ok: mean:  21 median:  21
>      master gerrit plg jobs: 462 ok: 297 ko:  92 fail ratio: 19.91% time for ok: mean:  36 median:  33
>      master gerrit win jobs: 498 ok: 292 ko:  75 fail ratio: 15.06% time for ok: mean:  65 median:  64
>      master gerrit mac jobs: 478 ok: 291 ko:  74 fail ratio: 15.48% time for ok: mean:  62 median:  37
>      master gerrit all jobs: 434 ok: 240 ko: 165 fail ratio: 38.02% time for ok: mean: 120 median:  85
>     + https://dev-www.libreoffice.org/tmp/gerrit_190509.html
>     + notable test failures in last seven days:
>        4 JunitTest_forms_unoapi_2
>        5 UITest_conditional_format
>        7 CppunitTest_sc_bugfix_test
>       11 UITest_writer_tests5
>     + dis-connected bots over the weekend
>     + one failing mac killed a lot of builds until disabled
>     + 2 Mac Pros dis-connected, one mis-behaving, one in power-save state ?
>        + right now – 1 Mac short.
>        + 1 is at Norbert’s company – needs an ask for a reboot, or re-route
>     + could we get another Mac Pro for hosting ? (Thorsten)
>        + hard to host – but no mounting for racks (Christian)
>           + newer Mac Minis are perhaps suitable.
>        + Macs shouldn’t be the bottleneck anyway
>        + Windows is the limiting factor

Whenever I recently happened to look at Gerrit/Jenkins builds that
hadn't finished yet, it was the Mac build that was still outstanding, so
I think we do have a bottleneck there after all.

Witness:

* My "May 16 22:51" comment at
<https://gerrit.libreoffice.org/#/c/72433/>: "gerrit_mac is hopelessly
overloaded right now, with a 30+ long build queue (waiting builds #33071
through #33095)" (though that's rather 20+ than 30+, of course)

* Armin's "May 16 23:32" comment at
<https://gerrit.libreoffice.org/#/c/72401/>: "sooo... Build is blocked
prob due to Mac missing - due to no Macs available...?"
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Thorsten Behrens-6 Thorsten Behrens-6
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...

Stephan Bergmann wrote:
> Whenever I recently happened to look at Gerrit/Jenkins builds that
> hadn't finished yet, it was the Mac build that was still
> outstanding, so I think we do have a bottleneck there after all.
>
Yeah, same anecdotal evidence here. Is the assumption we don't need
more Mac build power based on hard data?

Cheers,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (1K) Download Attachment
Luboš Luňák Luboš Luňák
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...

On Friday 17 of May 2019, Thorsten Behrens wrote:
> Stephan Bergmann wrote:
> > Whenever I recently happened to look at Gerrit/Jenkins builds that
> > hadn't finished yet, it was the Mac build that was still
> > outstanding, so I think we do have a bottleneck there after all.
>
> Yeah, same anecdotal evidence here. Is the assumption we don't need
> more Mac build power based on hard data?


 IIRC one Mac bot had some problems a couple of days back, so maybe the
increased Mac load was just caused by catching up there (I certainly remember
waiting for those to finally pass). If I right now look at gerrit master
builds in progress, none of them is currently waiting exclusively for Mac,
but some of them are for Windows.


--
 Luboš Luňák
 [hidden email]
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Noel Grandin-2 Noel Grandin-2
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...



On Sat, 18 May 2019 at 12:15, Luboš Luňák <[hidden email]> wrote:
On Friday 17 of May 2019, Thorsten Behrens wrote:
> Stephan Bergmann wrote:
> > Whenever I recently happened to look at Gerrit/Jenkins builds that
> > hadn't finished yet, it was the Mac build that was still
> > outstanding, so I think we do have a bottleneck there after all.
>
> Yeah, same anecdotal evidence here. Is the assumption we don't need
> more Mac build power based on hard data?

Periodically the mac build bots maintained by Norbert will go offline or become unreachable and that can take a couple of days to resolve, and once they come back the build queue can take a day or so to get back to normal.

If we want more reliability, we are going to need more mac resources hosted where we can get to them faster.

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Luboš Luňák Luboš Luňák
Reply | Threaded
Open this post in threaded view
|

Jenkins build times (Re: minutes of ESC call ...)

In reply to this post by Luboš Luňák
On Saturday 18 of May 2019, Luboš Luňák wrote:
> On Friday 17 of May 2019, Thorsten Behrens wrote:
> > Yeah, same anecdotal evidence here. Is the assumption we don't need
> > more Mac build power based on hard data?
>
>  IIRC one Mac bot had some problems a couple of days back, so maybe the
> increased Mac load was just caused by catching up there (I certainly
> remember waiting for those to finally pass). If I right now look at gerrit
> master builds in progress, none of them is currently waiting exclusively
> for Mac, but some of them are for Windows.


 BTW, I've noticed that all setups except for gcc use --enable-dbgutil, which
in turn by default does --disable-optimized. Given that Jenkins builds also
run all checks and that takes time that cannot be avoided/cached in any way,
wouldn't it be in total faster to go with --enable-optimized? I'm not sure
how that'd turn out with the Windows build, but at least the ccache-enabled
ones could easily gain on that. I know unittests also try to print out a
backtrace if they crash, but do we need that for Jenkins builds? Quite
possibly adding --enable-optimized --disable-symbols could save quite some
build power.

 Also, every build has to build all the stuff in external/, over and over
again. That shouldn't matter much with ccache, but presumably the Windows
build spends quite some time there. We could try some silly^H^Hinteresting
improvements there, such as adding PCHs even for those.

--
 Luboš Luňák
 [hidden email]
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice