test infrastructure ideas appreciated ...

classic Classic list List threaded Threaded
52 messages Options
123 « Prev
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: regular lcov reports (was: test infrastructure ideas appreciated ...)

Hi,

On Sat, Jun 13, 2015 at 10:05:52AM -0500, Norbert Thiebaud wrote:
> why didn't I think of that.. oh wait

Hey, this is not about what we think about, but about what we intend to
implement and document so that it is available for others to use. ;)

Care to update

 https://wiki.documentfoundation.org/Development/Lcov

with the links and hints on where the tooling/source for that is hosted?

Also:

What kind of config does it run? Something "-O0" or "-fno-inline" and thus
providing useful info, or does it reuse something else.

> But true, it is not weekly... it's daily....

Does it keep historic records at least of the stats, thus allows watching
trends?

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Norbert Thiebaud Norbert Thiebaud
Reply | Threaded
Open this post in threaded view
|

Re: regular lcov reports (was: test infrastructure ideas appreciated ...)

On Sat, Jun 13, 2015 at 10:23 AM, Bjoern Michaelsen
<[hidden email]> wrote:
> Hi,
>
> On Sat, Jun 13, 2015 at 10:05:52AM -0500, Norbert Thiebaud wrote:
>> why didn't I think of that.. oh wait
>
> Hey, this is not about what we think about, but about what we intend to
> implement

It is not an 'intention'. A volunteer, mhoes, worked on a script to run lcov,
I helped him make that script jenkins ready, and found a home for it.

> and document so that it is available for others to use. ;)

you said:beside you said: "another that would be helpful would be a
box creating lcov reports, say once a week,"

This exist and has been running daily for months...
It has been mentioned in the ESC call, back in March, a couple of
times.. the minutes indicate that you where present and even commented
during that topic.
And the 'documention' for other to _use_ is:
go to lcov.libreoffice.org

>
> Also:
>
> What kind of config does it run? Something "-O0" or "-fno-inline" and thus
> providing useful info, or does it reuse something else.

Running ./configure with '--enable-debug' '--enable-python=internal'
'--disable-online-update' '--without-system-libs'
'--without-system-headers' '--disable-ccache' '--disable-coinmp'
'--srcdir=/home/tdf/lode/jenkins/workspace/internal_lcov_master'
'--enable-option-checking=fatal'


>
>> But true, it is not weekly... it's daily....
>
> Does it keep historic records at least of the stats, thus allows watching
> trends?

No. it is an automated job that do a debug build, run make check and
run lcov on it, daily.

Patch welcome.

I stepped-in to make resources available, I'll be glad to help anyone
that want to improve on it, secure and access the necessary infra
resources.
But I'm not volunteering, at this time, to adopt the baby.

Norbert
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: regular lcov reports (was: test infrastructure ideas appreciated ...)

Hi,

On Sat, Jun 13, 2015 at 12:10:31PM -0500, Norbert Thiebaud wrote:
> > What kind of config does it run? Something "-O0" or "-fno-inline" and thus
> > providing useful info, or does it reuse something else.
>
> Running ./configure with '--enable-debug' '--enable-python=internal'
> '--disable-online-update' '--without-system-libs'
> '--without-system-headers' '--disable-ccache' '--disable-coinmp'
> '--srcdir=/home/tdf/lode/jenkins/workspace/internal_lcov_master'
> '--enable-option-checking=fatal'

And no CFLAGS/CXXFLAGS/LDFLAGS for -ftest-coverage?

> > Does it keep historic records at least of the stats, thus allows watching
> > trends?
>
> No. it is an automated job that do a debug build, run make check and
> run lcov on it, daily.
>
> Patch welcome.

Whereto? As in: In which repo is the script for the job hosted?

> But I'm not volunteering, at this time, to adopt the baby.

Its great that we have it this far. Just would love to have it more known and
open to tinkering for everyone.

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Norbert Thiebaud Norbert Thiebaud
Reply | Threaded
Open this post in threaded view
|

Re: regular lcov reports (was: test infrastructure ideas appreciated ...)

On Sat, Jun 13, 2015 at 1:40 PM, Bjoern Michaelsen
<[hidden email]> wrote:
>
> Whereto? As in: In which repo is the script for the job hosted?

there is not tools that I am aware of that do lcov trending... so nowhere.
but patch welcome... dev-tools.git or buildbot.git  for instance

>
> Its great that we have it this far. Just would love to have it more known and
> open to tinkering for everyone.

everyone is free to tinker to his heart content.. once that yield
something useful, that we can deploy in prod, I'll be happy to help
with that.

if you want to take ownership of that, I'll be more than happy to give
you the needed auth on jenkins and on tb31

Norbert
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: regular lcov reports (was: test infrastructure ideas appreciated ...)

Hi,

On Sat, Jun 13, 2015 at 02:07:14PM -0500, Norbert Thiebaud wrote:
> if you want to take ownership of that, I'll be more than happy to give
> you the needed auth on jenkins and on tb31

I'd love to invest some time into tracking long term trends there, yes. It
should be reasonably easy to get something started there.

OTOH, I dont think I need or should need auth on any machine for that: This
can be tested well locally and thus just having whatever tooling runs on those
boxes hosted in a git repo should allow all the tinkering needed.

Lets discuss details in an upcoming ESC call.

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: test-infra proposal: master-tested branch (was: test infrastructure ideas appreciated ...)

In reply to this post by Bjoern Michaelsen
Hi,

On Wed, Jun 10, 2015 at 03:04:41PM +0200, Bjoern Michaelsen wrote:
> Having master-tested might help us a lot in having more of the first and less
> of the second.

I learned today that I could have spared myself the work of describing this, as
git comes with a best practices advice to do pretty much the same. Type:

 git help workflows

in your shell and you get it as a man page (it names the branches 'master' and
'next').

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Norbert Thiebaud Norbert Thiebaud
Reply | Threaded
Open this post in threaded view
|

Re: test-infra proposal: master-tested branch (was: test infrastructure ideas appreciated ...)

On Wed, Jun 17, 2015 at 6:17 AM, Bjoern Michaelsen
<[hidden email]> wrote:

> Hi,
>
> On Wed, Jun 10, 2015 at 03:04:41PM +0200, Bjoern Michaelsen wrote:
>> Having master-tested might help us a lot in having more of the first and less
>> of the second.
>
> I learned today that I could have spared myself the work of describing this, as
> git comes with a best practices advice to do pretty much the same. Type:
>
>  git help workflows
>
> in your shell and you get it as a man page (it names the branches 'master' and
> 'next').

That nice, but that is not at all the workflow we use

For one, we do not have a maintainer-driven process
All that man page describe is from 1/ pull-branch +merge based
development 2/ with a maintainer handling the pulls and merge
'promotion'.
last but not least, multi-platform is not addressed, because the
tagging/promotion mechanism described is not meant to be
automated/ci-driven

So to restate the actual real requirement for us.

"Having an easy and reliable way for people to checkout a known-green
recent commit form master so that they can rely on gerrit build result
as an indication of the performance of their patch rather than the
hazardous state of master."



Norbert
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Anytime builder VMs for devs (was: test infrastructure ideas appreciated ...)

In reply to this post by Michael Meeks-5
Hi,

On Wed, Jun 03, 2015 at 02:33:23PM +0100, Michael Meeks wrote:
> Constructive thoughts appreciated in reply here.

Budget for 2-5 of our Hackfest VMs[1] to always be on standby for someone
wanting to do a fast build, and provision for Cloph to be able to hand out
log-ins at a quick ping on IRC.

Rationale:
- Cloud instances are already faster than even the fastest, most expensive and
  newest notebook and unlike the latter they do not run out of battery if you
  start compiling or do a 'make check' e.g. on an airport.
- with cloud instances getting cheaper and notebook performance has been
  stagnating in recent years, this trend will be accelerating
- building on a fast remote machine will encourage the test-oriented mindset
  that we aim for

To extend on the last point: Yes, logging into a remote machine via VNC/X11 is
somewhat cumbersome. However, that is exactly encouraging breaking bad old
habits (ad-hoc manual testing with UI-interaction) in favour of good practices:
verifying your code and changes to work with tests that do not need manual
interaction on a UI. These tests ultimately will also allow to test the
code/changes with CI on all platforms.

So in short: Remote builders instances for devs encourage good practices, are
faster than what is possible locally right now and are -- if we set this up
right -- significantly more flexible and faster to get to a developer in need.

Best,

Bjoern
 
[1] https://wiki.documentfoundation.org/Hackfests/VMs
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Norbert Thiebaud Norbert Thiebaud
Reply | Threaded
Open this post in threaded view
|

Re: Anytime builder VMs for devs (was: test infrastructure ideas appreciated ...)

On Thu, Jun 18, 2015 at 4:14 AM, Bjoern Michaelsen
<[hidden email]> wrote:
> Hi,
>
> On Wed, Jun 03, 2015 at 02:33:23PM +0100, Michael Meeks wrote:
>>       Constructive thoughts appreciated in reply here.
>
> Budget for 2-5 of our Hackfest VMs[1] to always be on standby for someone
> wanting to do a fast build, and provision for Cloph to be able to hand out
> log-ins at a quick ping on IRC.

The budget is already there.
some tooling to do that smartly is what is needed, as relying on cloph
being alvailable for a quick ping on IRC does not scale very well.

iow to use spot instance provisionning, and to control how many of
them there is and turn them off automatically after a while (based on
idling etc.)
there ia also the credential part of it... and the discovery part of it....
One possibility would be:
have a minimal template root disk with a basic setup and a salt-client
preset to call home a given salt-server (on our infra)
have a script on the salt-server do
1/ accept incomming new client
2/ run some high-state on such client as needed (that will setup
things like user + ssh-keys for the requester on the client) and
report when the box is ready.

Note: using spot instance (which are very cheap on linux) means you
really want to use that for build.. not for coding as the instance may
disappear on you
so code on your laptop send patch/rsync/whatever code-change to the
vm, build and test there.

Note: for actual hackfest, we may provision real instance (not spot)
for the duration of the hakfest..
you can get a quite good box (like the one that I use to do windows
build in 30-40 minutes) at $60 for 72 hours
and for a hackfest we could have a setup with say 6 of these in a
icecream network... for less than $400 for the week-end
which is negligible comparing to travel expenses for these things...
and could support easely a dozen of hackers or more

Norbert
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

track and present status of individual tests (was: test infrastructure ideas appreciated ...)

In reply to this post by Michael Meeks-5
Hi,

On Wed, Jun 03, 2015 at 02:33:23PM +0100, Michael Meeks wrote:
> Constructive thoughts appreciated in reply here.

This has some synergies with the dashboard proposal[1] but it is sufficiently
distict to be included as an ideas on its own:

We lack a good representation of test status. tinderbox.libreoffice.org is only
very basic and overloads one view trying to show way too much at one:
- e.g. http://tinderbox.libreoffice.org/MASTER/status.html shows:
    - when something broke
    - where (on which machine) something broke
      (which is already creates a ~useless table a fullscreen wide on
      my 30" 21:9 screen at default zoom levels -- quite an "achievement")
  however it doesnt show:
    - _what_ (which test or module) broke

The "what" is the most important information for a developer checking if he
broke something, followed by the "when", while the "on which machine" is
less relevant in most cases.

If we really want to value tests we should be able to present a view on our
automated testing that shows what broke first -- and then allows ivestigating
the when and where from there. Take the "Test Statistics Grid" at the end of:

 https://jenkins.qa.ubuntu.com/view/vivid/view/AutoPkgTest/

as an starting point, showing e.g. the status and the stability of each and every
CppunitTest_ JunitTest_ and PythonTest_ individually.

Best,

Bjoern


[1] http://nabble.documentfoundation.org/TDF-Grant-Request-Proposal-LibreOffice-project-dashboard-quot-All-about-LibreOffice-quot-td4151652.html
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Christian Lohmaier-3 Christian Lohmaier-3
Reply | Threaded
Open this post in threaded view
|

Re: Anytime builder VMs for devs (was: test infrastructure ideas appreciated ...)

In reply to this post by Bjoern Michaelsen
Hi Björn, *,

On Thu, Jun 18, 2015 at 11:14 AM, Bjoern Michaelsen
<[hidden email]> wrote:
> On Wed, Jun 03, 2015 at 02:33:23PM +0100, Michael Meeks wrote:
>>       Constructive thoughts appreciated in reply here.
>
> Budget for 2-5 of our Hackfest VMs[1] to always be on standby for someone
> wanting to do a fast build, and provision for Cloph to be able to hand out
> log-ins at a quick ping on IRC.

What is the suggested "standby time"? Creating a disk-image/EC2
virtual machine from a stored snapshot takes a while (>20min).

ciao
Christian
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: test infrastructure ideas appreciated ... - Help / Doc screenshots ...

In reply to this post by Thorsten Behrens
On 06/05/2015 02:26 PM, Thorsten Behrens wrote:

> Michael Meeks wrote:
>> * Find some way to automate the creation of (translated)
>>  screenshots for help and documentation
>> + this can be used to keep the help up-to-date
>> + and also to test significant dialog open/close with it
>> + and to add more screenshots (cheaply) to
>>  the documentation
>>
> Nice - and we'd get a 'smoke-test all our dialogs' for free with
> it. :)

Now that that screen shooting feature is available,
<https://wiki.documentfoundation.org/Development/Tinderbox#14> includes
'make screenshot' in it's 'make check' phase.  (Locally hacked into the
buildbot scripting for now.  It sometimes prints "make: Nothing to be
done for ´screenshot´" but still apparently does run the relevant tests;
go figure.)
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
123 « Prev