Re: [Libreoffice-qa] minutes of ESC call ...

classic Classic list List threaded Threaded
12 messages Options
Budea, Áron Budea, Áron
Reply | Threaded
Open this post in threaded view
|

Re: [Libreoffice-qa] minutes of ESC call ...

Hi,

On Monday, September 19, 2016 12:43 CEST, Xisco Fauli <[hidden email]> wrote:
 
>      - needAdvise: Used when help from developers in needed to confirm
> an UNCONFIRMED bug. Info:
> https://wiki.documentfoundation.org/QA/Bugzilla/Fields/Keywords#needAdvice.
>          * Problem 1: Name is confusing.
>              + Action: Rename it to 'needsConfirmationAdvise'
>          * Problem 2: No New or resolved bugs should use it.
>              + Action: Clean it up and create a new gardening task.

I think it would make sense to keep needAdvice keyword unrestricted from states.

Reasons:
- NEW doesn't always mean QA was able to reproduce the bug (if TRIAGED status
was in place, that would), see bug 101898 for example.
- if further details are posted in a bug report that require timely reaction, but QA can't evaluate
them, advice from developers would be needed regardless of status. Again, a recent example:
bug 101821 (a question in this case, I haven't added the keyword, yet).

My suggestion is to name it something like needsDevAdvice, remove if advice was given, keyword
was used unnecessarily or report got closed. It should still be used very conservatively, of course.

 >      - needsDevEval: Should be used when a plausible easyhack lacks the
> code pointer, the difficulty, the topic or the skill and a developer > needs to provide the information missing.
>          * Problem 1: It has been used to propose easyhacks over the > last months.
>              + Action: Update the wiki accordingly:
> https://wiki.documentfoundation.org/QA/Bugzilla/Fields/Keywords#needsDevEval 
> and
> https://wiki.documentfoundation.org/Development/EasyHacks/Creating_a_new_Easy_Hack

What is the problem here? Is a proposed easy hack fundamentally different from an easy hack that
lacks necessary details? Both are for the same people: the developers who have understanding of
the relevant pieces of code.

What is the workflow if I see a bug report that might be easy enough to be an easy hack, but I have
zero knowledge about the underlying code? My ability to evaluate it stops before the first action step
detailed in response to Problem 4. So, CC Jan and ask his opinion? Does any keyword need to be set?

Thanks,
Aron


_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
jan iversen jan iversen
Reply | Threaded
Open this post in threaded view
|

Re: [Libreoffice-qa] minutes of ESC call ...


> What is the workflow if I see a bug report that might be easy enough to be an easy hack, but I have
> zero knowledge about the underlying code? My ability to evaluate it stops before the first action step
> detailed in response to Problem 4. So, CC Jan and ask his opinion? Does any keyword need to be set?

Whenever somebody sets keyword=easyhack I get a message next morning, so need for a keyword in that case. No need to put me CC.

I control all easyhack and currently set status=NEEDINFO if information is missing. I agree that using keyword=needsDevEval instead would make more sense.

rgds
jan I.

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: [Libreoffice-qa] minutes of ESC call ...

In reply to this post by Budea, Áron
Hi,

On Tue, Sep 20, 2016 at 12:55:59AM +0200, Áron Budea wrote:
>
> I think it would make sense to keep needAdvice keyword unrestricted from states.
>
> Reasons:
> - NEW doesn't always mean QA was able to reproduce the bug (if TRIAGED status
> was in place, that would), see bug 101898 for example.

NEW _should_ mean triaged for all matters. That it is not called the more
descriptive TRIAGED (like it is on e.g. launchpad) is due to historic reasons.
A bug on bugs.documentfoundation.org should be in NEW only if a developer could
start working on it. Otherwise it should be in UNCOFIRMED or NEEDINFO (as
should be for 101898: It was independently confirmed, but there still isnt a
reliable reproduction scenario that allows a developer to work on it).

In general, bug states should be a rather clear mapping to which group of
people is responsible to take it to the next step:

Bug State    | Who                       | What
--------------------------------------------------------------------------------------------------
UNCORFIRMED  | users and QA              | independantly confirm the bug
NEEDINFO     | users and QA              | triage bug to reproducation scenario the dev needs
NEW          | devs                      | fix the bug
ASSIGNED     | whoever is in assigned to | fix the bug [1]
REOPENED     | devs                      | fix the bug [2]
RESOLVED     | affected users and QA     | verify fix
VERIFIED     | releng                    | announce publication of fixed release (unused in LO)[3]
CLOSED       | -                         | dead issue

On first attempt it is ok to assume that a independantly confirmed bug is
triaged and can go to NEW. However, if a dev cannot go on fixing with what is
provided, it will go back to NEEDINFO.

> My suggestion is to name it something like needsDevAdvice, remove if advice
> was given, keyword was used unnecessarily or report got closed. It should
> still be used very conservatively, of course.

Yeah maybe[4]. Or needsDevTriage. Essentially this keyword means that "dev
needs to look at this issue, even though it isnt triaged" yet, marking an
explicit exception to their general guidance that the dont need to look at
UNCORFIRMED or NEEDINFO bugs (yet).

Please consider renaming keywords for esthetic reasons reasons carefully
though: Even a ninja-edit, which not creating email, adds noise to the issues
and makes it more likely a developer will give up on an issue for having to
much noise and distraction.

> What is the problem here? Is a proposed easy hack fundamentally different from an easy hack that
> lacks necessary details? Both are for the same people: the developers who have understanding of
> the relevant pieces of code.

The difference is that a 'easy hack that lacks necessary detail' was already
considered by a dev to really be easy enough. This might not at all be true for
a 'proposed easy hack'.

> What is the workflow if I see a bug report that might be easy enough to be an easy hack, but I have
> zero knowledge about the underlying code?

In general, I dont think we lack easy hacks. The bottle neck is still mentoring
power. So as long as we dont run out of easy hack or the existing one are
stalling because nobody want to take them on, we dont have an urgent need for
new 'proposed easy hacks' anyway -- unless they come from a dev, because _that_
implies commitment to mentor the issue.

Best,

Bjoern

[1] effectively mostly irrelevant bug state, exists for historic reasons, de
    facto the same as NEW
[2] realistically, only devs that take ownership of a bug should REOPEN.
    Everyone else is much better served with filing a follow-up bug as finding a
    dev to fix a new bug cleaning stating what was missing is much easier that
    finding a dev for a bug that has a 50-comment discussion with lots of dead ends
    inbetween.
[3] And mostly redundant as the commit bot already announces the releases that
    will have a fix. Could be implemented with a bot though trivially, I guess.
[4] "There are only two hard things in Computer Science: cache invalidation,
    naming things and off-by-one errors."

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Budea, Áron Budea, Áron
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...

Hi,
 
On Tuesday, September 20, 2016 11:18 CEST, "Bjoern Michaelsen" <[hidden email]> wrote:
 
> NEW _should_ mean triaged for all matters. That it is not called the more
> descriptive TRIAGED (like it is on e.g. launchpad) is due to historic reasons.
> A bug on bugs.documentfoundation.org should be in NEW only if a developer could
> start working on it. Otherwise it should be in UNCOFIRMED or NEEDINFO (as
> should be for 101898: It was independently confirmed, but there still isnt a
> reliable reproduction scenario that allows a developer to work on it).
>
> In general, bug states should be a rather clear mapping to which group of
> people is responsible to take it to the next step:

I'd consider a bug triaged with the following steps also done, which are rarely
done when the bug is set to NEW:
-check in other OSes,
-check if bug is regression.

Yes, NEW means a developer can start working on it, but there should be a status
meaning a bug requires no further QA attention (except bibisecting), so in my
book NEW doesn't mean triaged in itself.

Another important distinction: any interested contributor having the latest
LibreOffice version can confirm/unconfirm a bug, but more experience and
preparation is required to further triage it.

The problem with independent confirmations only is that the status is hardly
different from UNCONFIRMED. Assuming QA tried and couldn't reproduce the issue,
the bug is likely hard to reproduce, and setting to NEEDINFO won't help, as the
reporter won't know what else to add, and it's a developer who can determine
what kind of further information is needed, and the steps to acquire it.
Thus bug 101898 still needs attention from a developer (if a reporter can debug
that's a nice bonus, but not a realistic expectation).

Additionally, at the time there are at least two interested people willing to
provide additional information. When a developer picks up the bug later, there
might be none.
Thus a keyword should exist to denote that a bug needs developer attention
outside the regular workflow.

Thanks,
Aron

P.S.: subject orrected


_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...

Hi,

On Wed, Sep 21, 2016 at 12:24:37AM +0200, Áron Budea wrote:
> I'd consider a bug triaged with the following steps also done, which are rarely
> done when the bug is set to NEW:
> -check in other OSes,
> -check if bug is regression.
>
> Yes, NEW means a developer can start working on it, but there should be a status
> meaning a bug requires no further QA attention (except bibisecting), so in my
> book NEW doesn't mean triaged in itself.

Bug states should support the workflow to get the bug fixed -- not the other
way around. OS crosschecking is irrelevant in more than 90% of cases for the
work of a dev to start working on it. While regression info is quite helpful,
very often it is provided already by the initial report even. Both are not
reasons for devs to punt on the issue solely because of this -- thus, because
bug states should support workflows and not the other way around, not to have
yet-another-state and just have devs assume a crossplatform non-regression by
default.

> The problem with independent confirmations only is that the status is hardly
> different from UNCONFIRMED. Assuming QA tried and couldn't reproduce the issue,
> the bug is likely hard to reproduce, and setting to NEEDINFO won't help, as the
> reporter won't know what else to add, and it's a developer who can determine
> what kind of further information is needed, and the steps to acquire it.

That might be so, but spending developer ressources on the kinds of obscure bugs
that cant be triaged by the projects QA ressources isnt a wise choice either.
If users cant properly triage a bug with the help of projects QA ressources and
there is no volunteer dev jumping on it on own initiative, there are various L3
support options welcoming to help them out.

Best,

Bjoern
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Budea, Áron Budea, Áron
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...

Hi,
 
On Wednesday, September 21, 2016 01:11 CEST, "Bjoern Michaelsen" <[hidden email]> wrote:
> Bug states should support the workflow to get the bug fixed -- not the other
> way around. OS crosschecking is irrelevant in more than 90% of cases for the
> work of a dev to start working on it. While regression info is quite helpful,
> very often it is provided already by the initial report even. Both are not
> reasons for devs to punt on the issue solely because of this -- thus, because
> bug states should support workflows and not the other way around, not to have
> yet-another-state and just have devs assume a crossplatform non-regression by
> default.

Regardless whether there is a separate status for "triaged" bugs or not, the
mentioned actions only involve QA, and no developer activity. Thanks for the
feedback, though, it's helpful to know what is worthwhile to spend time on.

My original point was that the keyword asking for developer advice should not
be restricted to UNCONFIRMED bugs if confirmed bugs don't have the
requirement of reproducibility by QA.

Nevertheless, I agree that effort is better spent elsewhere than pursuing
really obscure bugs, it seems to be something where careful prioritization is
important.
 
Cheers,
Aron


_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Cor Nouws Cor Nouws
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...

Áron Budea wrote on 21-09-16 10:25:

> On Wednesday, September 21, 2016 01:11 CEST,<[hidden email]> wrote:
>> Bug states should support the workflow to get the bug fixed -- not the other
>> way around.

And then when the bug status also support QA work, more bugs will be
well triaged and clean, and easy to pick and understand by developers ;)
So both are valid.

And what I read from Árons mail: just New is not enough for QA to see
that a bug is properly triaged.
I bet you that there are many many bugs confirmed and set to New, to
keep the number of unconfirmed low, but where no or hardly any real
triage has been done.
Nice challenge to find those and make sure the information in the
reports is used at it's best.

> Nevertheless, I agree that effort is better spent elsewhere than pursuing
> really obscure bugs, it seems to be something where careful prioritization is
> important.

Not always easy to understand what is obscure and if that makes issues
less relevant. It's always good to see when developers, for whatever
reason, pick up 'minor' issues.

Ciao - Cor



--
Cor Nouws
GPD key ID: 0xB13480A6 - 591A 30A7 36A0 CE3C 3D28  A038 E49D 7365 B134 80A6
- vrijwilliger http://nl.libreoffice.org
- volunteer http://www.libreoffice.org
- The Document Foundation Membership Committee Member
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Alex Thurgood Alex Thurgood
Reply | Threaded
Open this post in threaded view
|

Re: [Libreoffice-qa] minutes of ESC call ...

In reply to this post by Bjoern Michaelsen
Le 20/09/2016 à 11:18, Bjoern Michaelsen a écrit :

If I might interject my ha'pence-worth into the discussion :


> NEW _should_ mean triaged for all matters. That it is not called the more

This is where, in my experience with LO-QA, developer expectations and
QA/user actions do not always coincide. Some developers ask for full
backtrace with symbols before even deigning to sniff at a bug report. It
would be useful methinks to have some kind of consensus here, at least
from the developers, with a view to how that would impact the number of
bug reports that would then necessarily remain in the UNCONFIRMED
setting if we were to take such a demanding point of view.






> NEW          | devs                      | fix the bug
> ASSIGNED     | whoever is in assigned to | fix the bug [1]
> REOPENED     | devs                      | fix the bug [2]

I would take issue with the premise that only a dev can re-open the bug
report, simply because this almost never happens today.

At present, BZ allows normal bug reporters to re-open a report, which, I
would agree, probably isn't the ideal situation, however, if the
reporter is the only one to test the fix other than the developer and it
doesn't solve the issue as initially reported then what else is that
person supposed to do ? In quite a few instances, QA is simply not
around or unaware to be even able to test the fix and provide separate
confirmation/denial that the fix has solved the issue (unless they are
one and the same person). Such a narrow approach also relies on the fact
that the bug report is based on a specific code path fix, whereas the
problematic behaviour reported might actually be dependent on several
code execution sequences that form the whole behaviour.


An example (since it gives an idea of the mismatch in expectations) is
the extensions manager under OSX, where users have not been able to add
or update their extensions for quite a while now. The users expect to be
able to just update their existing extensions, irrespective of whether
it be by double-clicking on an OXT, or using the built-in dialog in the
manager. The fact that only one of these has been fixed is irrelevant to
them, in their eyes, "it still doesn't work" as all the other
possibilities are denied them. The fix is at best "partial".

From a developer point of view, the above situation involves several
code execution paths, and the solution is to address each one
independently. However, that is clearly not how a user understands the
situation. How would one then set such a bug ? Is it fixed, or
unconfirmed, or new, or re-opened or needinfo ? I can understand that
fixing the behaviour of the Add button is not the same as fixing the
behaviour of the Update button, but we need then to be able to convey
this to our users. I get the feeling that here we will end up with a
"you can't fool all of the people half of the time situation" with an
additional layer of fog thrown in to confuse the issue of communication
to the public, which can only serve to be detrimental long term.


Alex





_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Eike Rathke-2 Eike Rathke-2
Reply | Threaded
Open this post in threaded view
|

Re: [Libreoffice-qa] minutes of ESC call ...

Hi Alexander,

On Wednesday, 2016-09-21 12:22:49 +0200, Alexander Thurgood wrote:

> > NEW _should_ mean triaged for all matters. That it is not called the more
>
> This is where, in my experience with LO-QA, developer expectations and
> QA/user actions do not always coincide. Some developers ask for full
> backtrace with symbols before even deigning to sniff at a bug report.

For me it depends on whether I can reproduce a bug or not. If a crash
happens only in the scenario of the user reporting it, then not having
a backtrace a bug is quite useless for a dev. If I can reproduce a crash
then I don't need a backtrace attached to the bug.

> > NEW          | devs                      | fix the bug
> > ASSIGNED     | whoever is in assigned to | fix the bug [1]
> > REOPENED     | devs                      | fix the bug [2]
>
> I would take issue with the premise that only a dev can re-open the bug
> report, simply because this almost never happens today.

That's not what was written, re-adding part of the quote you omitted:

| bug states should be a rather clear mapping to which group of
| people is responsible to take it to the next step:

So, REOPENED -> dev means that the dev is responsible, not that only
a dev can reopen. Of course QA can also close a bug again if it was
reopened under false premises.

  Eike

--
LibreOffice Calc developer. Number formatter stricken i18n transpositionizer.
GPG key "ID" 0x65632D3A - 2265 D7F3 A7B0 95CC 3918  630B 6A6C D5B7 6563 2D3A
Better use 64-bit 0x6A6CD5B765632D3A here is why: https://evil32.com/
Care about Free Software, support the FSFE https://fsfe.org/support/?erack

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (836 bytes) Download Attachment
Bjoern Michaelsen Bjoern Michaelsen
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...

In reply to this post by Cor Nouws
Hi,

On Wed, Sep 21, 2016 at 11:33:28AM +0200, Cor Nouws wrote:
> And then when the bug status also support QA work, more bugs will be
> well triaged and clean, and easy to pick and understand by developers ;)
> So both are valid.
>
> And what I read from Árons mail: just New is not enough for QA to see
> that a bug is properly triaged.

Well, my point is if we make that an explicit requirement (e.g. "properly
triaged bugs need to be checked for if they are crossplatform or appear on just
one") the net effects will be that devs dont yet look at the issue as it is not
properly triaged. However, platformdependant bugs are common enough that we can
optimistically assume a bug to be crossplatform by default without too much
hurt. If that assumption fails (rarely), a sane dev will notice they cant
reproduce, mention in a comment, move it to NEEDINFO asking to be better
triaged and move on.

> I bet you that there are many many bugs confirmed and set to New, to
> keep the number of unconfirmed low, but where no or hardly any real
> triage has been done.

As long as devs are not complaining, I think this is ok. FWIW, if I run in a
interesting undertriaged bug, I:
- give it a try with whatever info is there
- triage what I need in addition, but only when I feel like it
- if Im not interested the in the needed additional info, I explicitly state
  what I miss in a comment, put it back to NEEDINFO and move on.

> Nice challenge to find those and make sure the information in the
> reports is used at it's best.

Not all triage info is needed for devs in all cases. Good triage improves
chances of a bug being picked up by a dev in general, but that is not a binary
"triaged"/"not triaged" thing. Making every bug perfectly triaged on every
detail before a dev sees it will waste resources, while just giving devs the
freedom to push issues back quickly to NEEDINFO with specific details on what
exactly they would like to know beyond basic triage makes for a much more
targeted triage.

Of course, to have a specific issues that somebody cares about being fixed,
providing as much triage upfront before even explicitly requested by a dev is
still an excellent strategy[1]. This is because I (and I assume others too)
look at bibisected bugs a lot more often and longer than at other regressions,
and at regressions a lot longer than at other confirmed bugs, etc.

Best,

Bjoern

[1] Still no guarantees on being solved by volunteers obviously.
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Michael Stahl-2 Michael Stahl-2
Reply | Threaded
Open this post in threaded view
|

Re: [Libreoffice-qa] minutes of ESC call ...

In reply to this post by Eike Rathke-2
On 21.09.2016 17:28, Eike Rathke wrote:

> Hi Alexander,
>
> On Wednesday, 2016-09-21 12:22:49 +0200, Alexander Thurgood wrote:
>>> NEW          | devs                      | fix the bug
>>> ASSIGNED     | whoever is in assigned to | fix the bug [1]
>>> REOPENED     | devs                      | fix the bug [2]
>>
>> I would take issue with the premise that only a dev can re-open the bug
>> report, simply because this almost never happens today.
>
> That's not what was written, re-adding part of the quote you omitted:
>
> | bug states should be a rather clear mapping to which group of
> | people is responsible to take it to the next step:
>
> So, REOPENED -> dev means that the dev is responsible, not that only
> a dev can reopen. Of course QA can also close a bug again if it was
> reopened under false premises.

actually i haven't felt responsible for REOPENED bugs so far, because my
experience is that they are usually not re-opened by QA folks but by
random users who usually either don't understand the workflow (the
current release is still buggy! well the fix just hit master today...),
or actually have a completely different bug that matches the overly
generic title of the bugzilla issue ("DOC file looks bad after import" -
wow i've got the same problem!).

so i'd welcome it if QA folks look at REOPENED bugs and move them to NEW
if appropriate :)



_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Cor Nouws Cor Nouws
Reply | Threaded
Open this post in threaded view
|

Re: minutes of ESC call ...

In reply to this post by Bjoern Michaelsen
Hi Bjoern,

Bjoern Michaelsen wrote on 21-09-16 18:45:

> Well, my point is if we make that an explicit requirement (e.g. "properly
> triaged bugs need to be checked for if they are crossplatform or appear on just
> one") the net effects will be that devs dont yet look at the issue as it is not
> properly triaged.

Sure. Now I do not support the idea that 'being proper triaged' is a
prerequisite before developers will/may look at reports.
Apart from that, the repeated example of crossplatform doesn't seem the
most exited example of proper triage to me.
Details of the reason of the bug, clear summary (finding, recognizing),
component, duplicates (may ~indicate importance) regressions (user
expectations, statistics), bibisection come to my mind much earlier.

> [...]  Making every bug perfectly triaged on every
> detail before a dev sees it will waste resources, while just giving devs the
> freedom to push issues back quickly to NEEDINFO with specific details on what
> exactly they would like to know beyond basic triage makes for a much more
> targeted triage.

Sure, my point is that triage is useful for QA and devs. And indeed in a
situation that looks as if there's not really enough hands to confirm
all reports, let alone do proper triage where it would be good, let
alone to do useful clean ups and in depth investigations on certain
areas.., spending too many words on what/how the prefect route &
handling is, seems a bit superfluous ;)
So basically I don't see where we disagree :)

Ciao - Cor


--
Cor Nouws
GPD key ID: 0xB13480A6 - 591A 30A7 36A0 CE3C 3D28  A038 E49D 7365 B134 80A6
- vrijwilliger http://nl.libreoffice.org
- volunteer http://www.libreoffice.org
- The Document Foundation Membership Committee Member
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice