Callback deadlock

classic Classic list List threaded Threaded
10 messages Options
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

Callback deadlock

Hi,

I'm intermittently experiencing the attached deadlock in some Python code.

As far as I can tell, the deadlock appears to be between a call to an
XTopWindowListener in thread 1 (the listener, also in Python code,
doesn't yet do anything more than print "I am here" on a callback) and
the main thread which is attempting to enumerate the paragraphs of a
text document in thread 2.

Could anyone with more experience in UNO and locking matters help me to
understand what's going on here? If what I'm trying to do is essentially
not going to work, there is the alternative of busy-waiting for windows
to appear, but I'd like to avoid that if possible.


Regards
Matthew Francis
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Miklos Vajna-4 Miklos Vajna-4
Reply | Threaded
Open this post in threaded view
|

Re: Callback deadlock

Hi,

On Mon, Jun 29, 2015 at 01:58:09PM +0800, "Matthew J. Francis" <[hidden email]> wrote:
> I'm intermittently experiencing the attached deadlock in some Python code.

You forgot to attach it. :-)

Regards,

Miklos

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (188 bytes) Download Attachment
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

Re: Callback deadlock

On 29/06/2015 15:08, Miklos Vajna wrote:
> Hi,
>
> On Mon, Jun 29, 2015 at 01:58:09PM +0800, "Matthew J. Francis" <[hidden email]> wrote:
>> I'm intermittently experiencing the attached deadlock in some Python code.
>
> You forgot to attach it. :-)

I attached it, but compressed for size - if it didn't arrive, some
filter must have kindly eaten the attachment while leaving the message

Maybe like this then - http://pastebin.com/52FkaF2w

Regards
Matthew Francis
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: Callback deadlock

In reply to this post by Matthew Francis
On 06/29/2015 07:58 AM, Matthew J. Francis wrote:

> I'm intermittently experiencing the attached deadlock in some Python code.
>
> As far as I can tell, the deadlock appears to be between a call to an
> XTopWindowListener in thread 1 (the listener, also in Python code,
> doesn't yet do anything more than print "I am here" on a callback) and
> the main thread which is attempting to enumerate the paragraphs of a
> text document in thread 2.
>
> Could anyone with more experience in UNO and locking matters help me to
> understand what's going on here? If what I'm trying to do is essentially
> not going to work, there is the alternative of busy-waiting for windows
> to appear, but I'd like to avoid that if possible.

In <http://pastebin.com/52FkaF2w>, the main thread 1 issued an URP
request of css.awt.XTopWindowListener.windowActivated (and blocks
waiting on the reply), while thread 2 serves an incoming URP request on
SwXText::getPropertySetInfo (which happens to block waiting on the
SolarMutex).

No other thread 3--9 looks like it has the SolarMutex locked, so it must
be thread 1 that has it locked while making the windowActivated call,
which would be a programming error somewhere in the call chain leading
up to there.
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: Callback deadlock

On 06/29/2015 09:25 AM, Stephan Bergmann wrote:

> On 06/29/2015 07:58 AM, Matthew J. Francis wrote:
>> I'm intermittently experiencing the attached deadlock in some Python
>> code.
>>
>> As far as I can tell, the deadlock appears to be between a call to an
>> XTopWindowListener in thread 1 (the listener, also in Python code,
>> doesn't yet do anything more than print "I am here" on a callback) and
>> the main thread which is attempting to enumerate the paragraphs of a
>> text document in thread 2.
>>
>> Could anyone with more experience in UNO and locking matters help me to
>> understand what's going on here? If what I'm trying to do is essentially
>> not going to work, there is the alternative of busy-waiting for windows
>> to appear, but I'd like to avoid that if possible.
>
> In <http://pastebin.com/52FkaF2w>, the main thread 1 issued an URP
> request of css.awt.XTopWindowListener.windowActivated (and blocks
> waiting on the reply), while thread 2 serves an incoming URP request on
> SwXText::getPropertySetInfo (which happens to block waiting on the
> SolarMutex).
>
> No other thread 3--9 looks like it has the SolarMutex locked, so it must
> be thread 1 that has it locked while making the windowActivated call,
> which would be a programming error somewhere in the call chain leading
> up to there.

(But the remote Python end is presumably also doing things wrong.  It
smells like processing of the incoming windowActivated request is
blocked by the outgoing getPropertySetInfo request waiting for its reply.)
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
pietervo pietervo
Reply | Threaded
Open this post in threaded view
|

Re: Callback deadlock

In reply to this post by Matthew Francis
Matthew J. Francis wrote:

 > Hi,
 >
 > I'm intermittently experiencing the attached deadlock in some Python code.
 >
 > As far as I can tell, the deadlock appears to be between a call to an
 > XTopWindowListener in thread 1 (the listener, also in Python code,
 > doesn't yet do anything more than print "I am here" on a callback) and
 > the main thread which is attempting to enumerate the paragraphs of a
 > text document in thread 2.
 >
 > Could anyone with more experience in UNO and locking matters help me to
 > understand what's going on here? If what I'm trying to do is essentially
 > not going to work, there is the alternative of busy-waiting for windows
 > to appear, but I'd like to avoid that if possible.

It is not completely clear if you are calling from a remote Python program, or from inside LO.
Assuming that it is remote, yes, there are problems with listeners in this situation. It can easily deadlock. I have posted a similar situation on this list some months ago, where I did a paragraph enumeration in a listener callback and it also deadlocked. See lists.freedesktop.org/archives/libreoffice/2015-April/067601.html

Basically the conclusion was the LO does too much locking here, but that it could not be avoided because of severe problems if it wouldn't lock.
--
Piet van Oostrum <[hidden email]>
WWW: http://pietvanoostrum.com/
PGP key: [8DAE142BE17999C4]

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

Re: Callback deadlock

On 29/06/2015 15:47, Piet van Oostrum wrote:
> It is not completely clear if you are calling from a remote Python program, or from inside LO.
> Assuming that it is remote, yes, there are problems with listeners in this situation. It can easily deadlock. I have posted a similar situation on this list some months ago, where I did a paragraph enumeration in a listener callback and it also deadlocked. See lists.freedesktop.org/archives/libreoffice/2015-April/067601.html
>
> Basically the conclusion was the LO does too much locking here, but that it could not be avoided because of severe problems if it wouldn't lock.

It's remote - a test running as an external process.

The Python side of the deadlock is http://pastebin.com/qPkyVTkk
(this is a different run - the corresponding soffice.bin trace is
http://pastebin.com/g4F8zcUi )

The two things that seem to be happening are:

In thread 1, the main Python thread has got the Text property for the
text document being worked on, and PyUNO_new_UNCHECKED() is called to
make a Python object for the result, which calls
createInstanceWithArguments()

Meanwhile, in thread 2, the windowActivated() callback bas been
received, and pyuno::PyUNO_new_UNCHECKED() is called to make a Python
object for the argument, which calls createInstanceWithArguments()


If this sequence doesn't work, I'm not sure what if anything the Python
side can do better here.

Regards
Matthew Francis
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: Callback deadlock

On 06/29/2015 10:53 AM, Matthew J. Francis wrote:

> It's remote - a test running as an external process.
>
> The Python side of the deadlock is http://pastebin.com/qPkyVTkk
> (this is a different run - the corresponding soffice.bin trace is
> http://pastebin.com/g4F8zcUi )
>
> The two things that seem to be happening are:
>
> In thread 1, the main Python thread has got the Text property for the
> text document being worked on, and PyUNO_new_UNCHECKED() is called to
> make a Python object for the result, which calls
> createInstanceWithArguments()
>
> Meanwhile, in thread 2, the windowActivated() callback bas been
> received, and pyuno::PyUNO_new_UNCHECKED() is called to make a Python
> object for the argument, which calls createInstanceWithArguments()
>
>
> If this sequence doesn't work, I'm not sure what if anything the Python
> side can do better here.

ah, that's a poorly implemented Implementation::inspect
(stoc/source/inspect/introspection.cxx), which locks some m_aMutex first
thing and for the duration of the whole function call, and both
Pyhton-side threads 1 and 2 are in this function...
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

Re: Callback deadlock

On 29/06/2015 20:18, Stephan Bergmann wrote:

> ah, that's a poorly implemented Implementation::inspect
> (stoc/source/inspect/introspection.cxx), which locks some m_aMutex first
> thing and for the duration of the whole function call, and both
> Pyhton-side threads 1 and 2 are in this function...

When it comes to thousand line functions which feel the need to be
(heavily) commented in German, I have a very strong urge to 'nicht
gefingerpoken'.

Unless you feel generous enough to debug further, I'll go with plan B
and just poll for now rather than risk changing it. While not optimally
pretty, the objective of the exercise is after all to produce more tests
and fewer bugs!

(In due course I can commit a test for this bug too)

Regards
Matthew Francis

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: Callback deadlock

On 06/29/2015 03:43 PM, Matthew J. Francis wrote:

> On 29/06/2015 20:18, Stephan Bergmann wrote:
>> ah, that's a poorly implemented Implementation::inspect
>> (stoc/source/inspect/introspection.cxx), which locks some m_aMutex first
>> thing and for the duration of the whole function call, and both
>> Pyhton-side threads 1 and 2 are in this function...
>
> When it comes to thousand line functions which feel the need to be
> (heavily) commented in German, I have a very strong urge to 'nicht
> gefingerpoken'.
>
> Unless you feel generous enough to debug further, I'll go with plan B
> and just poll for now rather than risk changing it. While not optimally
> pretty, the objective of the exercise is after all to produce more tests
> and fewer bugs!
>
> (In due course I can commit a test for this bug too)

<https://bugs.documentfoundation.org/show_bug.cgi?id=92440> "Deadlock in
callback to Python," for the record

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice