PyUNO usability improvements

classic Classic list List threaded Threaded
13 messages Options
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

PyUNO usability improvements

Hi,

For the past few weeks I've been working on some upgrades to PyUNO,
which have now reached (what I hope is) a sufficient level of sanity and
stability. I've pushed the changes to:

   https://gerrit.libreoffice.org/16272

Given that this is a fairly large API change (if for a scripting
language that is not -yet- heavily used by our users), it probably
needs a careful review, but I have a suspicion that probably no one
developer currently has knowledge of all the interlocking requirements
at play here - or PyUNO might look somewhat different to begin with ;)
If anyone feels kind enough to take on the work of reviewing this,
please give me a shout and I'll offer what help I can on understanding
how all the pieces fit together.

* Parts of the test suite depend on
   https://gerrit.libreoffice.org/16073
Without this fix, various of the tests will fail


Regards
Matthew Francis


------------------------------------------------------------------------

     Make PyUNO provide more Pythonic behaviour

     - Simplifies working with UNO objects by giving the behaviour of
     Python lists, dicts and iterators to objects which implement UNO
     container interfaces

     - Applies a custom behaviour to allow objects which implement
     com::sun::star::table::XCellRange to yield cells and cell ranges by
     subscript

     - When UNO container objects are addressed in the new style,
     eliminates the requirement to manually construct Any objects for
     contained elements which are typed sequences

     - Allows lists and iterators to be passed wherever a UNO method
     accepts a sequence

     - Relaxes the requirements for initialising UNO structs to allow
     some members to be skipped when all initialisers are passed by name

     1. Collection interfaces
     ========================

     Objects which implement core UNO collection interfaces are made to
     behave in a way that is more natural for Python code.

     com::sun::star::container::XIndexAccess
     com::sun::star::container::XIndexReplace
     com::sun::star::container::XIndexContainer
     - Objects provide Python list access semantics
         num = len(obj)              # Number of elements
         val = obj[0]                # Access by index
         val1,val2 = obj[2:4]        # Access by slice
         val1,val2 = obj[0:3:2]      # Access by extended slice
         if val in obj: ...          # Test value presence
         for val in obj: ...         # Implicit iterator (values)
         itr = iter(obj)             # Named iterator (values)
         obj[0] = val                # Replace by index
         obj[2:4] = val1,val2        # Replace by slice
         obj[0:3:2] = val1,val2      # Replace by extended slice
         obj[2:3] = val1,val2        # Insert/replace by slice
         obj[2:2] = (val,)           # Insert by slice
         obj[2:4] = (val,)           # Replace/delete by slice
         obj[2:3] = ()               # Delete by slice (implicit)
         del obj[0]                  # Delete by index
         del obj[2:4]                # Delete by slice

     com::sun::star::container::XNameAccess
     com::sun::star::container::XNameReplace
     com::sun::star::container::XNameContainer
     - Objects provide Python dict access semantics
         num = len(obj)              # Number of keys
         val = obj[key]              # Access by key
         if key in obj: ...          # Test key presence
         for key in obj: ...         # Implicit iterator (keys)
         itr = iter(obj)             # Named iterator (keys)
         obj[key] = val              # Replace by key
         obj[key] = val              # Insert by key
         del obj[key]                # Delete by key

     com::sun::star::container::XEnumerationAccess
     - Objects provide Python iterable semantics
         for val in obj: ...         # Implicit iterator
         itr = iter(obj)             # Named iterator

     com::sun::star::container::XEnumeration
     - Objects provide Python iterator semantics
         for val in itr: ...         # Iteration of named iterator
         if val in itr: ...          # Test value presence

     Objects which implement both XIndex* and XName* are supported, and
     respond to both integer and string keys. However, iterating over
     such an object will return the keys (like a Python dict) rather than
     the values (like a Python list).

     2. Cell ranges
     ==============

     A custom behaviour is applied to objects which implement
     com::sun::star::table::XCellRange to allow their cells and cell
     ranges to be addressed by subscript, in the style of a Python list
     or dict (read-only). This is applicable to Calc spreadsheet sheets,
     Writer text tables and cell ranges created upon these.
         cell = cellrange[0,0]       # Access cell by indices
         rng = cellrange[0,1:2]      # Access cell range by index,slice
         rng = cellrange[1:2,0]      # Access cell range by slice,index
         rng = cellrange[0:1,2:3]    # Access cell range by slices
         rng = cellrange['A1:B2']    # Access cell range by descriptor
         rng = cellrange['Name']     # Access cell range by name

     Note that the indices used are in Python/C order, and differ from
     the arguments to methods provided by XCellRange.
     - The statement cellrange[r,c], which returns the cell from row r
     and column c, is equivalent to calling
         XCellRange::getCellByPosition(c,r)
     - The statement cellrange[t:b,l:r], which returns a cell range
     covering rows t to b(non-inclusive) and columns l to r(non-
     inclusive), is equivalent to calling
         XCellRange::getCellRangeByPosition(l,t,r-1,b-1).

     In contrast to the handling of objects implementing XIndex*,
     extended slice syntax is not supported. Negative indices (from-end
     addresses) are supported only for objects which also implement
     com::sun::star::table::XColumnRowRange (currently Calc spreadsheet
     sheets and cell ranges created upon these). For such objects, the
     following syntax is also available:
         rng = cellrange[0]          # Access cell range by row index
         rng = cellrange[0,:]        # Access cell range by row index
         rng = cellrange[:,0]        # Access cell range by column index

     3. Elimination of explicit Any
     ==============================

     PyUNO has not previously been able to cope with certain method
     arguments which are typed as Any but require a sequence of specific
     type to be passed. This is a particular issue for container
     interfaces such as XIndexContainer and XNameContainer.

     The existing solution to dealing with such methods is to use a
     special method to pass an explicitly typed Any, giving code such as:

         index = doc.createInstance("com.sun.star.text.ContentIndex");
         ...
         uno.invoke( index.LevelParagraphStyles , "replaceByIndex",
                     (0, uno.Any("[]string", ('Caption',))) )

     The new Pythonic container access is able to correctly infer the
     expected type of the sequences required by these arguments. In the
     new style, the above call to .replaceByIndex() can instead be
     written:

         index.LevelParagraphStyles[0] = ('Caption',)

     4. List and iterator arguments
     ==============================

     Wherever a UNO API expects a sequence, a Python list or iterator can
     now be passed. This enables the use of list comprehensions and
     generator expressions for method calls and property assignments.

     Example:

         tbl = doc.createInstance('com.sun.star.text.TextTable')
         tbl.initialize(10,10)
         # ... insert table ...
         # Assign numbers 0..99 to the cells using a generator expression
         tbl.Data = ((y for y in range(10*x,10*x + 10)) for x in range(10))

     5. Tolerant struct initialisation
     =================================

     Previously, a UNO struct could be created fully uninitialised, or by
     passing a combination of positional and/or named arguments to its
     constructor. However, if any arguments were passed, all members were
     required to be initialised or an exception was thrown.
     This requirement is relaxed such that when all arguments passed to a
     struct constructor are by name, some may be omitted. The existing
     requirement that all members must be explicitly initialised when
     some constructor arguments are unnamed (positional) is not affected.

     Example:

         from com.sun.star.beans import PropertyValue
         prop = PropertyValue(Name='foo', Value='bar')



_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Riccardo Magliocchetti Riccardo Magliocchetti
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

Hello Matthew,

Il 14/06/2015 15:26, Matthew J. Francis ha scritto:
> Hi,
>
> For the past few weeks I've been working on some upgrades to PyUNO,
> which have now reached (what I hope is) a sufficient level of sanity and
> stability. I've pushed the changes to:
>
>    https://gerrit.libreoffice.org/16272

Cool stuff!

> Given that this is a fairly large API change (if for a scripting
> language that is not -yet- heavily used by our users), it probably

Have you tried this against unoconv?
https://github.com/dagwieers/unoconv

> needs a careful review, but I have a suspicion that probably no one
> developer currently has knowledge of all the interlocking requirements
> at play here - or PyUNO might look somewhat different to begin with ;)
> If anyone feels kind enough to take on the work of reviewing this,
> please give me a shout and I'll offer what help I can on understanding
> how all the pieces fit together.

Speaking of pythonic api there was this nice talk which i think fits quite well:
https://www.youtube.com/watch?v=wf-BqAjZb8M

TLDR; what about adding a pythonic wrapper on top of the actual API to make it
pythonic?

thanks a lot for your work

--
Riccardo Magliocchetti
@rmistaken
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

Hi,

On 15/06/2015 16:33, Riccardo Magliocchetti wrote:
> Have you tried this against unoconv?
> https://github.com/dagwieers/unoconv

I have now, although you have to do a bit of a funny dance to get it to work
correctly against a random LO instance.
Specifically, against a current build I had to:
     UNO_PATH=$PWD/instdir PYTHONPATH=$PWD/instdir/program unoconv <args>

It seems to work correctly, and in general I would expect this to be the case.
You *could* deliberately write code that was forward-incompatible with the new
syntax options, but you'd have to try something quite silly to do so.

For instance, deliberately relying on the fact that UNO collection objects
throw exceptions when you try to access them Python style would be incompatible.

> Speaking of pythonic api there was this nice talk which i think fits quite well:
> https://www.youtube.com/watch?v=wf-BqAjZb8M
>
> TLDR; what about adding a pythonic wrapper on top of the actual API to make it
> pythonic?

Thanks for the link, that was an interesting talk - and suggested one more
feature that could be implemented cheaply, which is to implement a context
manager for XModel to manage controller locking.

So instead of

     doc.lockControllers()
     try:
         ... # operate on the document
     finally:
         doc.unlockControllers()

You could simply

     with doc:
         ... # operate on the document

(Can anyone suggest any more interfaces which could benefit from this treatment?)


As for the other elements suggested in the talk, UNO properties are already
mapped to Python properties, which is one of the areas where the existing code
is quite successful in providing a Pythonic interface; and providing proper
list, iterator and dict accessors is of course the main thrust of the patch.
The UNO API is huge, and whereas trying to implement a perfect Python
interface for each UNO object and interface individually would be an enormous
amount of work, doing this kind of global transform on the whole API should be
highly effective in achieving (very nearly) the same result.

Regards
Matthew Francis



_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Noel Grandin-2 Noel Grandin-2
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements



On 2015-06-15 02:46 PM, Matthew J. Francis wrote:

> Thanks for the link, that was an interesting talk - and suggested one more feature that could be implemented cheaply,
> which is to implement a context manager for XModel to manage controller locking.
>
> So instead of
>
>      doc.lockControllers()
>      try:
>          ... # operate on the document
>      finally:
>          doc.unlockControllers()
>
> You could simply
>
>      with doc:
>          ... # operate on the document
>
> (Can anyone suggest any more interfaces which could benefit from this treatment?)
>


Try this:
    find . -name *.idl | xargs grep 'close()'
    find . -name *.idl | xargs grep 'unlock'
should find you a bunch like

com::sun::star::connection::XConnection
com::sun::star::connection::XBroadcaster
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

On 15/06/2015 20:53, Noel Grandin wrote:

> Try this:
>     find . -name *.idl | xargs grep 'close()'
>     find . -name *.idl | xargs grep 'unlock'
> should find you a bunch like
>
> com::sun::star::connection::XConnection
> com::sun::star::connection::XBroadcaster

So, having searched a bit further along these lines,

com.sun.star.connection.XConnection
- Could technically be done, but I can't see any use of it outside the
bridge code (where block level context management would be fairly pointless)

com.sun.star.connection.XBroadcaster
- Referenced by svx/source/table/tablemodel.cxx, but I see precisely
zero mention of it being used in user code. No UNO service or interface
actually declares its use. Unless anyone can come up with an actual use
case, I'd skip it.


The more likely candidates seem to be:

com.sun.star.sdbc.XCloseable
- This one is on pretty solid ground and covers all sorts of things like
database statements and result sets which could usefully be context
managed at the block level

com.sun.star.frame.XModel
- As previously mentioned, locking/unlocking controllers is a pretty
common thing to do and would benefit from the sugar. Riccardo
Magliocchetti separately pointed out on IRC an example where he used a
wrapper class context manager to control the closing of documents, which
would correspond instead to a controller on com.sun.star.util.XCloseable
- which is also implemented by document components. These two behaviours
wouldn't happily coexist on the same object, and I think in terms of
macro programming at least, the controller locking is more useful. As
(com.sun.star.util.)XCloseable.close() also takes an argument, which
couldn't be supplied using the "with obj: ..." syntax, I think this use
case would be better dealt with by a function or class decorator if it's
generally useful, rather than in the PyUNO core at the interface level.

com.sun.star.frame.XLayoutManager
- Could be used to lock interface updates while changing UI. I see one
or two examples of user code using this.

com.sun.star.drawing.framework.XConfigurationController
- Possible but unsure how useful. I see one example of a user having
(unsuccessfully) attempted to use this. It couldn't in any case be
implemented without a working use case to test.

com.sun.star.registry.XRegistryKey
com.sun.star.registry.XSimpleRegistry
- Could be occasionally useful


Others:

com.sun.star.document.XUndoManager
- Potentially quite useful, but as with com.sun.star.util.XCloseable it
requires an argument on the setup side, and would be better served by a
decorator style controller


Regards
Matthew Francis



_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

On 06/16/2015 04:27 AM, Matthew J. Francis wrote:
> com.sun.star.registry.XRegistryKey
> com.sun.star.registry.XSimpleRegistry
> - Could be occasionally useful

Entities in css.registry were tailor-made to match the legacy UNO types
and services rdb format, a use which is rather deprecated (the legacy
format only being used for extensions by now, mainly for
backwards-compatibility reasons).  And I doubt these entities will ever
be re-used for something else.
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Michael Stahl-2 Michael Stahl-2
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

In reply to this post by Matthew Francis
On 14.06.2015 15:26, Matthew J. Francis wrote:
> Hi,
>
> For the past few weeks I've been working on some upgrades to PyUNO,
> which have now reached (what I hope is) a sufficient level of sanity and
> stability. I've pushed the changes to:

wow, this all looks really nice!

>      2. Cell ranges
>      ==============
>
>      A custom behaviour is applied to objects which implement
>      com::sun::star::table::XCellRange to allow their cells and cell
>      ranges to be addressed by subscript, in the style of a Python list
>      or dict (read-only). This is applicable to Calc spreadsheet sheets,
>      Writer text tables and cell ranges created upon these.
>          cell = cellrange[0,0]       # Access cell by indices
>          rng = cellrange[0,1:2]      # Access cell range by index,slice
>          rng = cellrange[1:2,0]      # Access cell range by slice,index
>          rng = cellrange[0:1,2:3]    # Access cell range by slices
>          rng = cellrange['A1:B2']    # Access cell range by descriptor
>          rng = cellrange['Name']     # Access cell range by name
>
>      Note that the indices used are in Python/C order, and differ from
>      the arguments to methods provided by XCellRange.
>      - The statement cellrange[r,c], which returns the cell from row r
>      and column c, is equivalent to calling
>          XCellRange::getCellByPosition(c,r)
>      - The statement cellrange[t:b,l:r], which returns a cell range
>      covering rows t to b(non-inclusive) and columns l to r(non-
>      inclusive), is equivalent to calling
>          XCellRange::getCellRangeByPosition(l,t,r-1,b-1).

i'm not sure if using a different order for row and column here is a
good idea or a bad idea.  maybe some Calc developer has an opinion on it?


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Eike Rathke-2 Eike Rathke-2
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

Hi,

On Tuesday, 2015-06-16 12:34:15 +0200, Michael Stahl wrote:

> >      2. Cell ranges
> >      ==============
> >
> >          cell = cellrange[0,0]       # Access cell by indices
> >          rng = cellrange[0,1:2]      # Access cell range by index,slice
> >          rng = cellrange[1:2,0]      # Access cell range by slice,index
> >          rng = cellrange[0:1,2:3]    # Access cell range by slices
> >          rng = cellrange['A1:B2']    # Access cell range by descriptor
> >          rng = cellrange['Name']     # Access cell range by name
> >
> >      Note that the indices used are in Python/C order, and differ from
> >      the arguments to methods provided by XCellRange.
> >      - The statement cellrange[r,c], which returns the cell from row r
> >      and column c, is equivalent to calling
> >          XCellRange::getCellByPosition(c,r)
Is there a specific reason for this? Why not keep the order the API
uses? My concern is, that if these get mixed the user will get
confused..

> >      - The statement cellrange[t:b,l:r], which returns a cell range
> >      covering rows t to b(non-inclusive) and columns l to r(non-
> >      inclusive), is equivalent to calling
> >          XCellRange::getCellRangeByPosition(l,t,r-1,b-1).

Which when keeping c,r order would be cellrange[l:r,t:b], slightly
nearer to the API.

  Eike

--
LibreOffice Calc developer. Number formatter stricken i18n transpositionizer.
GPG key "ID" 0x65632D3A - 2265 D7F3 A7B0 95CC 3918  630B 6A6C D5B7 6563 2D3A
Better use 64-bit 0x6A6CD5B765632D3A here is why: https://evil32.com/
Care about Free Software, support the FSFE https://fsfe.org/support/?erack

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice

attachment0 (836 bytes) Download Attachment
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

On 16/06/2015 21:45, Eike Rathke wrote:

> Is there a specific reason for this? Why not keep the order the API
> uses? My concern is, that if these get mixed the user will get
> confused..

Doing it the other way round would give a different obvious violation
of the principle of least confusion. Taking one of my original examples,

 > tbl.Data = ((y for y in range(10*x,10*x + 10)) for x in range(10))

 > tbl.Data
((0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0),
  (10.0, 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0),
  (20.0, 21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0),
  (30.0, 31.0, 32.0, 33.0, 34.0, 35.0, 36.0, 37.0, 38.0, 39.0),
  (40.0, 41.0, 42.0, 43.0, 44.0, 45.0, 46.0, 47.0, 48.0, 49.0),
  (50.0, 51.0, 52.0, 53.0, 54.0, 55.0, 56.0, 57.0, 58.0, 59.0),
  (60.0, 61.0, 62.0, 63.0, 64.0, 65.0, 66.0, 67.0, 68.0, 69.0),
  (70.0, 71.0, 72.0, 73.0, 74.0, 75.0, 76.0, 77.0, 78.0, 79.0),
  (80.0, 81.0, 82.0, 83.0, 84.0, 85.0, 86.0, 87.0, 88.0, 89.0),
  (90.0, 91.0, 92.0, 93.0, 94.0, 95.0, 96.0, 97.0, 98.0, 99.0))

 > tbl.Data[9][0]
90.0

 > tbl[9,0].Value
90.0


If the XCellRange specialisation used c,r order, the last statement
would give 9, not 90

(The .Data comes from the XChartDataArray interface; there is also e.g.
.DataArray from XCellRangeData)

Regards
Matthew Francis
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

So far I've had some valuable feedback - thanks to those who
contributed.

I also have the nagging feeling that any changes to behaviour will be
with us for a long time, and there are still a few questions in my mind
over the advisability of some of the details. Part of the reason for
this is that while it all seems a good idea in theory, there aren't any
actual demonstration use cases yet apart from the few unit tests.

So, given that:
- To begin with there's no way I'd recommend rushing this in for 5.0,
and nobody but me is clamouring to actually use the changes
- Some time to mature the interface changes seems like a good idea
- I have a significant actual use case (UI testing) separately in
progress that this could be developed alongside

What about converting this to a feature branch and running with it for
a while, with the aim of landing a properly mature feature for 5.1?


Regards
Matthew Francis


_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Samuel Mehrbrodt-2 Samuel Mehrbrodt-2
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

Hi,

I also have the nagging feeling that any changes to behaviour will be
with us for a long time, and there are still a few questions in my mind
over the advisability of some of the details. Part of the reason for
this is that while it all seems a good idea in theory, there aren't any
actual demonstration use cases yet apart from the few unit tests.

I don't know if this is what you are looking for, but here is a python app using the uno bridge.

Samuel

_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

In reply to this post by Matthew Francis
On 06/18/2015 11:34 AM, Matthew J. Francis wrote:
> What about converting this to a feature branch and running with it for
> a while, with the aim of landing a properly mature feature for 5.1?

Avoid feature branches at all cost.  If you are reasonably confident
about a change, push it to master.  If it later turns out to be a bad
idea, revert.

Feature branches are virtually non-existent in peoples' minds (except
the author's, maybe).  And an unpleasant surprise for all when they do
hit master.
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice
Matthew Francis Matthew Francis
Reply | Threaded
Open this post in threaded view
|

Re: PyUNO usability improvements

On 18/06/2015 20:27, Stephan Bergmann wrote:
> On 06/18/2015 11:34 AM, Matthew J. Francis wrote:
>> What about converting this to a feature branch and running with it for
>> a while, with the aim of landing a properly mature feature for 5.1?
>
> Avoid feature branches at all cost.  If you are reasonably confident
> about a change, push it to master.  If it later turns out to be a bad
> idea, revert.

OK. That being the case, I will go ahead with what I'm confident of and
keep working on the rest.

Regards
Matthew Francis
_______________________________________________
LibreOffice mailing list
[hidden email]
http://lists.freedesktop.org/mailman/listinfo/libreoffice