About putting back Firebird experimental

classic Classic list List threaded Threaded
12 messages Options
julien2412 julien2412
Reply | Threaded
Open this post in threaded view
|

About putting back Firebird experimental

Hello,

Considering the number of bugs about Firebird migration (see https://bugs.documentfoundation.org/show_bug.cgi?id=116968) and the fact that these bugs may not be trivial to fix (see https://wastack.wordpress.com/2018/07/25/database-migration-in-libreoffice-bug-fixes-and-more/), thought it could be relevant to put Firebird back to experimental or at least stop to propose Firebird migration when opening an odb file with hsqldb embedded.

Remark: don't misunderstand me, I'd enjoy to replace HSQLDB to help to remove Java dependency. Also, I know that HSQLDB upgrade has been delayed/cancelled because we had plan to replace it by Firebird but let's face it, I don't think we're ready to propose Firebird widely.
Let's not forget there are too few dev experts working on Base part (Lionel + Tamás + Jean-Pierre Ledure for specific Access part).
(I'm trying sometimes to work on Base part but most of bugs are too difficult for me and require a good understanding of Base mechanisms).

Now perhaps I'm too pessimistic here, so would like your opinion.

Any thoughts?

Regards,

Julien
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Alex Thurgood Alex Thurgood
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

Le 20/08/2019 à 10:28, Julien a écrit :

Hi Julien,

+1 from me.

As a QA triager, I can but agree with Julien given the current stage of
development. The migration code still fails to handle multiple, basic
elements of embedded hsqldb ODB files found in actual use. If it can't
even do the basics of migration properly, what hope do we have of
convincing users to switch ?

Forcing users to be the unwitting testers (as is currently perceived to
be the case - indeed, I saw such a recrimination in a migration bug
report I dealt with today) of such a system is bound to provoke at best
apathy or resignation, at worst anger, frustration and searching for
another tool that won't trash your data.


Clearly, the work required for the tender by TDF was underestimated. I
see this currently as a kind of unwanted offspring, neither here, nor
there, and certainly not fit for purpose, despite all of the very good
work put in.

Anyway, my 2c as usual.

Alex
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Robert Großkopf Robert Großkopf
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

In reply to this post by julien2412
Hi Julien,

migration wizard is a good help for people, who know how to fix the
other issues, which would appear. I have migrated some databeses from
the Base-Handbook to Firebird.

I know I first have to save the old file. So I open the fiel in 6.3.0.4
with a backup and let the migration wizard work. Save the file and edit
the content-xml and remove the entry, which makes it unpossible to work
with subforms. After this I close 6.3 and open 6.2.6.2 for moving all
the parts from HSQLDB to Firebird, which couldn't be moved correctly
(like views with unknown functions, like not correctly imported fields
...) I won't do this in 6.3, because the wizard appears soon and wants
to migrate my old HSQLDB-file.

Most of the databases I have migrated with the wizard are unusable
directly after the migration process. I call such a tool "experimental"
and so LO 6.3.0.4 is completly experimental and not the LO version I am
using for daily work with databases, because the wizard will appear
every time and one (unconcentrated) moment I click the wrong button and
 ... shit happens.

Regards

Robert
--
Homepage: http://robert.familiegrosskopf.de
LibreOffice Community: http://robert.familiegrosskopf.de/map_3


_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Caolán McNamara Caolán McNamara
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

In reply to this post by Alex Thurgood
On Tue, 2019-08-20 at 14:51 +0200, Alexander Thurgood wrote:
> Le 20/08/2019 à 10:28, Julien a écrit :
>
> Hi Julien,
>
> +1 from me.
>
> As a QA triager, I can but agree with Julien given the current stage
> of development. The migration code still fails to handle multiple,
> basic elements of embedded hsqldb ODB files found in actual use.

The last time the firebird issue was brought to the ESC the outcome was
this commit[1] to keep the prompt to migrate, but set the default
button to "no" and add a link to
https://wiki.documentfoundation.org/Documentation/HowTo/MigrateFromHSQLDB

The suggestion in the title of the email is about putting firebird as
experimental, but is the problem solely (or overwhelmingly) that of the
migration process ? Would changing the migration to be experimental,
while leaving the default for new databases as firebird, work ?

something like restoring a use of
SvtMiscOptions::IsExperimentalMode around
https://cgit.freedesktop.org/libreoffice/core/tree/dbaccess/source/core/dataaccess/datasource.cxx#n619


[1]
https://cgit.freedesktop.org/libreoffice/core/commit/?id=0b2eaf8c9595cc5f91b5d31b770e6f23e3b5b27c

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Lionel Elie Mamane Lionel Elie Mamane
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

I note that the tracker bug
https://bugs.documentfoundation.org/show_bug.cgi?id=51780
which is supposed to list bugs we want to see fixed _before_ switching
to Firebird by default for new files has approx 35 blockers.

Note that these bugs are not (supposed to) include any bugs about
migration, that is followed by another tracker bug
https://bugs.documentfoundation.org/show_bug.cgi?id=116968
which blocks the HSQLDB removal tracker bug
https://bugs.documentfoundation.org/show_bug.cgi?id=116970

How many of the blocking bugs should really be blocking bugs is
another question...

On Thu, Aug 22, 2019 at 09:59:56AM +0100, Caolán McNamara wrote:

> On Tue, 2019-08-20 at 14:51 +0200, Alexander Thurgood wrote:
> > Le 20/08/2019 à 10:28, Julien a écrit :
> >
> > Hi Julien,
> >
> > +1 from me.
> >
> > As a QA triager, I can but agree with Julien given the current stage
> > of development. The migration code still fails to handle multiple,
> > basic elements of embedded hsqldb ODB files found in actual use.
>
> The last time the firebird issue was brought to the ESC the outcome was
> this commit[1] to keep the prompt to migrate, but set the default
> button to "no" and add a link to
> https://wiki.documentfoundation.org/Documentation/HowTo/MigrateFromHSQLDB
>
> The suggestion in the title of the email is about putting firebird as
> experimental, but is the problem solely (or overwhelmingly) that of the
> migration process ? Would changing the migration to be experimental,
> while leaving the default for new databases as firebird, work ?
>
> something like restoring a use of
> SvtMiscOptions::IsExperimentalMode around
> https://cgit.freedesktop.org/libreoffice/core/tree/dbaccess/source/core/dataaccess/datasource.cxx#n619
>
>
> [1]
> https://cgit.freedesktop.org/libreoffice/core/commit/?id=0b2eaf8c9595cc5f91b5d31b770e6f23e3b5b27c
>
>
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
JPL JPL
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

In reply to this post by julien2412
Hi,

the migration of a (single-user) database from one RDBMS to another
copes with 3 main families of issues:

(1) the SQLs are not compatible: each has its own dialect, the list of
builtin functions is different, etc. This is visible each time "Direct
SQL" is used in LO, i.e. when making a view or building the SQL
statement programmatically or as soon as the query you need is a bit
complex.

(2) the datatypes are not compatible: e.g. TIMESTAMP has a time zone in
Hsqldb, has none in Firebird, ...

(3) the limits are different: e.g. VARCHAR max. length is < 2Gb in
Hsqldb and  < 32K in Firebird, the max. length of a table row is < 2Gb
in Firebird and <64K in Firebird, the max. length of a column name is
128B in Hsqldb and 31B in Firebird.

NB: the figures above are for Hsqldb 2.x, but, AFAIK they are also valid
for version 1.8. Source =
https://en.wikipedia.org/wiki/Comparison_of_relational_database_management_systems

All this means that a 100% correct and automatic migration of 98+% of
the worldwide LO + embedded Hsqldb files in use IS A DREAM.

My recommendations are:

- Forget forever the idea to firmly invite for a data migration at each
database opening.
- Replace this by a specific menu item by which the user DECIDES to (try
to ... ?) migrate.
- End the migration process with a "Save As ..." type of dialog to store
the result of the migration into a new file.
- If the migration was partial because of incompatibilities, an error
report should list them.

Other valid questions are: should we drop the migration functionality ?
Should we ever drop Hsqldb ? Should we propose Hsqldb 2.x (migration of
database is done by RDBMS itself ... :) ? Are there enough devs ? And,
finally, what are the real benefits of running Firebird i.o. Hsqldb for
our users when designing/running a NEW database ? And let's sell them,
if they exist !

My 2 cents.

Jean-Pierre


Le 20/08/19 à 10:28, Julien a écrit :

> Hello,
>
> Considering the number of bugs about Firebird migration (see https://bugs.documentfoundation.org/show_bug.cgi?id=116968) and the fact that these bugs may not be trivial to fix (see https://wastack.wordpress.com/2018/07/25/database-migration-in-libreoffice-bug-fixes-and-more/), thought it could be relevant to put Firebird back to experimental or at least stop to propose Firebird migration when opening an odb file with hsqldb embedded.
>
> Remark: don't misunderstand me, I'd enjoy to replace HSQLDB to help to remove Java dependency. Also, I know that HSQLDB upgrade has been delayed/cancelled because we had plan to replace it by Firebird but let's face it, I don't think we're ready to propose Firebird widely.
> Let's not forget there are too few dev experts working on Base part (Lionel + Tamás + Jean-Pierre Ledure for specific Access part).
> (I'm trying sometimes to work on Base part but most of bugs are too difficult for me and require a good understanding of Base mechanisms).
>
> Now perhaps I'm too pessimistic here, so would like your opinion.
>
> Any thoughts?
>
> Regards,
>
> Julien
> _______________________________________________
> LibreOffice mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/libreoffice
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
julien2412 julien2412
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

In reply to this post by Caolán McNamara
Hi Caolán,

Considering all the feedback, I submited https://bugs.documentfoundation.org/show_bug.cgi?id=127180 about putting Migration + propose by default Firebird in experimental.
To fullfil this, reverting https://cgit.freedesktop.org/libreoffice/core/commit/?id=8a1a9899e05b0ebbc3ff43f2db242724c8feb18f should respond to the second part.
For the first part, I'll take a look at the code pointer you proposed.

IMHO it should be in master branch + 6.3 branch.

Julien
Le jeudi 22 août 2019 à 11:00:01 UTC+2, Caolán McNamara <[hidden email]> a écrit :


On Tue, 2019-08-20 at 14:51 +0200, Alexander Thurgood wrote:
> Le 20/08/2019 à 10:28, Julien a écrit :
>
> Hi Julien,
>
> +1 from me.
>
> As a QA triager, I can but agree with Julien given the current stage
> of development. The migration code still fails to handle multiple,
> basic elements of embedded hsqldb ODB files found in actual use.

The last time the firebird issue was brought to the ESC the outcome was
this commit[1] to keep the prompt to migrate, but set the default
button to "no" and add a link to

The suggestion in the title of the email is about putting firebird as
experimental, but is the problem solely (or overwhelmingly) that of the
migration process ? Would changing the migration to be experimental,
while leaving the default for new databases as firebird, work ?

something like restoring a use of
SvtMiscOptions::IsExperimentalMode around


[1]


_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
julien2412 julien2412
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

In reply to this post by julien2412
For information, I pushed the patch https://gerrit.libreoffice.org/#/c/78240/

If someone sees some regressions or wants to comment, don't hesitate to tell
of course so we can push an additional patch or revert this one if it brings
too much regression/confusion.



--
Sent from: http://document-foundation-mail-archive.969070.n3.nabble.com/Dev-f1639786.html
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Xisco Fauli Xisco Fauli
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

Hi Julien,

if we want to put it back to experimental, personally I would prefer to
do it *only* in 6.3 and leave it as it's in master so we can test it and
improve it.

In the past, We did the same in 6.2 <
https://cgit.freedesktop.org/libreoffice/core/commit/?h=libreoffice-6-2&id=7991a4d718c282f1fd999e76f683e333b5c220af
> and 6.1 <
https://cgit.freedesktop.org/libreoffice/core/commit/?h=libreoffice-6-1&id=0b3ee8ffbb6b33827c17af56c62dc5285f3ba258
>

Regards

El 29/8/19 a les 20:57, julien2412 ha escrit:

> For information, I pushed the patch https://gerrit.libreoffice.org/#/c/78240/
>
> If someone sees some regressions or wants to comment, don't hesitate to tell
> of course so we can push an additional patch or revert this one if it brings
> too much regression/confusion.
>
>
>
> --
> Sent from: http://document-foundation-mail-archive.969070.n3.nabble.com/Dev-f1639786.html
> _______________________________________________
> LibreOffice mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/libreoffice

--
Xisco Faulí
Libreoffice QA Team
IRC: x1sc0

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Alex Thurgood Alex Thurgood
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

In reply to this post by JPL
Le 22/08/2019 à 12:12, Jean-Pierre Ledure a écrit :

Hi all,

> (3) the limits are different: e.g. VARCHAR max. length is < 2Gb in
> Hsqldb and  < 32K in Firebird, the max. length of a table row is < 2Gb
> in Firebird and <64K in Firebird, the max. length of a column name is
> 128B in Hsqldb and 31B in Firebird.
>

And therein lies the problem, we have substituted the default db engine
for one that is less, or depending on how you look at it, otherwise
capable, in many respects...without providing the means for users of the
existing one to have their data survive the transition, or rather, we
let them believe that it will survive, and then fail at the task.

I feel we would do well to remember that this is people's live and
valuable data we are potentially messing with here, and not all of these
users are DBAs, in fact rather the contrary. It matters not a jot that
db engine XYZ can outperform db engine ABC under circumstance PQR if the
data that the user originally had gets screwed up, or if the yearbook,
contacts with photos, or multilingual accounts DB they were running no
longer functions correctly, or at all, they won't forgive us for it.

And yes, I appreciate that at some stage, a decision on whether we have
reached a sufficiently advanced stage of conversion therapy will need to
be taken and acted upon. The question then is what level of
success/failure is deemed acceptable ?


Alex

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
julien2412 julien2412
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

In reply to this post by Xisco Fauli
Xisco Fauli wrote
> if we want to put it back to experimental, personally I would prefer to
> do it *only* in 6.3 and leave it as it's in master so we can test it and
> improve it.
> ...

It can be tested on master sources just by enabling experimental.
We already put in release notes for 6.3 that migration/Firebird part wasn't
experimental anymore.

If I remember well, we wanted to give a try to this part by putting it by
default.
Considering the number of bugs, we can see the test failed, we're not ready.
So the goal is to avoid letting it this part by default for next release
6.4.0 (except if we can fix enough bugs related to it of course).

Now, if somebody wants to revert the patch which reenabled experimental only
for this part, feel free to do it but I won't do it myself since I'm
strongly against it.

Julien




--
Sent from: http://document-foundation-mail-archive.969070.n3.nabble.com/Dev-f1639786.html
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Terrence Enger Terrence Enger
Reply | Threaded
Open this post in threaded view
|

Re: About putting back Firebird experimental

In reply to this post by Alex Thurgood
On Wed, 2019-09-04 at 16:59 +0200, Alexander Thurgood wrote:
>
> I feel we would do well to remember
that this is people's live and
> valuable data we are potentially messing with here, and not all of
these
> users are DBAs, in fact rather the contrary. It matters not a jot that
> db engine XYZ can
outperform db engine ABC under circumstance PQR if the
> data that the user originally had gets
screwed up, or if the yearbook,
> contacts with photos, or multilingual accounts DB they were running
no
> longer functions correctly, or at all, they won't forgive us for it.

Indeed, I shudder to think what a customer of mine would have done if
I had given them a database conversion with the problems seen in the
conversion we are offering.

(To be fair, my commercial work was easier: I was always converting
one particular database.  And knowledge of the business process and
the customer's goal usually combined to make it pretty obvious how to
handle an infelicity which in the general case would be a problem
without a good solution.)

Still, as always, we can only do what we can do.  The most important
thing we can do is to not surprise the user.  I think we should try
very hard to avoid this grave sin.

To that end, I propose a pre-conversion report showing the problems
which will arise in the conversion of a particular database.  In the
case of a declared-too-long VARCHAR column, the report would show the
afflicted table and column name.  For each such column it would also
show the key values of--let us say--up to three rows where the value
actually stored is too long for Firebird.  And so forth.

I make this proposal diffidently, as it involves more work than I can
do.  I invite your thoughts.

Terry.

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice