[TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

classic Classic list List threaded Threaded
37 messages Options
Next » 12
Guilhem Moulin Guilhem Moulin
Reply | Threaded
Open this post in threaded view
|

[TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

Dear developers,

As announced a few hours ago in the infra call minutes, we deployed
another git browser — Gitiles —  alongside gitweb.  It's also what the
Google folks are using (in fact they're upstream), and IMHO has quite a
nicer feel than the venerable gitweb/cgit:

  * summary: https://gerrit.libreoffice.org/plugins/gitiles/core/
    vs. https://gerrit.libreoffice.org/gitweb?p=core.git;a=summary

  * commit: https://gerrit.libreoffice.org/plugins/gitiles/core/+/master
    vs. https://gerrit.libreoffice.org/gitweb?p=core.git;a=commit;h=refs/heads/master

  * diff: https://gerrit.libreoffice.org/plugins/gitiles/core/+/master%5E%21/
    vs. https://gerrit.libreoffice.org/gitweb?p=core.git;a=commitdiff;h=refs/heads/master

  * log: https://gerrit.libreoffice.org/plugins/gitiles/core/+log/master/
    vs. https://gerrit.libreoffice.org/gitweb?p=core.git;a=shortlog;h=refs/heads/master

  * tree: https://gerrit.libreoffice.org/plugins/gitiles/core/+/master/
    vs. https://gerrit.libreoffice.org/gitweb?p=core.git;a=tree;h=refs/heads/master

It also integrates better with gerrit (in the form a plugin), scales
better, and doesn't have CGI.pm's clunkiness, subtle bugs, and not so
pretty security record (yet).  And of course fancy Markdown rendering is
always a plus :-)

Right now https://gerrit.libreoffice.org/#/c/$CHANGENUM/ has both gitweb
and Gitiles links, but unless we hear strong objection we'll disable
gitweb at the end of the month, and serve requests to
‘/gitweb?p=${REPOSITORY}.git;a=${ACTION};h=${REF}’ as 302-redirections
to the corresponding Gitiles paths.

Also, while AFAIK the canonical VCS browser for LibreOffice is currently
https://cgit.freedesktop.org/libreoffice/core/ , if you'd like to use
something in TDF infra, like https://git.libreoffice.org/core/, we could
alias it to https://gerrit.libreoffice.org/plugins/gitiles/core/ .  (And
similarly for the other repos.)


Lastly, it's now possible to clone and fetch git repositories over
https:// .  While git:// URLs will remain supported for the foreseeable
future, they're intentionally no longer advertised in gerrit, and we
encourage you to upgrade the scheme of your ‘remote.<name>.url’ to
secure transports (SSH for authenticated access, or HTTPS for anonymous
access).  We'll update ‘lode’ and chase remaining git:// URLs shortly.

Cheers,
--
Guilhem, for The Document Foundation's infrastructure team.

PS: please preserve the CC when replying.

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Miklos Vajna-4 Miklos Vajna-4
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

Hi Guilhem,

On Wed, Oct 17, 2018 at 04:27:54AM +0200, Guilhem Moulin <[hidden email]> wrote:
> Right now https://gerrit.libreoffice.org/#/c/$CHANGENUM/ has both gitweb
> and Gitiles links, but unless we hear strong objection we'll disable
> gitweb at the end of the month, and serve requests to
> ‘/gitweb?p=${REPOSITORY}.git;a=${ACTION};h=${REF}’ as 302-redirections
> to the corresponding Gitiles paths.

The gitweb link takes me to a diff of the whole change, while in the
gitiles case I need one more additional click to the "diff" text to get
there. Would it be possible to keep showing the diff by default?

It's a tiny thing, but if you review code a lot, this saves a lot of
time.

A second point: gitweb supports searching by name, e.g.
<https://gerrit.libreoffice.org/gitweb?p=core.git&a=search&st=author&s=caolan>.
Does gitiles support something like this, or from now on one must clone
the repo to be able to do queries like this?

Thanks,

Miklos

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (201 bytes) Download Attachment
sberg sberg
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

On 17/10/2018 09:26, Miklos Vajna wrote:

> Hi Guilhem,
>
> On Wed, Oct 17, 2018 at 04:27:54AM +0200, Guilhem Moulin <[hidden email]> wrote:
>> Right now https://gerrit.libreoffice.org/#/c/$CHANGENUM/ has both gitweb
>> and Gitiles links, but unless we hear strong objection we'll disable
>> gitweb at the end of the month, and serve requests to
>> ‘/gitweb?p=${REPOSITORY}.git;a=${ACTION};h=${REF}’ as 302-redirections
>> to the corresponding Gitiles paths.
>
> The gitweb link takes me to a diff of the whole change, while in the
> gitiles case I need one more additional click to the "diff" text to get
> there. Would it be possible to keep showing the diff by default?
>
> It's a tiny thing, but if you review code a lot, this saves a lot of
> time.

+1

>
> A second point: gitweb supports searching by name, e.g.
> <https://gerrit.libreoffice.org/gitweb?p=core.git&a=search&st=author&s=caolan>.
> Does gitiles support something like this, or from now on one must clone
> the repo to be able to do queries like this?
>
> Thanks,
>
> Miklos
>
>
> _______________________________________________
> LibreOffice mailing list
> [hidden email]
> https://lists.freedesktop.org/mailman/listinfo/libreoffice
>

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Eike Rathke-2 Eike Rathke-2
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Guilhem Moulin
Hi Guilhem,

On Wednesday, 2018-10-17 04:27:54 +0200, Guilhem Moulin wrote:

>   * diff: https://gerrit.libreoffice.org/plugins/gitiles/core/+/master%5E%21/
>     vs. https://gerrit.libreoffice.org/gitweb?p=core.git;a=commitdiff;h=refs/heads/master

For diffs I much prefer the gerrit view because it highlights changes
within changed lines, for example

https://gerrit.libreoffice.org/plugins/gitiles/core/+/9672d034b9e760f24ac9a6652ab45dee15ee260a%5E%21/

vs

https://gerrit.libreoffice.org/gitweb?p=core.git;a=commitdiff;h=9672d034b9e760f24ac9a6652ab45dee15ee260a

Would that be possible also with gitiles?


> Lastly, it's now possible to clone and fetch git repositories over
> https:// .  While git:// URLs will remain supported for the foreseeable
> future, they're intentionally no longer advertised in gerrit, and we
> encourage you to upgrade the scheme of your ‘remote.<name>.url’ to
> secure transports (SSH for authenticated access, or HTTPS for anonymous
> access).  We'll update ‘lode’ and chase remaining git:// URLs shortly.

Why is git:// deprecated? From what I know it's more efficient when
fetching/pulling than https:// (or ssh://?)

  Eike

--
LibreOffice Calc developer. Number formatter stricken i18n transpositionizer.
GPG key 0x6A6CD5B765632D3A - 2265 D7F3 A7B0 95CC 3918  630B 6A6C D5B7 6563 2D3A

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Guilhem Moulin Guilhem Moulin
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Miklos Vajna-4
Hi Miklos,

On Wed, 17 Oct 2018 at 09:26:40 +0200, Miklos Vajna wrote:

> On Wed, Oct 17, 2018 at 04:27:54AM +0200, Guilhem Moulin <[hidden email]> wrote:
>> Right now https://gerrit.libreoffice.org/#/c/$CHANGENUM/ has both gitweb
>> and Gitiles links, but unless we hear strong objection we'll disable
>> gitweb at the end of the month, and serve requests to
>> ‘/gitweb?p=${REPOSITORY}.git;a=${ACTION};h=${REF}’ as 302-redirections
>> to the corresponding Gitiles paths.
>
> The gitweb link takes me to a diff of the whole change, while in the
> gitiles case I need one more additional click to the "diff" text to get
> there. Would it be possible to keep showing the diff by default?
Ah, yeah.  There is even a custom setting in gerrit.config's “[gitweb]” section
in that effect: “revision = "?p=${project}.git;a=commitdiff;h=${commit}"” :-)

Done on our gerrit stage instance (https://vm178.documentfoundation.org/); will
do the same thing on the prod instance next time we have to restart the service.

> A second point: gitweb supports searching by name, e.g.
> <https://gerrit.libreoffice.org/gitweb?p=core.git&a=search&st=author&s=caolan>.
> Does gitiles support something like this, or from now on one must clone
> the repo to be able to do queries like this?

It does indeed, but it's a bit hidden :-P

    https://gerrit.libreoffice.org/plugins/gitiles/core/+log/?author=caolan

How attached are you to the search form?  We could definitely add one to
the template, but for now I only changed it (on the stage instance) to
make authors & committers clickable on the relevant views, and filter on
their email:

    commit        https://vm178.documentfoundation.org/plugins/gitiles/help/+/master
    commitdiff    https://vm178.documentfoundation.org/plugins/gitiles/help/+/master%5E%21
    log (oneline) https://vm178.documentfoundation.org/plugins/gitiles/help/+log/master/
    log (full)    https://vm178.documentfoundation.org/plugins/gitiles/help/+log/master/?pretty=full

Cheers,
--
Guilhem.

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
slacka slacka
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Guilhem Moulin
Thanks Guilhem! I like the look of gitiles, but they both need better
integration. Could you add hyperlinks to referenced commit messages and to
bugzilla?

cgit:
https://cgit.freedesktop.org/libreoffice/core/commit/?id=048b8e45813f6a19a4ff56e1d676fe9450325cd2
gitiles:
https://gerrit.libreoffice.org/plugins/gitiles/core/+/048b8e45813f6a19a4ff56e1d676fe9450325cd2
gitweb:https://gerrit.libreoffice.org/gitweb?p=core.git;a=commit;h=048b8e45813f6a19a4ff56e1d676fe9450325cd2

Note how gitiles is missing both and gitweb does not link to the bug
tracker.



--
Sent from: http://document-foundation-mail-archive.969070.n3.nabble.com/Dev-f1639786.html
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Guilhem Moulin Guilhem Moulin
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Eike Rathke-2
Hi,

On Wed, 17 Oct 2018 at 14:05:27 +0200, Eike Rathke wrote:

> On Wednesday, 2018-10-17 04:27:54 +0200, Guilhem Moulin wrote:
>
>> * diff: https://gerrit.libreoffice.org/plugins/gitiles/core/+/master%5E%21/
>>   vs. https://gerrit.libreoffice.org/gitweb?p=core.git;a=commitdiff;h=refs/heads/master
>
> For diffs I much prefer the gerrit view because it highlights changes
> within changed lines, for example
>
> https://gerrit.libreoffice.org/plugins/gitiles/core/+/9672d034b9e760f24ac9a6652ab45dee15ee260a%5E%21/
>
> vs
>
> https://gerrit.libreoffice.org/gitweb?p=core.git;a=commitdiff;h=9672d034b9e760f24ac9a6652ab45dee15ee260a
>
> Would that be possible also with gitiles?
Not that I know of, and looking at the source I don't think so

    https://gerrit.googlesource.com/gitiles/+/master/java/com/google/gitiles/HtmlDiffFormatter.java#142

There is a feature request for a side-by-side view in Gitiles (mimicking
the gerrit view), which I assume includes change highlighting; no update
there, though: https://code.google.com/archive/p/gitiles/issues/2 .
 
>> Lastly, it's now possible to clone and fetch git repositories over
>> https:// .  While git:// URLs will remain supported for the foreseeable
>> future, they're intentionally no longer advertised in gerrit, and we
>> encourage you to upgrade the scheme of your ‘remote.<name>.url’ to
>> secure transports (SSH for authenticated access, or HTTPS for anonymous
>> access).  We'll update ‘lode’ and chase remaining git:// URLs shortly.
>
> Why is git:// deprecated? From what I know it's more efficient when
> fetching/pulling than https:// (or ssh://?)

Since v1.6.6 it's no longer true [0], cf. git-http-backend(1) and
https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes .  We're
using the so-called “smart” HTTP protocol, with a git-upload-pack(1)
service on the server.

SSH is only used for transport, a git processed is exec()'ed on the
remote just like for git-daemon(1), so the only overhead is
crypto-related.  The handshake is a one-off thing, thus negligible when
you're transferring a large amount of data at once; and if you're
connecting so often to an SSH remote that the handshake undermines
performance, you should probably activate connection multiplexing in
your client, cf. ssh_config(5) :-)  As for symmetric crypto overhead,
these days everyone has CPUs with AES-NI instructions (at least on build
machines), hence the overhead should be negligible.

Cheers,
--
Guilhem.

[0] https://github.com/git/git/blob/master/Documentation/RelNotes/1.6.6.txt

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Xisco Fauli Xisco Fauli
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Guilhem Moulin
Hi Guilhem,

El 17/10/18 a les 20:10, Guilhem Moulin ha escrit:
> It does indeed, but it's a bit hidden :-P
>
>     https://gerrit.libreoffice.org/plugins/gitiles/core/+log/?author=caolan
>
> How attached are you to the search form?  

Sometimes i use the range search from cgit. e.g

https://cgit.freedesktop.org/libreoffice/core/log/?qt=range&q=647fc41763d1310479d59262734caa296f6e558d..044122de1153bd785ed555c96b6aca67e9d0f739

Do you know how I can do it with gitiles?

Regards

--
Xisco Faulí
Libreoffice QA Team
IRC: x1sc0


_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Guilhem Moulin Guilhem Moulin
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Guilhem Moulin
On Wed, 17 Oct 2018 at 22:31:37 +0000, Xisco Fauli wrote:
> Sometimes i use the range search from cgit. e.g
>
> https://cgit.freedesktop.org/libreoffice/core/log/?qt=range&q=647fc41763d1310479d59262734caa296f6e558d..044122de1153bd785ed555c96b6aca67e9d0f739
>
> Do you know how I can do it with gitiles?

Looks like

    https://gerrit.libreoffice.org/plugins/gitiles/core/+log/647fc4..044122d

if you want the commit messages you can also add a “pretty=full” to the
query string:

    https://gerrit.libreoffice.org/plugins/gitiles/core/+log/647fc4..044122d?pretty=full

or if you want the diff:

    https://gerrit.libreoffice.org/plugins/gitiles/core/+/647fc4..044122d

(These URLs work with the full 40-hexdigits commit IDs, too.)

--
Guilhem.

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Guilhem Moulin Guilhem Moulin
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Guilhem Moulin
On Wed, 17 Oct 2018 at 18:51:31 +0000, slacka wrote:
> Thanks Guilhem! I like the look of gitiles, but they both need better
> integration. Could you add hyperlinks to referenced commit messages
> and to bugzilla?
> […]
> Note how gitiles is missing both and gitweb does not link to the bug
> tracker.

I'd like to get rid of gitweb so I'm reluctant to try to fix this at the
moment; especially since AFAIK it's been like this forever. :-P

But the Gitiles instance does anchor issue tracker references as well
now (same links and regexps as gerrit), and also the full commit IDs.

--
Guilhem.

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Miklos Vajna-4 Miklos Vajna-4
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Guilhem Moulin
Hi Guilhem,

On Wed, Oct 17, 2018 at 08:10:01PM +0200, Guilhem Moulin <[hidden email]> wrote:
> Done on our gerrit stage instance (https://vm178.documentfoundation.org/); will
> do the same thing on the prod instance next time we have to restart the service.

Thanks!

> It does indeed, but it's a bit hidden :-P
>
>     https://gerrit.libreoffice.org/plugins/gitiles/core/+log/?author=caolan
>
> How attached are you to the search form?

That's good enough for me.

Regards,

Miklos

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (201 bytes) Download Attachment
Samuel Mehrbrodt-2 Samuel Mehrbrodt-2
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Guilhem Moulin
Am 17.10.2018 um 20:10 schrieb Guilhem Moulin:
> It does indeed, but it's a bit hidden:-P
>
>      https://gerrit.libreoffice.org/plugins/gitiles/core/+log/?author=caolan
>
> How attached are you to the search form?  We could definitely add one to
> the template, but for now I only changed it (on the stage instance) to
> make authors & committers clickable on the relevant views, and filter on
> their email:
I would prefer having a search form similiar to cgit (where you can
choose searching by commit message, by author, by committer and by range.
I do use that quite often and it should be no big deal to add (as the
search feature itself is already there).

Thanks
Samuel
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Lionel Elie Mamane Lionel Elie Mamane
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Guilhem Moulin
On Wed, Oct 17, 2018 at 09:03:45PM +0200, Guilhem Moulin wrote:
> On Wed, 17 Oct 2018 at 14:05:27 +0200, Eike Rathke wrote:
>> On Wednesday, 2018-10-17 04:27:54 +0200, Guilhem Moulin wrote:

>>> Lastly, it's now possible to clone and fetch git repositories over
>>> https:// .  While git:// URLs will remain supported for the foreseeable
>>> future, they're intentionally no longer advertised in gerrit, and we
>>> encourage you to upgrade the scheme of your ‘remote.<name>.url’ to
>>> secure transports (SSH for authenticated access, or HTTPS for anonymous
>>> access).  We'll update ‘lode’ and chase remaining git:// URLs shortly.

>> Why is git:// deprecated? From what I know it's more efficient when
>> fetching/pulling than https:// (or ssh://?)

> Since v1.6.6 it's no longer true [0], cf. git-http-backend(1) and
> https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes

That webpage doesn't seem to contain a discussion of the efficiency of
the various protocols.

> We're using the so-called “smart” HTTP protocol, with a
> git-upload-pack(1) service on the server.

Interesting. Didn't know that. I'll fight less hard to use git: now.

> SSH is only used for transport, a git processed is exec()'ed on the
> remote just like for git-daemon(1), so the only overhead is
> crypto-related.  The handshake is a one-off thing, thus negligible
> when you're transferring a large amount of data at once; (...) As
> for symmetric crypto overhead, (...) the overhead should be
> negligible.

All I know is that about 1/2/3 years ago ('t was I think in some
coworking space in Brussels, probably a hackfest) I showed Michael
Meeks how to have a separate "push" url (with ssh: protocol) and
"pull" url (with git: protocol) and he was very happy at the
speed-up.

--
Lionel
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Guilhem Moulin Guilhem Moulin
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

On Mon, 22 Oct 2018 at 11:51:35 +0200, Lionel Elie Mamane wrote:

> On Wed, Oct 17, 2018 at 09:03:45PM +0200, Guilhem Moulin wrote:
>> On Wed, 17 Oct 2018 at 14:05:27 +0200, Eike Rathke wrote:
>>> On Wednesday, 2018-10-17 04:27:54 +0200, Guilhem Moulin wrote:
>>>> Lastly, it's now possible to clone and fetch git repositories over
>>>> https:// .  While git:// URLs will remain supported for the foreseeable
>>>> future, they're intentionally no longer advertised in gerrit, and we
>>>> encourage you to upgrade the scheme of your ‘remote.<name>.url’ to
>>>> secure transports (SSH for authenticated access, or HTTPS for anonymous
>>>> access).  We'll update ‘lode’ and chase remaining git:// URLs shortly.
>
>>> Why is git:// deprecated? From what I know it's more efficient when
>>> fetching/pulling than https:// (or ssh://?)
>
>> Since v1.6.6 it's no longer true [0], cf. git-http-backend(1) and
>> https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes
>
> That webpage doesn't seem to contain a discussion of the efficiency of
> the various protocols.
My bad, I probably copy the URL from a wrong tab.  This is what I intended
to share: https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols .
As you can see the protocols are essentially equivalent.

For a high-level overview and pros and cons of each protocol, there is
also https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols ,
which reads

    “There is very little advantage that other protocols have over
    Smart HTTP for serving Git content.” :-)

To be fair, it also says that “The Git protocol is often the fastest
network transfer protocol available”, but that's just because no
encryption is always faster than the fastest encryption.  In practice
however, this argument is moot on modern CPUs.

FWIW, GitHub doesn't mentioned git:// URLs either (even though they're still
supported): https://help.github.com/articles/which-remote-url-should-i-use/ .
 

>> SSH is only used for transport, a git processed is exec()'ed on the
>> remote just like for git-daemon(1), so the only overhead is
>> crypto-related.  The handshake is a one-off thing, thus negligible
>> when you're transferring a large amount of data at once; (...) As
>> for symmetric crypto overhead, (...) the overhead should be
>> negligible.
>
> All I know is that about 1/2/3 years ago ('t was I think in some
> coworking space in Brussels, probably a hackfest) I showed Michael
> Meeks how to have a separate "push" url (with ssh: protocol) and
> "pull" url (with git: protocol) and he was very happy at the
> speed-up.
Might be orthogonal to the git:// vs. https:// vs. ssh:// discussion.
Gerrit uses JGit as Git implementation, while git-daemon(1) spawns
“normal” (C-based) git-upload-pack(1) processes.  I recall Norbert and I
sat down during FOSDEM 2017 to solve perf issues with our JGit
deployment.  Perhaps you configured your ‘remote.<name>.pushurl’ at the
same time :-)

Anyway, it's easy enough to benchmark no-op `git fetch` on core.  master
is currently at c99732d59bc6, and I'm fetching from the same datacenter
to avoid metrics being polluted with network hiccups.

    $ git config remote.origin.url git://git.libreoffice.org/core && time git fetch
    0:01.62 (0.42 user, 0.64 sys)  142108k maxres
    ## Network usage: up 252kiB (4312 packets), down 10168kiB (7197 packets)

    $ git config remote.origin.url https://git.libreoffice.org/core && time git fetch
    0:01.63 (0.81 user, 0.29 sys)  141688k maxres
    ## Network usage: up 56kiB (924 packets), down 4194kiB (1241 packets)

    $ git config remote.origin.url "ssh://$[hidden email]:29418/core" && time git fetch
    0:01.55 (0.62 user, 0.46 sys)  141588k maxres
    ## Network usage: up 67kiB (993 packets), down 9859kiB (1305 packets)

Pretty much equivalent, aren't they? :-)  (Network usage for https:// is
smaller because the TLS termination proxy is also compressing responses
from the git backend.  For git:// I guess the system time is higher than
the user time because it uses use sendfile(2) and friends since there
are no user-space encryption.)

As one might notice, network usage (~10MiB down, and growing) is really
high for a no-op `git fetch`.  That's caused by the >140k refs/changes/…
in the initial git-upload-pack advertisement(1):

    $ git ls-remote https://git.libreoffice.org/core | awk '
        $1 ~ /^[0-9a-f]{40}$/ {
            refs++;
            if ($2 ~ /^refs\/changes\//)
                changes++;
        }
        END {
            printf "refs=%d, changes=%d (%.1f%%)\n",
                refs, changes, 100*changes/refs;
        }
    '
    refs=144709, changes=142676 (98.6%)

All remote types are affected.  Since the number of changesets seems to
grow linearly [0], we should try to find a solution if we want the repository
to keep scaling.  I had an attempt at setting ‘uploadpack.hideRefs’ (and
‘uploadpack.allowTipSHA1InWant’) last Friday, to exclude refs/changes/…
from the initial advertisement, but that broke CI hence needs more work.
There is no urgency anyway (it's not a regression) and although it's
getting worse over time, by the time it's unbearable the Git protocol v2
[1] might save us :-)

--
Guilhem.

[0] https://dashboard.documentfoundation.org/app/kibana#/dashboard/Gerrit
[1] https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Lionel Elie Mamane Lionel Elie Mamane
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

On Mon, Oct 22, 2018 at 04:33:21PM +0200, Guilhem Moulin wrote:
> On Mon, 22 Oct 2018 at 11:51:35 +0200, Lionel Elie Mamane wrote:
>> On Wed, Oct 17, 2018 at 09:03:45PM +0200, Guilhem Moulin wrote:

>>> SSH is only used for transport, a git processed is exec()'ed on the
>>> remote just like for git-daemon(1), so the only overhead is
>>> crypto-related.  The handshake is a one-off thing, thus negligible
>>> when you're transferring a large amount of data at once; (...) As
>>> for symmetric crypto overhead, (...) the overhead should be
>>> negligible.

>> All I know is that about 1/2/3 years ago ('t was I think in some
>> coworking space in Brussels, probably a hackfest) I showed Michael
>> Meeks how to have a separate "push" url (with ssh: protocol) and
>> "pull" url (with git: protocol) and he was very happy at the
>> speed-up.

> Might be orthogonal to the git:// vs. https:// vs. ssh://
> discussion.  Gerrit uses JGit as Git implementation, while
> git-daemon(1) spawns “normal” (C-based) git-upload-pack(1)
> processes.

For us developers of LibreOffice, and thus consumers of the Gerrit /
Git service of freedesktop.org and TDF, whether the difference comes
from the protocol itself or a different git implementation on the
server to serve the different protocols is intellectually interesting
(thanks for that!), but materially inconsequential: if using git: will
be faster, we will use git:.

> I recall Norbert and I sat down during FOSDEM 2017 to solve perf
> issues with our JGit deployment.  Perhaps you configured your
> ‘remote.<name>.pushurl’ at the same time :-)

I can easily believe it was earlier.

> Anyway, it's easy enough to benchmark no-op `git fetch` on core.  master
> is currently at c99732d59bc6, and I'm fetching from the same datacenter
> to avoid metrics being polluted with network hiccups.

Yes, but no. You also test in an environment where a network RTT is
probably about one fifth to one third of a millisecond, and bandwidth
at least 100Mbps if not 1000Mbps? In that case, everything will be
fast. Time difference will be lost in noise. The interesting cases
will be:

1) Someone's out in the woods home DSL line in the woods; fiber hasn't
   come to that village yet, or has come to the town but not that
   particular street. RTT time about 50ms; bandwidth about 20 Mbps
   down (or less), much less up.

2) Case 1 added "on the other side of the world" (South-east asia?
   South America? New Zealand?), you can easily get RTT times of about
   300ms. Even if you are in a ultra-fast network (like university
   network). It is the other side of the world.

3) A coworking space that has good-for-typical-use connection, but
   then TDF does a hackfest there and a bunch of geeks (us) overflow
   the connection.

4) I'm at a conference, half listening to the presentation, half
   hacking on LibreOffice. The conference WiFi is overrun by everyone
   doing the same, people's laptops and pocket computers
   ("smartphones") automatically downloading updates (technical and
   social ones...), etc. How usable will it be? E.g. CCC (the Chaos
   Communication Congress) was known for having a totally overwhelmed
   WiFi; every year a new vendor would "gift" their better solution
   and this year the wireless network would actually be good! But
   every year it wasn't. (Has it actually improved in the last years
   that I didn't go?)

Are these protocols (or the *implementations* of these protocols) more
sensitive to RTT than another? They do more roundtrips? Or not?

--
Lionel
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Wols Lists Wols Lists
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

On 22/10/2018 16:25, Lionel Elie Mamane wrote:
> 1) Someone's out in the woods home DSL line in the woods; fiber hasn't
>     come to that village yet, or has come to the town but not that
>     particular street. RTT time about 50ms; bandwidth about 20 Mbps
>     down (or less), much less up.

This isn't a backwoods place - you're describing my internet (ADSL 2)
which gives me 17Mb down, about 500Kb up, and I'm in London ... (FTTC
isn't an option, and FTTP costs a bomb ...)

(That's assuming my local broadband connection hasn't fallen over - it
regularly locks up of an evening and I can only guess that the local
telephone exchange is overloaded. BT would love for me to pay for an
upgrade but why should I when what I've got is plenty when it's working.)

Cheers,
Wol
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Guilhem Moulin Guilhem Moulin
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Lionel Elie Mamane
On Mon, 22 Oct 2018 at 17:25:11 +0200, Lionel Elie Mamane wrote:

> On Mon, Oct 22, 2018 at 04:33:21PM +0200, Guilhem Moulin wrote:
>> On Mon, 22 Oct 2018 at 11:51:35 +0200, Lionel Elie Mamane wrote:
>>> On Wed, Oct 17, 2018 at 09:03:45PM +0200, Guilhem Moulin wrote:
>>>> SSH is only used for transport, a git processed is exec()'ed on the
>>>> remote just like for git-daemon(1), so the only overhead is
>>>> crypto-related.  The handshake is a one-off thing, thus negligible
>>>> when you're transferring a large amount of data at once; (...) As
>>>> for symmetric crypto overhead, (...) the overhead should be
>>>> negligible.
>
>>> All I know is that about 1/2/3 years ago ('t was I think in some
>>> coworking space in Brussels, probably a hackfest) I showed Michael
>>> Meeks how to have a separate "push" url (with ssh: protocol) and
>>> "pull" url (with git: protocol) and he was very happy at the
>>> speed-up.
>
>> Might be orthogonal to the git:// vs. https:// vs. ssh://
>> discussion.  Gerrit uses JGit as Git implementation, while
>> git-daemon(1) spawns “normal” (C-based) git-upload-pack(1)
>> processes.
>
> For us developers of LibreOffice, and thus consumers of the Gerrit /
> Git service of freedesktop.org and TDF, whether the difference comes
> from the protocol itself or a different git implementation on the
> server to serve the different protocols is intellectually interesting
> (thanks for that!), but materially inconsequential: if using git: will
> be faster, we will use git:.
Following the same logic, you want gerrit.libreoffice.org to serve
content over plain http:// so you can save the two round-trips when you
launch your browser to submit your reviews? Oo

FWIW, we're stuck with git:// for the years to come because there is no
smooth upgrade path for clients; if I were to deploy the service today I
would most likely skip git-daemon(1).  Things have changed since 2012,
encryption is faster (there are modern stream ciphers and hardware
acceleration is more widespread), and for situations like this one there
is no reason not to encrypt data in transit.
 
>> I recall Norbert and I sat down during FOSDEM 2017 to solve perf
>> issues with our JGit deployment.  Perhaps you configured your
>> ‘remote.<name>.pushurl’ at the same time :-)
>
> I can easily believe it was earlier.

Then it was before my time, so no idea what the bottleneck was.

>> Anyway, it's easy enough to benchmark no-op `git fetch` on core.  master
>> is currently at c99732d59bc6, and I'm fetching from the same datacenter
>> to avoid metrics being polluted with network hiccups.
>
> Yes, but no. You also test in an environment where a network RTT is
> probably about one fifth to one third of a millisecond, and bandwidth
> at least 100Mbps if not 1000Mbps? In that case, everything will be
> fast. Time difference will be lost in noise.

I was arguing that C git and Jgit's performances are indistinguishable
on the current instance.  Transport overhead is the normal batch-mode
SSH (resp. TLS) overhead for ssh:// (resp. https://) remotes.

As mentioned earlier the protocol is essentially the same for git:// and
http:// (on servers supporting smart HTTP).  In both cases there is a
first round-trip (client hello + server git-upload-pack advertisement),
and another if the client is missing some object(s) (client sends list
of missing objects and receives a pack back).  For http:// these are
done in two sequential requests to the same connection (resp. ‘GET
/$REPO/info/refs?service=git-upload-pack’ and ‘POST
/$REPO/git-upload-pack’); for git:// there are equivalent requests in
the Git wire protocol.

https:// is just http:// wrapped in TLS, which costs an extra 2 round-trips
(TLS 1.3 brings that down to a single extra round-trip, but we don't offer
it yet).

For ssh://, what happens under the hood (as witnessed when adding
“GIT_TRACE=1” to the environment) is that an ssh process is spawned to
run git-upload-pack on the remote machine:

    ssh -p 29418 gerrit.libreoffice.org git-upload-pack "/core"

Counting round-trips for SSH isn't trivial, but let me try:
  * Client & server greet each other (in parallel)
  * Client & server initiate KEX (in parallel)
  * Key EXchange
  * Client & server send NEWKEYS (in parallel)
  * Client requests service, wait for response
  * Client send each pubkey one at a time, waits for response; for the
    one that's accepted by the server, it sends the signed response and
    waits for the server to ack
  * Client asks to open a new channel, waits for response
  * Client asks to execute command in said channel, wait for response
  * […]
  * Server sends EOF and closes channel
  * Client acks, closes channels, and disconnects

So if the latency is symmetric and the first key offered is accepted by
the server, that makes a constant overhead of 8.5 round-trips.  (When
using an existing — multiplexed — connection the overhead becomes 2.5
round-trips.)  Additionally, the sending side must wait for the client
to adjust the window size when it's full. (OpenSSH's window size is 2MiB
at compile-time and is adjusted on the fly depending on network
conditions, cf. RFC 4254 sec. 5.2 for details.)

> Are these protocols (or the *implementations* of these protocols) more
> sensitive to RTT than another? They do more roundtrips? Or not?

Given the current (and growing) size of the git-upload-pack advertisement,
I doubt latency will be the bottleneck here.  Not until we manage to
shrink it.

FWIW there is another advantage of using HTTP as pull URL, namely that
capable clients can send HTTP headers such “Accept-Encoding: deflate,
gzip” (on Debian Jessie and later it's compiled in, not sure if that's
an upstream default).  That way the backend can compress responses it
thinks are worth compressing.  As shown in my earlier message, this
halves the size of git-upload-pack advertisement, despite the fact that
it contains 145k random 40-hexdigits long strings.  AFAIK compression of
data in transit isn't in the git protocol, hence not available for
ssh:// and git:// URLs.  (For SSH one doesn't want transport-level
compression, as packs are already compressed by the git protocol.)
Saving 5MiB per fetch is certainly interesting in low-bandwidth
networks.

Also, TCP port 443 is less likely to be blocked than 9418 :-)

--
Guilhem.

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (849 bytes) Download Attachment
Lionel Elie Mamane Lionel Elie Mamane
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

On Tue, Oct 23, 2018 at 07:34:54AM +0200, Guilhem Moulin wrote:
> On Mon, 22 Oct 2018 at 17:25:11 +0200, Lionel Elie Mamane wrote:
>> On Mon, Oct 22, 2018 at 04:33:21PM +0200, Guilhem Moulin wrote:

>>> Might be orthogonal to the git:// vs. https:// vs. ssh://
>>> discussion.  Gerrit uses JGit as Git implementation, while
>>> git-daemon(1) spawns “normal” (C-based) git-upload-pack(1)
>>> processes.

>> For us developers of LibreOffice, and thus consumers of the Gerrit
>> / Git service of freedesktop.org and TDF, whether the difference
>> comes from the protocol itself or a different git implementation on
>> the server to serve the different protocols is intellectually
>> interesting (thanks for that!), but materially inconsequential: if
>> using git: will be faster, we will use git:.

> Following the same logic, you want gerrit.libreoffice.org to serve
> content over plain http:// so you can save the two round-trips when
> you launch your browser to submit your reviews? Oo

Submission (write access) is something else entirely than code
download (read access); the security requirements are massively
different. (Yes, I would prefer to be certain that the code I get is
the right one; however, if I don't and try to submit a patch on top of
code that is not the on in the TDF repo, it will fail. Unless the
attacker also constructed git that exploits SHA collisions?)

(The above analysis does not apply to gerrit-as-a-website, because
 there the link between the code I see and the code I approve is not
 on my local machine, but depends on the security of the connection;
 and because I don't know how to secure reading but not writing on a
 website.)

If it is only two round-trips per operation, I'm likely to not notice
and thus be perfectly happy to use whatever protocol. If the two round
trips are multiplied by many many requests to serve one operation,
then I may notice. Where "operation" is one action for the user, not
one action for the program. E.g., "git fetch", "git push", "load a
complete web page with all images, javascript, dependencies, etc",
"post an answer to a gerrit change", as opposite to "get one
commit / changepack among 2473 to be downloaded", "get one image / one
javascript file", etc.

> Things have changed since 2012, encryption is faster (there are
> modern stream ciphers and hardware acceleration is more widespread),
> and for situations like this one there is no reason not to encrypt
> data in transit.

I would have thought symmetric encryption already wasn't a bottlneck,
at least on the client side (maybe on the server side it was?) in
2012; I think one could even back then easily saturate a 100Mbps
connection using less than 100% of one core of an entry-level desktop
CPU. Many public key crypto operations per seconds is (was?) another
issue altogether.

Possibly the difference I was very clearly feeling back then was
C git vs JGit, and not having a clue that different implementations
were being used, were just carelessly blamed on crypto overhead.

>>> Anyway, it's easy enough to benchmark no-op `git fetch` on core.
>>> master is currently at c99732d59bc6, and I'm fetching from the
>>> same datacenter to avoid metrics being polluted with network
>>> hiccups.

>> Yes, but no. You also test in an environment where a network RTT is
>> probably about one fifth to one third of a millisecond, and
>> bandwidth at least 100Mbps if not 1000Mbps? In that case,
>> everything will be fast. Time difference will be lost in noise.

> I was arguing that C git and Jgit's performances are
> indistinguishable on the current instance.

Ah, I understand now.

> Counting round-trips for SSH isn't trivial, but let me try:
> (...)  So if the latency is symmetric and the first key offered is
> accepted by the server, that makes a constant overhead of 8.5
> round-trips. (... ) Additionally, the sending side must wait for the
> client to adjust the window size when it's full.

Since we were feeling a difference between git: and ssh:, not between
git: and http(s): (which we didn't even consider), possibly that's
also a factor.

--
Lionel
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Fitoschido Fitoschido
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

In reply to this post by Guilhem Moulin
Hi, Guilhem:

> Right now https://gerrit.libreoffice.org/#/c/$CHANGENUM/ has both gitweb
> and Gitiles links, but unless we hear strong objection we'll disable
> gitweb at the end of the month

Here’s an objection! :P

Well, I have this use case: every diff in the translations repository
is huge, as translation-update commits touch hundreds of files and
diffs are very large. This has always been a problem with FDO’s cgit,
which isn’t even able to load any page. So I use gitweb to review the
commits and spot possible typos. It’s part of the QA I do for my
translations.

Since loading the whole diff for a commit would potentially crash my
browser, I find it very helpful to load each file’s individual diff in
a commit like this [1]. Also, I can open per-file diffs in separate
tabs for better organization.

Gitiles lacks this kind of per-file-diff links, so I can only load a
diff of all the files touched by the commit. This isn’t nice for my
underpowered computer, and it really breaks my workflow.

Adolfo
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Kaganski Mike Kaganski Mike
Reply | Threaded
Open this post in threaded view
|

Re: [TDF infra] Announcing Gitiles VCS browser (gitweb replacement) and https:// anon git URIs

On 10/23/2018 12:29 PM, Adolfo Jayme Barrientos wrote:
> Gitiles lacks this kind of per-file-diff links, so I can only load a
> diff of all the files touched by the commit. This isn’t nice for my
> underpowered computer, and it really breaks my workflow.

Well - e.g.,
https://gerrit.libreoffice.org/plugins/gitiles/core/+/b0c6d587405af9e2263dc5073a9a965db46ff986 
shows the individual links for each file like this:

icon-themes/breeze/links.txt[diff]
icon-themes/breeze_dark/links.txt[diff]
icon-themes/breeze_svg/links.txt[diff]
icon-themes/colibre/links.txt[diff]
icon-themes/karasa_jaga/links.txt[diff]
officecfg/registry/data/org/openoffice/Office/UI/WriterCommands.xcu[diff]
sw/uiconfig/sglobal/popupmenu/frame.xml[diff]
sw/uiconfig/sweb/popupmenu/frame.xml[diff]
sw/uiconfig/swform/popupmenu/frame.xml[diff]
sw/uiconfig/swreport/popupmenu/frame.xml[diff]
sw/uiconfig/swriter/popupmenu/frame.xml[diff]
sw/uiconfig/swxform/popupmenu/frame.xml[diff]
12 files changed

Note the [diff] next to the filename.

What seems odd to me is why the log like
https://gerrit.libreoffice.org/plugins/gitiles/core/+log/HEAD shows
commit creatin time that is different from the time the commit was
pushed to the branch. I would like to see just-pushed commits to tell "5
minutes ago" instead "3 days ago" like it might be now.

--
Best regards,
Mike Kaganski
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Next » 12