CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

classic Classic list List threaded Threaded
25 messages Options
Next » 12
slacka slacka
Reply | Threaded
Open this post in threaded view
|

CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

With both gcc and clang on Fedora 25, the build is failing with an assertion failure. This issue started with commit:

https://cgit.freedesktop.org/libreoffice/core/commit/?id=8242c22f84cef1bbc8c385875b2da4713b542329

[CUT] xmlsecurity_signing
SigningTest::testODFBrokenStreamGPG finished in: 639ms
SigningTest::testODFBrokenDsigGPG finished in: 5475ms
/core/unotest/source/cpp/macros_test.cxx:52:SigningTest::testODFEncryptedGPG
assertion failed
- Expression: xComponent.is()
- loading failed: file:///core//xmlsecurity/qa/unit/signing/data/encryptedGPG.odt

SigningTest::testODFEncryptedGPG finished in: 10168ms
macros_test.cxx:52:Assertion
Test name: SigningTest::testODFEncryptedGPG
assertion failed
- Expression: xComponent.is()
- loading failed: file:///core//xmlsecurity/qa/unit/signing/data/encryptedGPG.odt

Failures !!!
Run: 24   Failure total: 1   Failures: 1   Errors: 0

Error: a unit test failed, please do one of:

make CppunitTest_xmlsecurity_signing CPPUNITTRACE="gdb --args"
    # for interactive debugging on Linux

...

/core/solenv/gbuild/CppunitTest.mk:114: recipe for target '/core/workdir/CppunitTest/xmlsecurity_signing.test' failed
make[1]: *** [/core/workdir/CppunitTest/xmlsecurity_signing.test] Error 1
make[1]: *** Waiting for unfinished jobs....
Makefile:280: recipe for target 'build' failed
make: *** [build] Error 2


 $clang --version
clang version 7.0.0 (trunk 322869)
Target: i686-pc-linux-gnu
$ gcc --version
gcc (GCC) 6.4.1 20170727 (Red Hat 6.4.1-1)

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Thorsten Behrens-6 Thorsten Behrens-6
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

Luke Benes wrote:
> /core/unotest/source/cpp/macros_test.cxx:52:SigningTest::testODFEncryptedGPG
> assertion failed
> - Expression: xComponent.is()
> - loading failed: file:///core//xmlsecurity/qa/unit/signing/data/encryptedGPG.odt
>
Hi Luke,

this is weird - path is constructed exactly the same as for the other
tests - can you perhaps run the tests with
SAL_LOG="+WARN.+INFO.xmlsecurity+INFO.comphelper.crypto" for some
extra diagnosis output?

Likely something deeper inside gpgme and/or gpg is failing here, would
be great if you could narrow that down to a backtrace perhaps?

All the best,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (968 bytes) Download Attachment
slacka slacka
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

This is not a 32/64 bit issue, as I'm seeing this on my 64 bit Ubuntu 17.10 system. The full, unedited log built with SAL_LOG="+WARN.+INFO.xmlsecurity+INFO.comphelper.crypto"

Can be found here:

https://pastebin.com/inwx0Hu0


The previous log was heavily trimmed to reduce size.  While the OS is different both builds share the same external drive. Could the path length be a problem? (I trimmed the full path in the previous log) Or could this be like the "Unit test PythonTest_solenv_python failing on est_gbuildtojson" issue where a make clean isn't enough and I have to manually remove some file from the source tree?

-Luke
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Rene Engelhard Rene Engelhard
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

On Mon, Jan 22, 2018 at 09:03:06PM +0000, Luke Benes wrote:
> The previous log was heavily trimmed to reduce size.  While the OS is different both builds share the same external drive. Could the path length be a problem? (I trimmed the full path in the previous log) Or could this be like the "Unit test PythonTest_solenv_python failing on est_gbuildtojson" issue where a make clean isn't enough and I have to manually remove some file from the source tree?

That one was "gpg creates files the test could not handle". This
supposedly is different, though but I just saw this on
https://buildd.debian.org/status/fetch.php?pkg=libreoffice&arch=all&ver=1%3A6.0.0~rc3-1&stamp=1517070858&raw=0.

A clean build.

Regards,

Rene
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Rene Engelhard Rene Engelhard
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

On Sat, Jan 27, 2018 at 06:20:04PM +0100, Rene Engelhard wrote:
> On Mon, Jan 22, 2018 at 09:03:06PM +0000, Luke Benes wrote:
> > The previous log was heavily trimmed to reduce size.  While the OS is different both builds share the same external drive. Could the path length be a problem? (I trimmed the full path in the previous log) Or could this be like the "Unit test PythonTest_solenv_python failing on est_gbuildtojson" issue where a make clean isn't enough and I have to manually remove some file from the source tree?
>
> That one was "gpg creates files the test could not handle". This
> supposedly is different, though but I just saw this on
> https://buildd.debian.org/status/fetch.php?pkg=libreoffice&arch=all&ver=1%3A6.0.0~rc3-1&stamp=1517070858&raw=0.
>
> A clean build.

That said, it seems to have been successful in the i386 build:

https://buildd.debian.org/status/fetch.php?pkg=libreoffice&arch=i386&ver=1%3A6.0.0~rc3-1&stamp=1517067740&raw=0

Regards,

Rene
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Andika Triwidada Andika Triwidada
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

On Sun, Jan 28, 2018 at 12:31 AM, Rene Engelhard <[hidden email]> wrote:

> On Sat, Jan 27, 2018 at 06:20:04PM +0100, Rene Engelhard wrote:
>> On Mon, Jan 22, 2018 at 09:03:06PM +0000, Luke Benes wrote:
>> > The previous log was heavily trimmed to reduce size.  While the OS is different both builds share the same external drive. Could the path length be a problem? (I trimmed the full path in the previous log) Or could this be like the "Unit test PythonTest_solenv_python failing on est_gbuildtojson" issue where a make clean isn't enough and I have to manually remove some file from the source tree?
>>
>> That one was "gpg creates files the test could not handle". This
>> supposedly is different, though but I just saw this on
>> https://buildd.debian.org/status/fetch.php?pkg=libreoffice&arch=all&ver=1%3A6.0.0~rc3-1&stamp=1517070858&raw=0.
>>
>> A clean build.
>
> That said, it seems to have been successful in the i386 build:
>
> https://buildd.debian.org/status/fetch.php?pkg=libreoffice&arch=i386&ver=1%3A6.0.0~rc3-1&stamp=1517067740&raw=0

No, it was failed too:

SigningTest::testODFBrokenDsigGPG finished in: 222ms
/<<PKGBUILDDIR>>/unotest/source/cpp/macros_test.cxx:52:SigningTest::testODFEncryptedGPG
assertion failed
- Expression: xComponent.is()
- loading failed:
file:///<<PKGBUILDDIR>>//xmlsecurity/qa/unit/signing/data/encryptedGPG.odt

SigningTest::testODFEncryptedGPG finished in: 34ms
macros_test.cxx:52:Assertion
Test name: SigningTest::testODFEncryptedGPG
assertion failed
- Expression: xComponent.is()
- loading failed:
file:///<<PKGBUILDDIR>>//xmlsecurity/qa/unit/signing/data/encryptedGPG.odt

Failures !!!
Run: 24   Failure total: 1   Failures: 1   Errors: 0
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Rene Engelhard Rene Engelhard
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

Hi,

On Sun, Jan 28, 2018 at 08:03:58PM +0700, Andika Triwidada wrote:

> On Sun, Jan 28, 2018 at 12:31 AM, Rene Engelhard <[hidden email]> wrote:
> > On Sat, Jan 27, 2018 at 06:20:04PM +0100, Rene Engelhard wrote:
> >> On Mon, Jan 22, 2018 at 09:03:06PM +0000, Luke Benes wrote:
> >> > The previous log was heavily trimmed to reduce size.  While the OS is different both builds share the same external drive. Could the path length be a problem? (I trimmed the full path in the previous log) Or could this be like the "Unit test PythonTest_solenv_python failing on est_gbuildtojson" issue where a make clean isn't enough and I have to manually remove some file from the source tree?
> >>
> >> That one was "gpg creates files the test could not handle". This
> >> supposedly is different, though but I just saw this on
> >> https://buildd.debian.org/status/fetch.php?pkg=libreoffice&arch=all&ver=1%3A6.0.0~rc3-1&stamp=1517070858&raw=0.
> >>
> >> A clean build.
> >
> > That said, it seems to have been successful in the i386 build:
> >
> > https://buildd.debian.org/status/fetch.php?pkg=libreoffice&arch=i386&ver=1%3A6.0.0~rc3-1&stamp=1517067740&raw=0
>
> No, it was failed too:
[...]

Ah, right, seems the parallelism fooled me once again. Looked for
xmlsecurity_signing but apparently not "way enough down" for it actually
failing.

Anyways, for now I did add that patch to my disable-flaky-tests.diff
(just the encrypt test)

Regards,

Rene
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Andika Triwidada Andika Triwidada
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

On Sun, Jan 28, 2018 at 8:16 PM, Rene Engelhard <[hidden email]> wrote:

> Hi,
>
> On Sun, Jan 28, 2018 at 08:03:58PM +0700, Andika Triwidada wrote:
>> On Sun, Jan 28, 2018 at 12:31 AM, Rene Engelhard <[hidden email]> wrote:
>> > On Sat, Jan 27, 2018 at 06:20:04PM +0100, Rene Engelhard wrote:
>> >> On Mon, Jan 22, 2018 at 09:03:06PM +0000, Luke Benes wrote:
>> >> > The previous log was heavily trimmed to reduce size.  While the OS is different both builds share the same external drive. Could the path length be a problem? (I trimmed the full path in the previous log) Or could this be like the "Unit test PythonTest_solenv_python failing on est_gbuildtojson" issue where a make clean isn't enough and I have to manually remove some file from the source tree?
>> >>
>> >> That one was "gpg creates files the test could not handle". This
>> >> supposedly is different, though but I just saw this on
>> >> https://buildd.debian.org/status/fetch.php?pkg=libreoffice&arch=all&ver=1%3A6.0.0~rc3-1&stamp=1517070858&raw=0.
>> >>
>> >> A clean build.
>> >
>> > That said, it seems to have been successful in the i386 build:
>> >
>> > https://buildd.debian.org/status/fetch.php?pkg=libreoffice&arch=i386&ver=1%3A6.0.0~rc3-1&stamp=1517067740&raw=0
>>
>> No, it was failed too:
> [...]
>
> Ah, right, seems the parallelism fooled me once again. Looked for
> xmlsecurity_signing but apparently not "way enough down" for it actually
> failing.
>
> Anyways, for now I did add that patch to my disable-flaky-tests.diff
> (just the encrypt test)

something like this?

$ git diff
diff --git a/xmlsecurity/qa/unit/signing/signing.cxx
b/xmlsecurity/qa/unit/signing/signing.cxx
index 39a6d0ca6299..13284fd1758f 100644
--- a/xmlsecurity/qa/unit/signing/signing.cxx
+++ b/xmlsecurity/qa/unit/signing/signing.cxx
@@ -143,7 +143,7 @@ public:
     CPPUNIT_TEST(testODFUntrustedGoodGPG);
     CPPUNIT_TEST(testODFBrokenStreamGPG);
     CPPUNIT_TEST(testODFBrokenDsigGPG);
-    CPPUNIT_TEST(testODFEncryptedGPG);
+    //CPPUNIT_TEST(testODFEncryptedGPG);
 #endif
     CPPUNIT_TEST_SUITE_END();

compiled fine with that

Regards,
Andika
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Rene Engelhard Rene Engelhard
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

On Mon, Jan 29, 2018 at 05:14:35PM +0700, Andika Triwidada wrote:

> > Anyways, for now I did add that patch to my disable-flaky-tests.diff
> > (just the encrypt test)
>
> something like this?
>
> $ git diff
> diff --git a/xmlsecurity/qa/unit/signing/signing.cxx
> b/xmlsecurity/qa/unit/signing/signing.cxx
> index 39a6d0ca6299..13284fd1758f 100644
> --- a/xmlsecurity/qa/unit/signing/signing.cxx
> +++ b/xmlsecurity/qa/unit/signing/signing.cxx
> @@ -143,7 +143,7 @@ public:
>      CPPUNIT_TEST(testODFUntrustedGoodGPG);
>      CPPUNIT_TEST(testODFBrokenStreamGPG);
>      CPPUNIT_TEST(testODFBrokenDsigGPG);
> -    CPPUNIT_TEST(testODFEncryptedGPG);
> +    //CPPUNIT_TEST(testODFEncryptedGPG);
>  #endif
>      CPPUNIT_TEST_SUITE_END();

Basically, yes. (Commented out the other occuranced of it, too,
although not needed.)

Obviously this is a hack, though, and this should be fixed properly :)

A data poinit: Works fine in my local builds, but doesn't work on the
buildds. Maybe something missing in those build chroots (which would
make it missing Build-Depends?), gpg is there at least.

(oh, and the buildds set $HOME to /sbuild-nonexistent/, which is
exactly what it says.)

Regards,

Rene

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Thorsten Behrens-6 Thorsten Behrens-6
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

In reply to this post by slacka
Hi Luke,

Luke Benes wrote:
> This is not a 32/64 bit issue, as I'm seeing this on my 64 bit
> Ubuntu 17.10 system. The full, unedited log built with
> SAL_LOG="+WARN.+INFO.xmlsecurity+INFO.comphelper.crypto"
>
Hmm, nothing really helpful standing out there. If you please could:

make CppunitTest_xmlsecurity_signing CPPUNIT_TEST_NAME=testODFEncryptedGPG CPPUNITTRACE="gdb --args"

, and then stick a breakpoint here:

break appopen.cxx:240

(and then run)?

If you hit that breakpoint - do a 'catch throw', then keep
single-stepping. If you reach the end of the CheckPasswd_Impl()
function, I'm at a loss. If you hit exceptions before, I'd be
interested in backtraces for all of them.

Cheers,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (968 bytes) Download Attachment
slacka slacka
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

Here's the log of the debugging session:

https://pastebin.com/60HvYeFa

This was on my 64-bit Ubuntu 17.10 box. Anything useful here?


From: [hidden email] <[hidden email]>
Sent: Thursday, February 1, 2018 5:13 PM
To: Luke Benes
Cc: Libreoffice Dev List
Subject: Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux
 

Hi Luke,

Luke Benes wrote:
> This is not a 32/64 bit issue, as I'm seeing this on my 64 bit
> Ubuntu 17.10 system. The full, unedited log built with
> SAL_LOG="+WARN.+INFO.xmlsecurity+INFO.comphelper.crypto"
>
Hmm, nothing really helpful standing out there. If you please could:

make CppunitTest_xmlsecurity_signing CPPUNIT_TEST_NAME=testODFEncryptedGPG CPPUNITTRACE="gdb --args"

, and then stick a breakpoint here:

break appopen.cxx:240

(and then run)?

If you hit that breakpoint - do a 'catch throw', then keep
single-stepping. If you reach the end of the CheckPasswd_Impl()
function, I'm at a loss. If you hit exceptions before, I'd be
interested in backtraces for all of them.

Cheers,

-- Thorsten
   
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Thorsten Behrens-6 Thorsten Behrens-6
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

Luke Benes wrote:
> This was on my 64-bit Ubuntu 17.10 box. Anything useful here?
>
Hi Luke,

no, not really - could you please do the 'catch throw' part, too, from
my earlier mail? And then single-step inside appopen.cxx, until you
hit an exception?

Cheers,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (968 bytes) Download Attachment
slacka slacka
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux


Sorry this one should be more useful:
https://pastebin.com/ZPFJh5Qp

And for all backtraces:
https://pastebin.com/k280xxEU

Let me know if there is anything else I can do to help.

-Luke


From: [hidden email] <[hidden email]>
Sent: Sunday, February 4, 2018 9:50 PM
To: Luke Benes
Cc: Libreoffice Dev List
Subject: Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux
 

Luke Benes wrote:
> This was on my 64-bit Ubuntu 17.10 box. Anything useful here?
>
Hi Luke,

no, not really - could you please do the 'catch throw' part, too, from
my earlier mail? And then single-step inside appopen.cxx, until you
hit an exception?

Cheers,

-- Thorsten
   
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Thorsten Behrens-6 Thorsten Behrens-6
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

Luke Benes wrote:
> Let me know if there is anything else I can do to help.
>
Hi Luke,

terribly sorry, but the BTs still don't give any useful hint at what's
going wrong. Are you building with optimisation? Since appopen.cxx:240
really contains no throwing (but the called function
decryptGpgSession() does in a few places).

So if I could ask you for one extra trial? Same setup as before, but
then instead of catch throw, please issue a break
docpasswordhelper.cxx:499 (or whatever the 1st line of
DocPasswordHelper::decryptGpgSession() is in your tree), and then
single-step that method, until an exception is thrown?

Cheers,

-- Thorsten

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice

signature.asc (968 bytes) Download Attachment
slacka slacka
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

Here is the new backtrace:

https://pastebin.com/uKsUJMUX

$ cat autogen.input
--enable-debug

So I think optimizations are all disabled.

Let me know if that helps.
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
slacka slacka
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

One more clue as to what is going on. I copied the source tree from my external HD to my home folder and the rebuilt. This time the build succeeded without any errors.


There something about the path on my system that's triggering this bug.


_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
slacka slacka
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

In reply to this post by slacka
Here is the strace log:

https://gist.githubusercontent.com/slacka/a07fac3b1ab0396df066968e2e216c8d/raw/6e1b1414b9db9da04a306a00dd27e1446f9e1377/strace.txt

Let me know if there's anything else I can do to help.
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
Michael Stahl-2 Michael Stahl-2
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

On 07.03.2018 01:10, Luke Benes wrote:
> Here is the strace log:
>
> https://gist.githubusercontent.com/slacka/a07fac3b1ab0396df066968e2e216c8d/raw/6e1b1414b9db9da04a306a00dd27e1446f9e1377/strace.txt
>
> Let me know if there's anything else I can do to help.

gpg fails with:

[pid 19515] write(2, "gpg: can't connect to the agent:"..., 51) = 51
[pid 19515] write(2, "\n", 1)           = 1
[pid 19515] write(2, "gpg: ", 5)        = 5
[pid 19515] write(2, "error: GnuPG agent unusable. Ple"..., 77) = 77
[pid 19515] write(2, "gpg: ", 5)        = 5
[pid 19515] write(2, "migration aborted\n", 18) = 18

the first message might give a reason after the colon but strace
truncates to 32 byte by default; try again with "-s 77" or so added to
get un-truncated messages.

the gpg-agent is [pid 19517] in the trace

[pid 19517] exit_group(2)               = ?
[pid 19517] +++ exited with 2 +++

there appears to be no documentation of gpg-status exit codes.

it's not obvious to me what causes it to fail; there are various ENOENT
returned but at some point strace stops printing strings and just prints
pointers for this process, which doesn't help debugging as it's a
mystery what files are being opened; not sure why it would do that.

[and sorry but i hit the wrong button before and forgot to send the
previous mail to the list]

_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
slacka slacka
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

In reply to this post by slacka
Here is the updated strace log:

https://gist.githubusercontent.com/slacka/412bea7ad1012cb60dac6af583712736/raw/3b8309b7a5dd568c35eb6758edbb15e771e290d2/strace2.

Made with $ make CppunitTest_xmlsecurity_signing CPPUNITTRACE="strace -f -s 77" 2>&1 | tee trace.log


Let me know if there's anything else I can do to help.    
_______________________________________________
LibreOffice mailing list
[hidden email]
https://lists.freedesktop.org/mailman/listinfo/libreoffice
slacka slacka
Reply | Threaded
Open this post in threaded view
|

Re: CppunitTest_xmlsecurity_signing Unit Test Failure on 32-bit Linux

In reply to this post by slacka
Next » 12