A word of warning about PDF text

classic Classic list List threaded Threaded
4 messages Options
pbw pbw
Reply | Threaded
Open this post in threaded view
|

A word of warning about PDF text

A word of warning about text retrieved from PDF documents.

Recovering text blocks from PDFs is inherently risky.  PDF is a page
definition format, and so it has no notion of the semantics of the text
it contains. It places bits of text at certain positions on the page.
You can create a whole page of text by taking the individual characters
and their attributes and position on the page, shuffling them, and
writing them to the file.  That will produce a readable file, but try
extracting the text from that file. Unless you have a very, very smart
text extractor that reverse-engineers the process of creating the page,
then calculates the _visual_ order of the text elements, you will end up
with gibberish.

_Most_ pdf text, _most_ of the time, is laid on the page in visual
order, but in even the best-behaved files, you are likely to be surprised.

If you don't _know_ that your PDF text extractor program is completely
visually accurate by design, don't tell your boss that you can easily
extract that PDF text, without allowing time for proof-reading every
page. You will get burned.

I don't know how LO extracts PDF text; perhaps it is very sophisticated.
I have my doubts.

--
Peter West
"Other seed fell among thorns, and the thorns grew up and choked it..."

--
To unsubscribe e-mail to: [hidden email]
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/global/users/
All messages sent to this list will be publicly archived and cannot be deleted
Cley Faye Cley Faye
Reply | Threaded
Open this post in threaded view
|

Re: A word of warning about PDF text

2014-01-31 Peter West <[hidden email]>:

> A word of warning about text retrieved from PDF documents.
>
> Recovering text blocks from PDFs is inherently risky.  PDF is a page
> definition format, and so it has no notion of the semantics of the text it
> contains. It places bits of text at certain positions on the page. You can
> create a whole page of text by taking the individual characters and their
> attributes and position on the page, shuffling them, and writing them to
> the file.  That will produce a readable file, but try extracting the text
> from that file. Unless you have a very, very smart text extractor that
> reverse-engineers the process of creating the page, then calculates the
> _visual_ order of the text elements, you will end up with gibberish.
>
> _Most_ pdf text, _most_ of the time, is laid on the page in visual order,
> but in even the best-behaved files, you are likely to be surprised.
>
> If you don't _know_ that your PDF text extractor program is completely
> visually accurate by design, don't tell your boss that you can easily
> extract that PDF text, without allowing time for proof-reading every page.
> You will get burned.
>
> I don't know how LO extracts PDF text; perhaps it is very sophisticated. I
> have my doubts.
>
>
You are right about the fact that a PDF is not meant to be opened for
modification/text recovery. However it is hardly relevant here, as LO is
not (as far as I know...) marketted as a PDF extractor.

While it is possible to open PDF with Draw, even the simplest file will
show you that it is not meant for full and easy recovery: embedded fonts
are not used, some graphics are off by a few pixels (sometime more), and
yes, text get split into an unexpected number of parts, even when the PDF
content is layered correctly in the final file.
For example you can get a single line of text split in three text elements,
or have a single text elements with (seemingly) random spaces inserted in
the middle of words. General page layout is also an issue: a very simple
PDF, containing only a single page of text, show up as two pages on Draw,
with the footer of the first page at the beginning of the second one.

But I do not think any of this is relevant as long as users know that
opening PDF is at most useful for recovering some select elements. Unless
the documentation state otherwise, it is fine, as it works very well for
this specific usage. Opening a PDF in Draw just does this: show the various
elements present in the PDF.

--
To unsubscribe e-mail to: [hidden email]
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/global/users/
All messages sent to this list will be publicly archived and cannot be deleted
Dominique Michel Dominique Michel
Reply | Threaded
Open this post in threaded view
|

Re: A word of warning about PDF text

In reply to this post by pbw
Le Fri, 31 Jan 2014 13:22:41 +1000,
Peter West <[hidden email]> a écrit :

> A word of warning about text retrieved from PDF documents.
>
> Recovering text blocks from PDFs is inherently risky.  PDF is a page
> definition format, and so it has no notion of the semantics of the
> text it contains. It places bits of text at certain positions on the
> page. You can create a whole page of text by taking the individual
> characters and their attributes and position on the page, shuffling
> them, and writing them to the file.  That will produce a readable
> file, but try extracting the text from that file. Unless you have a
> very, very smart text extractor that reverse-engineers the process of
> creating the page, then calculates the _visual_ order of the text
> elements, you will end up with gibberish.
>
> _Most_ pdf text, _most_ of the time, is laid on the page in visual
> order, but in even the best-behaved files, you are likely to be
> surprised.
>
> If you don't _know_ that your PDF text extractor program is
> completely visually accurate by design, don't tell your boss that
> you can easily extract that PDF text, without allowing time for
> proof-reading every page. You will get burned.

It is why I open the pdf file into a separated program and use the
mouse to select the text, and copy/past or Ctrl-C/Ctrl-V. That way, I
have full control on how the text will appear when I select it.

And I use other programs like pdfimages, pdftppm and convert to
extract the images directly from the pdf. They can be turned or
mirrored, it is why convert is useful too. When they are split in small
pieces, pdftoppm give me an exact copy of each page of the pdf, each
page into a ppm file, which is converted in jpeg. In that case, gimp is
useful to extract only the images from these files and cut the text.

The script I use for the images is joined. To use it, place it
somewhere in your path, control it is executable, go into the
directory where your pdf file is, and run 'pdf2jpg'. It will only issue
a help message. Be aware it will extract all the pdf files in that
directory on the fly. Be also aware that, if the final output is jpeg
files, ppm files are automatically used as middle men when needed,
the conversion will be much slower and they can use a lot of space on
the disk.

So, if you want to extract pictures from a 100MB pdf file, count at
least 2GB of temporary disk usage to be safe in all cases. (estimation
from memory, so make you own tests if you don't have a lot of free disk
space)

Also, with some distributions, you may have to adjust the name of the
pdfimages and pdftoppm commands in the script. They are part of poppler
on gentoo (poppler-utils or something like that on Debian), in the past,
they was part of xpdf.

Dominique

>
> I don't know how LO extracts PDF text; perhaps it is very
> sophisticated. I have my doubts.
>

--
To unsubscribe e-mail to: [hidden email]
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/global/users/
All messages sent to this list will be publicly archived and cannot be deleted

Dominique Michel Dominique Michel
Reply | Threaded
Open this post in threaded view
|

Re: A word of warning about PDF text

Le Sat, 1 Feb 2014 01:18:22 +0100,
Dominique Michel <[hidden email]> a écrit :

> Le Fri, 31 Jan 2014 13:22:41 +1000,
> Peter West <[hidden email]> a écrit :
>
> > A word of warning about text retrieved from PDF documents.
> >
> > Recovering text blocks from PDFs is inherently risky.  PDF is a
> > page definition format, and so it has no notion of the semantics of
> > the text it contains. It places bits of text at certain positions
> > on the page. You can create a whole page of text by taking the
> > individual characters and their attributes and position on the
> > page, shuffling them, and writing them to the file.  That will
> > produce a readable file, but try extracting the text from that
> > file. Unless you have a very, very smart text extractor that
> > reverse-engineers the process of creating the page, then calculates
> > the _visual_ order of the text elements, you will end up with
> > gibberish.
> >
> > _Most_ pdf text, _most_ of the time, is laid on the page in visual
> > order, but in even the best-behaved files, you are likely to be
> > surprised.
> >
> > If you don't _know_ that your PDF text extractor program is
> > completely visually accurate by design, don't tell your boss that
> > you can easily extract that PDF text, without allowing time for
> > proof-reading every page. You will get burned.
>
> It is why I open the pdf file into a separated program and use the
> mouse to select the text, and copy/past or Ctrl-C/Ctrl-V. That way, I
> have full control on how the text will appear when I select it.
>
> And I use other programs like pdfimages, pdftppm and convert to
> extract the images directly from the pdf. They can be turned or
> mirrored, it is why convert is useful too. When they are split in
> small pieces, pdftoppm give me an exact copy of each page of the pdf,
> each page into a ppm file, which is converted in jpeg. In that case,
> gimp is useful to extract only the images from these files and cut
> the text.
>
> The script I use for the images is joined. To use it, place it
> somewhere in your path, control it is executable, go into the
> directory where your pdf file is, and run 'pdf2jpg'. It will only
> issue a help message. Be aware it will extract all the pdf files in
> that directory on the fly. Be also aware that, if the final output is
> jpeg files, ppm files are automatically used as middle men when
> needed, the conversion will be much slower and they can use a lot of
> space on the disk.
>
> So, if you want to extract pictures from a 100MB pdf file, count at
> least 2GB of temporary disk usage to be safe in all cases. (estimation
> from memory, so make you own tests if you don't have a lot of free
> disk space)
>
> Also, with some distributions, you may have to adjust the name of the
> pdfimages and pdftoppm commands in the script. They are part of
> poppler on gentoo (poppler-utils or something like that on Debian),
> in the past, they was part of xpdf.
>
> Dominique

The script didn't make it. Here it is:
http://fvwm-crystal.sourceforge.net/other/pdf2jpg

Dominique

>
> >
> > I don't know how LO extracts PDF text; perhaps it is very
> > sophisticated. I have my doubts.
> >
>

--
To unsubscribe e-mail to: [hidden email]
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://listarchives.libreoffice.org/global/users/
All messages sent to this list will be publicly archived and cannot be deleted