Duplicated text

[witzatom]

Describe the bug (mandatory)

Some of the data read from this pdf is duplicated on a single page (first page). Visually the text is only once on the page. Also when you send the document into the Adobe PDF -> Doc converter, it does not produce any duplicated text on the first page.

pdf_doc.load_page(0).get_text(sort=True)

RESERVE BANK OF VANUATU\nRESERVE BANK OF VANUATU \n \nQUESTIONNAIRE FOR CONTROLLERS\nQUESTIONNAIRE FOR CONTROLLERS OF BANKS, CREDIT \n, CREDIT \nINSTITUTIONS AND ANY OTHER FINANCIAL INSTITUTIONS \nINSTITUTIONS AND ANY OTHER FINANCIAL INSTITUTIONS \nINSTITUTIONS AND ANY OTHER FINANCIAL INSTITUTIONS \nUNDER THE FIA \n(BEING A BODY CORPORATE)\n(BEING A BODY CORPORATE) \n \nNOTES FOR COMPLETION \nNOTES FOR COMPLETION\n \nA...

Unfortunately it is not exactly duplicated. So a workaround where I would deduplicate by checking for an exact span duplicate is not possible.

Possibly related issues: #379 and #218

This is one of many spans/blocks that are duplicated in the pdf attached.
file.pdf

{'bbox': (208.79991149902344,
          40.227928161621094,
          403.0159912109375,
          55.07319641113281),
 'lines': [{'bbox': (208.79991149902344,
                     40.227928161621094,
                     403.0159912109375,
                     55.07319641113281),
            'dir': (1.0, 0.0),
            'spans': [{'ascender': 0.86181640625,
                       'bbox': (208.79991149902344,
                                40.227928161621094,
                                403.0159912109375,
                                55.07319641113281),
                       'color': 0,
                       'descender': -0.26318359375,
                       'flags': 20,
                       'font': 'Garamond,Bold',
                       'origin': (208.79991149902344, 51.60028076171875),
                       'size': 13.185470581054688,
                       'text': 'RESERVE BANK OF VANUATU'}],
            'wmode': 0}],
 'number': 3,
 'type': 0}

{'bbox': (208.79991149902344,
          40.227928161621094,
          406.0125732421875,
          55.07319641113281),
 'lines': [{'bbox': (208.79991149902344,
                     40.227928161621094,
                     406.0125732421875,
                     55.07319641113281),
            'dir': (1.0, 0.0),
            'spans': [{'ascender': 0.86181640625,
                       'bbox': (208.79991149902344,
                                40.227928161621094,
                                406.0125732421875,
                                55.07319641113281),
                       'color': 0,
                       'descender': -0.26318359375,
                       'flags': 20,
                       'font': 'Garamond,Bold',
                       'origin': (208.79991149902344, 51.60028076171875),
                       'size': 13.183370590209961,
                       'text': 'RESERVE BANK OF VANUATU '}],
            'wmode': 0}],
 'number': 21,
 'type': 0}

To Reproduce (mandatory)

import fitz
from pprint import pprint
pdf_doc = fitz.open("file.pdf")
blocks = pdf_doc.load_page(0).get_text("dict", flags=fitz.TEXT_INHIBIT_SPACES, sort=True)['blocks']

pprint(blocks[0])
pprint(blocks[1])

Your configuration (mandatory)

>>> print(sys.version, "\n", sys.platform, "\n", fitz.__doc__)
3.8.10 (default, Mar 13 2023, 10:26:41) 
[GCC 9.4.0] 
 linux 
 
PyMuPDF 1.21.1: Python bindings for the MuPDF 1.21.1 library.
Version date: 2022-12-13 00:00:01.
Built for Python 3.8 on linux (64-bit).

[JorjMcKie]

This is no bug: as you have observed yourself, the text is stored like that in the file.
So all we can do is discussing options to deal with cases like this.

[witzatom]

The page does not visually contain the text duplicated, but pymupdf says there is duplicate text. I think that this still could be a bug. I was looking for some possible causes of this issue and noting that the header which is on the first page, is also on the 10th page. Idk if the cause is something like those weird control characters like in the issues I linked, or if simply the pdf is broken.

This still does not mean that pymupdf should return the header twice when reading the text on the first page, like i shown in the example.

It seems a less detailed description would have been better in this case, as I think I just caused confusion. Note that online pdf to doc converters do not manifest the same issue.

[witzatom]

First page screenshot

And contents of first page (load_page(0).get_text(sort=True) according to pymupdf):

RESERVE BANK OF VANUATU
RESERVE BANK OF VANUATU 
 
QUESTIONNAIRE FOR CONTROLLERS
QUESTIONNAIRE FOR CONTROLLERS OF BANKS, CREDIT 
, CREDIT 
INSTITUTIONS AND ANY OTHER FINANCIAL INSTITUTIONS 
INSTITUTIONS AND ANY OTHER FINANCIAL INSTITUTIONS 
INSTITUTIONS AND ANY OTHER FINANCIAL INSTITUTIONS 
UNDER THE FIA 
(BEING A BODY CORPORATE)
(BEING A BODY CORPORATE) 
 
NOTES FOR COMPLETION 
NOTES FOR COMPLETION
 
A. The questionnaire should be completed by each 
The questionnaire should be completed by each new controller (body corporate) 
(body corporate) of an 
independently and without assistance by the licensee.  If insufficient 
.  If insufficient 
space is available then separate sheets should be attached. This questionnaire should be 
space is available then separate sheets should be attached. This questionnaire should be 
existing licensee independently and without assistance by the 
space is available then separate sheets should be attached. This questionnaire should be 
completed as part of changes in 
changes in controllers of an existing approved licence holder
holder.  
 ...
 

[JorjMcKie]

Took a quick look at the first page:
As you wrote, there are blocks which simply are exact duplicates of other blocks, while yet others occupy the same place but contain longer text which still starts with the shorter text written at the same position. Interestingly, they also do not necessarily follow each other directly e.g., block 3 'RESERVE BANK OF VANUATU\n' and block 21 'RESERVE BANK OF VANUATU \n' (note the extra space before the line break).

MuPDF does recognize "bold simulation", which is the duplication of same text in (almost) the same position directly following each other. While this is certainly helpful, it cannot cover arbitrary craziness:
The duplication in this case is only a partial one and the duplicates do not follow each other.

So it is left to the wits of the developer to find a solution. Something coming into my mind is:

• make a list of all spans and sort them by (y1, x0, text) where y1, x0 are the span's bbox coordinates and text is span["text"].
• walk through the list of these spans in reverse order and do the following in each loop:
  • if predecessor of span[i] has the same bbox.bl and span[i]["text"].startswith(span[i-1]["text"]), then span at index i-1 can be forgotten. Collect such indices in a separate list to_delete.
• loop through the list to_delete (in reverse order!) and delete the respective span.
• Process the spans that have survived the treatment.

[witzatom]

Yes, I can attempt to deduplicate them. Its just I imagine there will be a lot of edge cases, where it will not work. It might be just better to give up in this case and OCR the pdf instead.
If the issue actually is that the text is duplicated in the pdf, why is this not manifesting in the online pdf->doc converters? For example https://www.adobe.com/ae_en/acrobat/online/pdf-to-word.html

[JorjMcKie]

If the issue actually is that the text is duplicated in the pdf, why is this not manifesting in the online pdf->doc converters?

Because there is a programmatic way dealing with this ... and they are using one such.

[JorjMcKie]

The more interesting question is, how it was possible to create such a mess ...

[witzatom]

Well, I have seen some other cases that are a bit more understandable. In this case I have no idea how this mess was created.

Another example that I have seen, and I know that pymu is handling it correctly (i think), is a 'shadow simulation'. You have some text, and slightly offset in grey behind this, you have text that looks like a shadow of the first text :D

[JorjMcKie]

Yes people are sometimes very creative ...

If you allow me this rather philosophical comment: PDF is a file format primarily intended to display content with the same appearance across platforms and devices - not as a data store to feed downstream applications.
Your examples nicely underpin this statement.

[pkalbuquerque]

So, one year later, how did you resolved the problem?