[Question] Differences between PyMuPDF blocks extraction in version 1.17 and 1.18

[Lary15]

I am working on an application that currently uses the 1.17.0 version of the pymupdf lib. However, recently I had a demand that would require to extract text within a rectangle object, a feature present in the 1.18.0 version of pymupdf. So I changed the lib version but encountered some differences in regards to the blocks extraction feature in both versions.

For example, the text described on this screenshot:

On the 1.17.0 PyMuPDF version, is extracted as a block like this:

(119.05406188964844, 189.93833923339844, 726.0125732421875, 207.86404418945312, 'ANO XLVIX EDIÇÃO EXTRA No- 3\nBRASÍLIA - DF, TERÇA-FEIRA, 21 DE JANEIRO DE 2020', 1, 0)

And on the 1.18.0 PyMuPDF, like this:

(119.05406188964844, 191.70281982421875, 726.0125732421875, 207.86404418945312, 'ANO XLVIX EDIÇÃO EXTRA No-3 BRASÍLIA - DF, TERÇA-FEIRA, 21 DE JANEIRO DE 2020', 1, 0)

The key aspect here is the conservation of linebreaks that occurs on the 1.17.0 version and which is needed for my application.
So, in resume, I wanted to ask if there is a way to mimic this behavior that maintains linebreaks in block extraction with the 1.18.0 version?

My configuration

• Operating system: ManjaroLinux, 21.1.4
• Python version: 3.8.11
• PyMuPDF installed with pip

[JorjMcKie]

I am sure there is.
Which text extraction option do you use? It looks like get_text("blocks").
The hierachy structure blocks->lines->spans is something that MuPDF creates based on whatever heuristics. It has no physical basis in the document itself.
But I am a little amazed to see that there was a change in the versions - if you use the same code. I am not aware of this.
Can you please show me the code snippet you are using? And if possible the document too - just one page.

I am going to convert this to a discussion ...

[Lary15]

Yes, I am using get_text("blocks"). The result shown is basically the usage of the method on this pdf document page.
I can share the document, here it is:
DODF 003 21-01-2020 EDICAO EXTRA.pdf

[JorjMcKie]

Looks fine - everything like you said you expect:

>>> import fitz
>>> fitz.version
('1.18.14', '1.18.0', '20210601081138')
>>> doc=fitz.open("DODF.003.21-01-2020.EDICAO.EXTRA.pdf")
>>> page=doc[0]
>>> blocks=page.get_text("blocks")
>>> from pprint import pprint
>>> pprint(blocks[:3])
[(56.691890716552734,
  55.67547607421875,
  765.3434448242188,
  214.10137939453125,
  '<image: DeviceGray, width: 1492, height: 330, bpc: 8>',
  0,
  1),
 (56.558387756347656,
  881.2556762695312,
  712.5414428710938,
  898.0530395507812,
  'Este documento pode ser verificado\n'
  'no endereço eletrônico http://www.in.g o v. b r / a u t e n t i c i d a d e '
  '. h t m l ,\n'
  'pelo código 50102020012100001\n'
  'Documento assinado digitalmente conforme MP nº 2.200-2 de 24/08/2001, que '
  'institui a\n'
  'Infraestrutura de Chaves Públicas Brasileira - ICP-Brasil.\n',
  1,
  0),
 (119.05406188964844,
  189.93833923339844,
  726.0125732421875,
  207.86404418945312,
  'ANO XLVIX EDIÇÃO EXTRA No- 3\n'
  'BRASÍLIA - DF, TERÇA-FEIRA, 21 DE JANEIRO DE 2020\n',
  2,
  0)]
>>> 

[Lary15]

Well, that is strange.
I don't know if that could mean something, but for some reason I cannot use the snake case methods. Here is an example of the actual code I am using to test:

>>> fitz.version
('1.18.0', '1.18.0', '20201006071559')
>>> from pprint import pprint
>>> pprint(file[0].getText("blocks")[:3])
[(56.558387756347656,
  881.2556762695312,
  712.5414428710938,
  898.0530395507812,
  'Este documento pode ser verificado no endereço eletrônico http://www.in.g o '
  'v. b r / a u t e n t i c i d a d e . h t m l , pelo código '
  '50102020012100001 Documento assinado digitalmente conforme MP nº 2.200-2 de '
  '24/08/2001, que institui a Infraestrutura de Chaves Públicas Brasileira - '
  'ICP-Brasil.',
  0,
  0),
 (119.05406188964844,
  189.93833923339844,
  726.0125732421875,
  207.86404418945312,
  'ANO XLVIX EDIÇÃO EXTRA No- 3 BRASÍLIA - DF, TERÇA-FEIRA, 21 DE JANEIRO DE '
  '2020',
  1,
  0),
 (201.19203186035156,
  295.4358215332031,
  259.4459228515625,
  309.9049987792969,
  'SEÇÃO II',
  2,
  0)]
>>> 

[JorjMcKie]

1. This is a different document than what you gave me.
2. The snake casing was introduced in full some versions after 1.18.0, but would not have any impact.

[Lary15]

I assure it's the same document. But in your case the get_text('blocks') seems to be capturing images by default. In my code, the getText('blocks') doesn't do that, so it may appear they are different texts but they are the same. Maybe this could indicate that there are some divergent behaviors between the get_text('blocks') and getText('blocks') in the end, but this is just speculating.

[JorjMcKie]

Hm, then for some obscure reason, there might be used a different setting of the flags parameter. Please try to set flags=0 for extraction.
This suppresses outputting image information and converting whitespace to spaces:

>>> import fitz
>>> doc=fitz.open("DODF.003.21-01-2020.EDICAO.EXTRA.pdf")
>>> page=doc[0]
>>> blocks=page.get_text("blocks", flags=0)  # no images, no whitespace conversion
>>> from pprint import pprint
>>> pprint(blocks[:3])
[(56.558387756347656,
  881.2556762695312,
  712.5414428710938,
  898.0530395507812,
  'Este documento pode ser verificado\n'
  'no endereço eletrônico http://www.in.g o v. b r / a u t e n t i c i d a d e '
  '. h t m l ,\n'
  'pelo código 50102020012100001\n'
  'Documento assinado digitalmente conforme MP nº 2.200-2 de 24/08/2001, que '
  'institui a\n'
  'Infraestrutura de Chaves Públicas Brasileira - ICP-Brasil.\n',
  0,
  0),
 (119.05406188964844,
  189.93833923339844,
  726.0125732421875,
  207.86404418945312,
  'ANO XLVIX EDIÇÃO EXTRA No- 3\n'
  'BRASÍLIA - DF, TERÇA-FEIRA, 21 DE JANEIRO DE 2020\n',
  1,
  0),
 (201.19203186035156,
  295.4358215332031,
  259.4459228515625,
  309.9049987792969,
  'SEÇÃO II\n',
  2,
  0)]
>>> 

[Lary15]

I got the same same result even when setting the flags=0. But I also tried to use the same version that you are using for testing, 1.18.14, and I got the blocks with linebreaks:

>>> fitz.version
('1.18.14', '1.18.0', '20210601081138')
>>> from pprint import pprint
>>> pprint(file[0].getText('blocks')[:3])
[(56.691890716552734,
  55.67547607421875,
  765.3434448242188,
  214.10137939453125,
  '<image: DeviceGray, width: 1492, height: 330, bpc: 8>',
  0,
  1),
 (56.558387756347656,
  881.2556762695312,
  712.5414428710938,
  898.0530395507812,
  'Este documento pode ser verificado\n'
  'no endereço eletrônico http://www.in.g o v. b r / a u t e n t i c i d a d e '
  '. h t m l ,\n'
  'pelo código 50102020012100001\n'
  'Documento assinado digitalmente conforme MP nº 2.200-2 de 24/08/2001, que '
  'institui a\n'
  'Infraestrutura de Chaves Públicas Brasileira - ICP-Brasil.\n',
  1,
  0),
 (119.05406188964844,
  189.93833923339844,
  726.0125732421875,
  207.86404418945312,
  'ANO XLVIX EDIÇÃO EXTRA No- 3\n'
  'BRASÍLIA - DF, TERÇA-FEIRA, 21 DE JANEIRO DE 2020\n',
  2,
  0)]

After this, I tried the 1.18.1 and it was also with linebreaks. So it seems to be a change isolated to the 1.17.7~1.18.0 versions.

[JorjMcKie]

Ok, you have a solution then.
I cannot recall what happened at those versions which suppress white hars.

[Lary15]

Yes. thank you for the support.