TempusSansITC based phrases displayed as ���������������������

[gbrault]

@JorjMcKie I have a document using TempusSansITC font and all text with this font is output as question mark: do you have any idea how to fix that?
I dumped the document with one of yours provided utility.
I am using fitz latest (1.19.6).
here is the code:

import fitz
def flags_decomposer(flags):
    """Make font flags human readable."""
    l = []
    if flags & 2 ** 0:
        l.append("superscript")
    if flags & 2 ** 1:
        l.append("italic")
    if flags & 2 ** 2:
        l.append("serifed")
    else:
        l.append("sans")
    if flags & 2 ** 3:
        l.append("monospaced")
    else:
        l.append("proportional")
    if flags & 2 ** 4:
        l.append("bold")
    return ", ".join(l)
doc = fitz.open("2022_05_10_ALLEE_DES_PRINCES_ENTREE_A_MANDELIEU_19.04.2022.pdf")
page = doc[0]
blocks = page.get_text("dict", flags=11)["blocks"]
for b in blocks:
    for l in b["lines"]:
        for s in l["spans"]:
            print("")
            font_properties = "Font: '%s' (%s), size %g, color #%06x" % (
                s["font"],  # font name
                flags_decomposer(s["flags"]),  # readable font flags
                s["size"],  # font size
                s["color"],  # font color
            )
            print(s["text"])
            print(font_properties)

Output
`
���������������������
Font: 'TempusSansITC' (serifed, proportional), size 9.95599, color #221f1f

�����������
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #65659a

���������������������
Font: 'TempusSansITC' (serifed, proportional), size 17.9931, color #353994

������� ����������
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #353994

!"#�$� ��%����������������&����'��(�������
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #353994

�����������������������)��* ��* ��* �����
Font: 'TempusSansITC' (serifed, proportional), size 15.9538, color #7c277d

����������������� �!������($��������
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #353994

(���+��������������������#&�!'�� ��#�(�������#
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #353994

(���+�������������������'�&����� ����, ��������'�
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #353994

����-�������� �"��� ����'��
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #353994

(���+����������������� ������ ����'�
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #353994

(���+�������������������'�&����� ���(�����'�
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #353994

�
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #65659a

�������������������������������
Font: 'TempusSansITC' (serifed, proportional), size 11.9953, color #65659a

�$$��.�������
Font: 'TempusSansITC' (serifed, proportional), size 14.0346, color #353994

�����)�(���������%����
Font: 'TempusSansITC' (serifed, proportional), size 14.0346, color #353994

������(� .���
Font: 'TempusSansITC' (serifed, proportional), size 14.0346, color #353994

SARL ELTRON
Font: 'TimesNewRomanPS-BoldMT' (serifed, proportional, bold), size 9.95599, color #221f1f

73 Avenue du Campon - Central Buro - 06110 LE CANNET
Font: 'TimesNewRomanPSMT' (serifed, proportional), size 8.99656, color #221f1f

SARL ELTRON au Capital de 2000 €uros - SIRET : 491 953 105 000 21
Font: 'TimesNewRomanPSMT' (serifed, proportional), size 8.99656, color #221f1f

Tel. : 0493812110 Fax : 04 93 81 21 18
Font: 'TimesNewRomanPSMT' (serifed, proportional), size 8.99656, color #221f1f
`
Could you provide some guidance please?

[gbrault]

The document I read has the following properties

[JorjMcKie]

This font probably does not include a glyph-to-unicode table (apart from very fonts MuPDF does not support).
This information is independent from the visual appearance of a font, i.e. the glyphs.
This cannot be fixed. You can try and confirm that utilities like pdftotext (from XPDF) and others neither can do this.
The only way out is using PyMuPDF's integrated OCR to interpret (only those parts of) the text.

[JorjMcKie]

Pixmap.pdfocr_tobytes(language="eng+fra") / Page.get_textpage_ocr(...language="eng+fra").

[gbrault]

Muchas gracias @JorjMcKie !

[JorjMcKie]

de rien/ de nada, un placer 😉, pero soy un Alemán,
also: keine Ursache, gern geschehen!

[gbrault]

Just one more question: Do you have a list of font to recommend to developers to avoid this kind of issue?

[JorjMcKie]

For output with PyMuPDF, a font with Font.is_writable is generally supported.
The existence of a CMAP (character map) table in the PDF font definition is an indicator, that reading text writtenwith font wil work. This table is pointed to by the PDF key /ToUnicode.
The absence of this key however does not mean, it does not work.

[gbrault]

@JorjMcKie I see something weird with this font TempusSansITC.

• I have one document which only use this font and which works perfectly
• I have another document which mix other font and this one doesn't work
  Any idea?
  2021_11_17_RAPPORT_ELTRON_PALAIS CHRISTINE_DU_16.11.2021.pdf
  This one works
  2022_05_10_ALLEE_DES_PRINCES_ENTREE_A_MANDELIEU_19.04.2022.pdf
  This one don't work

What do you think?

[JorjMcKie]

As I write in previous post:
Example 1 has a /ToUnicode 77 0 R. It is missing in example 2.

# example 1
pprint(page.get_fonts())
[(78, 'ttf', 'TrueType', 'BDBXSV+TempusSansITC', 'R8', '')]
print(doc.xref_object(78))
<<
  /BaseFont /BDBXSV+TempusSansITC
  /FirstChar 1
  /FontDescriptor 76 0 R
  /LastChar 87
  /Subtype /TrueType
  /ToUnicode 77 0 R  # <== sic!
  /Type /Font
  /Widths [ 611 436 500 585 343 351 255 568 254 250 561 376 664
      570 506 622 776 552 300 543 537 600 494 543 753 415 770
      818 600 731 532 657 534 476 495 776 508 895 525 485 387
      250 250 501 627 250 657 698 623 571 522 457 585 257 561
      257 525 250 481 456 457 312 425 865 531 508 430 430 342
      503 739 657 1099 570 257 454 436 544 255 457 920 457 430
      378 500 561 433 ]
>>
# example 2
page=doc[0]
pprint(page.get_fonts())
[(107, 'ttf', 'TrueType', 'MWSUCF+TempusSansITC', 'R15', ''),
 (120, 'ttf', 'TrueType', 'DPRTLH+TimesNewRomanPS-BoldMT', 'R22', ''),
 (124, 'ttf', 'TrueType', 'HAGICE+TimesNewRomanPSMT', 'R24', '')]
print(doc.xref_object(107))  # no /ToUnicode below :-((
<<
  /BaseFont /MWSUCF+TempusSansITC
  /FirstChar 1
  /FontDescriptor 106 0 R
  /LastChar 84
  /Subtype /TrueType
  /Type /Font
  /Widths [ 611 436 500 585 343 351 255 568 254 250 561 664 501
      485 495 570 622 776 552 300 543 537 600 494 250 753 543
      657 415 818 600 731 508 506 532 534 657 476 376 895 770
      776 525 387 250 698 571 522 457 585 257 561 257 525 250
      481 456 457 312 425 865 508 623 342 531 503 739 657 250
      570 1099 430 430 257 454 436 544 255 457 920 457 430 436
      378 ]
>>

That is why.

[gbrault]

@JorjMcKie Thanks again!
What should I say to the developper to select a unicode version of its TempusSansITC? Because in the first document, it works and in the second it doesn't?

In other word: BDBXSV+TempusSansITC <> MWSUCF+TempusSansITC !

[JorjMcKie]

Let's put it this way:
Why on earth did he not include a /ToUnicode CMAP in all of its uses of the font TempusSansITC?

The "ABCDEF+" prefixes only indicate subsets of the same font: subsets are built based on the glyphs used in a specific document.

Some motivation of such approaches may even be on purpose: to immunize a document against text extraction ...

[gbrault]

<<Some motivation of such approaches may even be on purpose: to immunize a document against text extraction ...>>

In this case, I am not sure: I will ask them. But Thanks to give me that hint @JorjMcKie

I believe they have used this font because they "like it" and they don't even figure out it "immunize against text extraction!"

I have to tell them how to check the font they are using to make sure it is unicode.
Assuming they download some TTF file, could we be sure that the following code will tell this font is OK?

from fontTools.ttLib import TTFont
fontpath = "TEMPSITC.TTF" # obviously the right file
for cmap in font['cmap'].tables:
    if cmap.isUnicode():
        print("This is unicode")

Would this ensure it would work or do they have something to verify on the editing tool they use as well?

[JorjMcKie]

I have no time to dig into fonttools for this, but you can check the presence of a /ToUnicode mapping in a given PDF (!) via PyMuPDF:

>>> doc1.xref_get_key(78, "ToUnicode")
("xref", "77 0 R")
>>> doc2.xref_get_key(107, "ToUnicode")
("null", "null")

The font file as such seems to have that info, but something removed it in the case of that file.

[gbrault]

Thanks @JorjMcKie ! It helps