Problem with checkboxes and comboboxes when convert_to_pdf (flattening)

[dantard]

Hello,
I'm facing a weird problem with some PDF files that contain checkboxes and comboboxes.
When I use the function convert_to_pdf() the library show black boxes instead of the expected checkboxes and comboboxes.
Here the code I used:

import fitz

doc1 = fitz.open("example.pdf")
pdfbytes = doc1.convert_to_pdf()
pdf = fitz.open("pdf", pdfbytes)
pdf.save("converted.pdf")

The example.pdf file is here: example.pdf

I'm using the latest PyMuPDF version (1.23.7). It only happens with some PDF files, not all of them. Any ideas?

On the other hand, I've been trying to workaround this applying a redact annotation and inserting the UTF8 marker checkbox symbol but neither insert_textbox() nor insert_text() seem to deal with UTF8 characters (it is replaced by a ? or a dot). Am I right? Is that a way to make it work?

Thanks in advance.

[JorjMcKie]

Well this PDF is full of errors and fonts unsupported by convert_to_pdf. In so far, a sound recommendation is out of reach 🤷‍♂️.

doc=fitz.open("example.pdf")
pdfdata=oc.convert_to_pdf()
cannot create ToUnicode mapping for ORCPHV+Calibri,Bold
cannot create ToUnicode mapping for RWOMYA+Calibri-Italic
syntax error: cannot find ExtGState resource 'BlendMode0'
encountered syntax errors; page may not be correct

These messages must have been raised at your experiments.

On another note:
insert_textbox() etc. of course deal with UTF8. What they insert depends on the font you give to them. If you let it use the default Helvetica, then of course only the character codes up to 256 are supported!

Just use a file-based, adult font and things will become better.

However, the easiest solution in this case is using the TextWriter class like this:

doc=fitz.open("input.pdf")
page=doc.new_page()
tw = fitz.TextWriter(page.rect)
tw.append((50,100),f"Signs: {chr(0x2610)}, {chr(0x2611)}, {chr(0x2612)} ...")
tw.write_text(page)
doc.save("test.pdf")

If you hand in text that contains characters unsupported by your specified font, it automatically searches the characters in some fallback font. So in this case you will see:

This text can also be extracted w/o problems:

print(page.get_text())
Signs: ☐, ☑, ☒ ...

[dantard]

Thank you for your quick response.
I just get the errors:

cannot create ToUnicode mapping for ORCPHV+Calibri,Bold
cannot create ToUnicode mapping for RWOMYA+Calibri-Italic

but I wasn't sure it was related to the CBs appearing as black boxes.

Regarding the UTF8 problem, I, in fact, suspected it could be a font problem and I tried to switch to other base14 fonts. I assume that was my mistake. Anyway, what font does TextWriter use then instead of Helvetica given that it works without specifying any?

Last question: is there a way to detect situations like these and apply the workaround? I mean, convert_to_pdf() returns the bytes anyway and no exception or return code, right? Would it be a good idea to read the command text output?

Thanks again.

[JorjMcKie]

1. The messages do not pertain to the widgets/fields but generally say that the conversion did not work completely.
2. To intercept this situation, compare fitz.TOOLS.mupdf_warnings() string before and after conversion: if unequal, the conversion did not work as expected.
3. TextWriter uses always file-based fonts - never Base14 (which are not embeddable, and thus PDF viewer/interpreter-dependent). However, it contains font files that look exactly the same as Helvetica, Times-Roman, etc. Without explicit font spec, the font file corresponding to Helvetica is used as default by TextWriter.
4. The black boxes go back to some erroneous specification - I didn't have the time to track down the errors, but there must be some.