insert_textbox rendering issue

[Andreasox]

Description of the bug

Glyphs with unicode 2000-200A are not rendered if used in the end of the string

How to reproduce the bug

According to #2970 I updated problematic glyphs (0-9 alternates) with Unicode codepoints in the range 2000-200F, see font https://www.dropbox.com/scl/fi/omdtevtwi9r6ps2ym2e44/GothamNarrow-Medium.otf?rlkey=bo7w24upjt9iopb8am51ks4ar&dl=0
insert_text seems to always render these digits correctly e.g. with:
shape.insert_text((286,157), '\u2001\u2001\u2001', fontname='GothamNarrow-Medium', fontsize=14.8)

insert_textbox will not render certain glyphs if ending the string with them, e.g.:
rect = Rect(197.5, 144.0, 397.5, 251.0)
shape.insert_textbox(rect, '\u2001\u200B\u2001', align=1, fontname='GothamNarrow-Medium', fontsize=14.8)

shape.insert_textbox(rect, '\u2001\u2001\u2001', align=1, fontname='GothamNarrow-Medium', fontsize=14.8)
and
shape.insert_textbox(rect, '\u2001\u2001\u2008', align=1, fontname='GothamNarrow-Medium', fontsize=14.8)
does not render any glyph at all

Is it possible to debug this confusing issue?
Is the range 2000-200A not possible to use with insert_textbox?
200B and above seems to work but have not tested all combos yet

PyMuPDF version

1.23.8

Operating system

MacOS

Python version

3.9

[JorjMcKie]

Can you provide the final font please - including your modifications?
Don't want to waste time experimenting around ...

[JorjMcKie]

Several reasons may be possible, like an error in the glyph bbox, etc.

[Andreasox]

Dropbox link to the font (not allowed to be attached as is): https://www.dropbox.com/scl/fi/omdtevtwi9r6ps2ym2e44/GothamNarrow-Medium.otf?rlkey=bo7w24upjt9iopb8am51ks4ar&dl=0

[Andreasox]

Realized that this may be a whitespace issue as the codepoints I selected are different kind of spaces (2000-200A).
I will test with other codepoints when time permits.

[JorjMcKie]

Ok I'll wait and mark this issue accordingly.

[JorjMcKie]

Thinking about it, for the time being this post is more a "Discussions" item than anything else.
Should we uncover a problem later, a more focused issue can be created.

[Andreasox]

It is a whitespace issue, codepoints 2000-200A and 202F will be stripped off from the right end of any string using insert_textbox (and probably with insert_htmlbox as well, not tested). Moving the glyphs to other codepoints do work.
As insert_text displays glyphs in any tested codepoint correctly, I assume this is a feature in mupdf.

I am still struggling to use a variable with syntax protecting from \ and unicode issues, cannot make it work.
Any suggestions are welcome. chr, repr etc. are tested without any success.
Font at https://www.dropbox.com/scl/fi/omdtevtwi9r6ps2ym2e44/GothamNarrow-Medium.otf?rlkey=bo7w24upjt9iopb8am51ks4ar&dl=0

rect = Rect(197.5, 144.0, 397.5, 251.0)
print(linenumber) # returns u2032u2032u2032

# this renders text as is, i.e.  \u2032\u2032\u2032

shape.insert_textbox(rect,linenumber.replace('u', r'\u'), align=1, fontname='GothamNarrow-Medium', fontsize=14.8)

# this renders the "correct" text 111

rect = Rect(197.5, 130.0, 397.5, 251.0)
shape.insert_textbox(rect,'\u2032\u2032\u2032', align=1, fontname='GothamNarrow-Medium', fontsize=14.8)

[Andreasox]

We think it may be a PyMuPDF issue that we cannot get unicode names in a Python variable to render the real glyphs.
Anyway, we moved the glyphs to unicode codepoints which represents "usable characters".
This works, i.e. renders '111':
linenumber = '⅓⅓⅓'
shape.insert_textbox(rect,linenumber, align=1, fontname='GothamNarrow-Medium', fontsize=14.8)

[JorjMcKie]

Downloaded the font from here and made this script

import fitz

fontfile = "GothamNarrow-Medium.otf"
doc = fitz.open()
page = doc.new_page()
page.insert_font(fontname="F0", fontfile=fontfile)
page.insert_text((100, 100), "⅓⅓⅓", fontname="F0", fontsize=14.8)
doc.save("test.pdf")

looks like this:

Same thing for textbox output.

[JorjMcKie]

This time I am getting

[Andreasox]

Aah, wrote the wrong character above. If you use ⅔⅔⅔ you should get 111

[JorjMcKie]

yes, indeed.

[JorjMcKie]

So we have no problem, have we.

[Andreasox]

Correct, we can use this solution. The only issue is if you would like to investigate why we cannot get this to work:
using font https://www.dropbox.com/scl/fi/ml40v668vlojvifjj06z9/GothamNarrow-Medium.240109-1605.otf?rlkey=h1dl8uyb12xwqnkkl8xfzgxh2&dl=0

-- this renders the "correct" text 111
shape.insert_textbox(rect,'\u2032\u2032\u2032', align=1, fontname='GothamNarrow-Medium', fontsize=14.8)

-- this renders \u2032\u2032\u2032 as described in earlier answer
-- we have tested all Python syntax we do know about (r, u, repr etc.) but we cannot get PyMuPDF to render 111
linenumber = 'u2032u2032u2032'
shape.insert_textbox(rect,linenumber.replace('u', r'\u'), align=1, fontname='GothamNarrow-Medium', fontsize=14.8)

[JorjMcKie]

This script works as desired:

import fitz

fontfile = "GothamNarrow-Medium.240109-1605.otf"
text = f"{chr(0x2032)}{chr(0x2032)}{chr(0x2032)}"
doc = fitz.open()
page = doc.new_page()
page.insert_font(fontname="F0", fontfile=fontfile)
page.insert_text((100, 100), text, fontname="F0", fontsize=14.8)
doc.save("test.pdf")