export_to_markdown() embeds NUL bytes (\x00) for superscript unit/exponent glyphs, causing silent truncation downstream

[davroths-PA]

Bug description

DocumentConverter().convert(...).document.export_to_markdown() embeds literal NUL bytes (\x00) in the exported markdown wherever the source PDF renders a superscript numeral as part of a unit or exponent (e.g. "mL⁻¹", "m⁻³"). Conversion itself reports ConversionStatus.SUCCESS with no errors/warnings, so the corruption is silent.

This is a serious downstream hazard: SQLite/libSQL (and any other C-string-based storage) truncates a TEXT value at the first embedded NUL byte on INSERT. In our pipeline this silently truncated a 77,007-character document down to 1,487 characters — just the abstract — with the write reporting success and no error at any layer.

Reproduction

from docling.document_converter import DocumentConverter

result = DocumentConverter().convert("<pdf>")
md = result.document.export_to_markdown()

assert "\x00" not in md  # fails

Source PDF: a 22-page two-column IntechOpen book chapter (open access, CC-BY): https://doi.org/10.5772/intechopen.77380

What we observed

• result.status → ConversionStatus.SUCCESS, result.errors → []
• len(md) → 77007, but md.count("\x00") → 12
• Every occurrence sits where a superscript unit/exponent should appear. Context around each (\x00 shown literally):

'nent (30.45 -70.60 mg.mL \x00 1 ). A high n'
'thoxyphenol (5 -11 mg.mL \x00 1 ) followed '
'methylphenol (2 -4 mg.mL \x00 1 ). Wood vin'
'              | SG (g.mL \x00 1 )          '
'              |\n| (mg.mL \x00 1 ) 1 L. leuc'
'                       | \x00 2-methoxyphen'
' rubber wood (1.012 g mL \x00 1 ) and bambo'
') and bamboo (1.010 g mL \x00 1 ) were with'
'nent (30.45 -70.60 mg mL \x00 1 ). A high n'
'thoxyphenol (5 -11 mg mL \x00 1 ) followed '
'methylphenol (2 -4 mg mL \x00 1 ), Table 2 '
' the tar oil 460 at kg/m \x00 3 , was effec'

In every case the NUL stands in for a superscript digit that should read as ⁻¹ or ⁻³ (i.e., "mL⁻¹", "m⁻³"). This looks like a font-glyph mapping issue during PDF text extraction — the superscript minus/exponent glyph resolves to codepoint U+0000 instead of the intended character.

Expected behavior

export_to_markdown() (and presumably the underlying TextItem/DoclingDocument text content) should never contain NUL bytes. At minimum, unmappable/control glyphs should degrade to something safe (e.g. dropped, or replaced with a placeholder) rather than passing through as \x00, since NUL is not valid character data in most downstream text pipelines and databases.

Environment

• docling: 2.114.0
• docling-core: 2.87.1
• Python: 3.13.14
• OS: macOS-26.5.2-arm64 (Apple Silicon)
• Conversion used default DocumentConverter() settings (no custom pipeline options)

Workaround

Stripping \x00 from export_to_markdown()'s output before further processing avoids the truncation, but the root cause (glyph mapping to NUL during PDF text extraction) should ideally be fixed upstream so no NUL bytes are emitted in the first place.

[dosubot]

Hi @davroths-PA! I'm Dosu and I'm helping the IBM Docling team.

This is a real gap in the text sanitization pipeline. The root cause is that docling-parse (the C++ PDF parser) produces U+0000 when it can't map a glyph to a Unicode codepoint, and nothing downstream filters it out.

The text extraction path works like this: docling-parse emits SegmentedPdfPage.textline_cells → the backend passes them through unchanged [1] → sanitize_text() runs typographic normalization (dehyphenation, curly quotes, ligature expansion) but has no control character filtering at all. The only control character currently handled anywhere is a hardcoded \x02 → - replacement [2]. NUL bytes (and other C0/C1 control characters) pass through to the final output unchecked.

This is the same class of problem as #3783, where unmapped glyphs get byte-decoded incorrectly — in your case the superscript minus/exponent glyphs resolve to U+0000 instead of the intended ⁻¹ / ⁻³ characters [3].

A fix in sanitize_text() to strip NUL bytes (and ideally other non-whitespace C0/C1 control characters) would be straightforward and prevent the silent downstream corruption you're seeing. The deeper fix — correctly mapping those superscript glyphs in docling-parse itself — would be the ideal long-term solution but is a separate effort upstream.

Your workaround of stripping \x00 from the output is sound in the meantime.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[VascoSch92]

Hey @dosu

Could I give a stab to this issue? The fix should not be too hard.