What are doctags

[pwright]

Question

I'm not familiar with doctags, are there any references you could share that might help me understand this format?

[PeterStaar-IBM]

@pwright Doctags is a custom format (a but inspired by xml) that is intended to be optimal for LLM's. We try to represent the content and visual structure with special tags, that can be added to the tokenizer of the LLM.

Our aim is to train/finetune LLM's on doctags to make it intuitively understand document structure.

[petri]

Is this the closest to canonical source so far? https://arxiv.org/html/2503.11576v1 (see A.4 doctags format)

[borisevstratov]

Is this the closest to canonical source so far? https://arxiv.org/html/2503.11576v1 (see A.4 doctags format)

According to the paper:

Each element can nest additional location tags encoding its position on the page as a bounding box, represented in DocTags as <loc_x1><loc_y1><loc_x2><loc_y2>.

I noticed something a bit confusing: the coordinates in JSON/YAML don’t seem to match the ones in the DocTags format. This makes the exported text from DocTags harder to use directly with LLMs.
Is there any additional context on how these formats are meant to align, or whether this mismatch is expected?

@PeterStaar-IBM would you be able to share some context on this?

[borisevstratov]

Okay, I figured it out

Basically the <loc>-tags in the doctags format are coordinates normalized to a 500×500 PDF page grid.
When you want to calculate back the original page coordinates you need to re-calculate them like this:

Let:

• (x1, y1, x2, y2) be the values from <loc_x1><loc_y1><loc_x2><loc_y2>
• (W, H) be the width and height of the page in pixels

Then the pixel coordinates are:

$$\begin{align} l &= x_1 \cdot \frac{W}{500} \\\ r &= x_2 \cdot \frac{W}{500} \\\ t &= y_1 \cdot \frac{H}{500} \\\ b &= y_2 \cdot \frac{H}{500} \end{align}$$

Assuming that the coordinates are already in TOPLEFT origin (y directed downward).
If your coordinates are in BOTTOMLEFT origin, you need to invert the y-axis:

$$t = H - y_2 \cdot \frac{H}{500}, \quad b = H - y_1 \cdot \frac{H}{500}$$

Soure: https://deepwiki.com/search/where-locx1locy1locx2locy2-tag_fbd48b19-ff34-4c90-9906-a8a5ea3c6f6a

[PeterStaar-IBM]

It is likely due to page-size normalization.

[borisevstratov]

By default there is a 500x500 grid, which can be modified by providing xsize and ysize params when exporting to doctags

xsize (int, default: 500 ) – int: (Default value = 500)
ysize (int, default: 500 ) – int: (Default value = 500)

https://docling-project.github.io/docling/reference/docling_document/#docling_core.types.doc.DoclingDocument.export_to_doctags