Reading order for pseudo-OCR pre-training task

[mustaszewski]

I would like to train the base model for a few more epochs on the pre-training pseudo-OCR task using a custom dataset. In what reading order should the individual words of the document image be passed to the model? The Donut paper states:

The model is trained to read all texts in the image in reading order (from top-left to bottom-right, basically). [...] This task can be interpreted as a pseudo-OCR task.

What does "top-left to bottom-right" mean for multi-column text? For instance, consider the attached dummy document with one heading and two text columns:

Should the document be transcribed as:

• Word1 Col1w1 Col1w2 Col2w1 Col2w2, or
• Word1 Col1w1 Col2w1 Col1w2 Col2w2 ?

I imagine that any dataset used for the pre-training pseudo-OCR task should adopt the same reading order policy as the pe-trained Donut base model. Unfortunately, I am not able to find any information of the exact implementation of "top-left to bottom-right", neither in the paper, the paper supplement, nor the source code. After all, "top-left to bottom-right" can be interpreted in different ways:

• top-to-bottom, left-to-right
• left-to-right, top-to-bottom
• clustering of words into text blocks to mimic semantically meaningful text paragraphs
• etc.

@gwkrsrch can you provide any guidance in this regard, please?

[mohamedsaeed8223]

If it is of any help to you, I tried synthesizing some images with text in them using synthdog ( I lowered the column and row count in the generation config) and had some samples similar to the dummy document you provided, and it had the text ordered in the manner:
Word1 Col1w1 Col2w1 Col1w2 Col2w2

you can try and do the same and generate like 100 samples and look for ones that you might see as interesting and look at their gt parse in the resulting jsonl file