Best Way to handle regular txt and json files

[simonschoe]

Hi there, I was wondering, is there an optimal way to handle/convert regular txt or json files (e.g., Jupyter notebooks) with docling? If so, which pipeline/converter would you recommend? Skimming the source code, it does not seem that such mime types are explicitly accounted for.

[dolfim-ibm]

Do you mean Jupyter notebooks as input?

One could use nbconvert before using Docling, or a new specific backend would be needed.

[simonschoe]

Yes, I basically mean .txt, .py (which could be treated as a regular .txt file), .json and .ipynb as inputs to the conversion pipeline. All of those would require new backends?

[dolfim-ibm]

The key part is to design how a DoclingDocument with that content should look like.

The current backends (e.g. txt) will only do a simple segmentation of paragraphs. This might not be the best choice for code. I assume for .py the best would be a DoclingDocument with a single code block.

JSON files (raw json or ipynb) would usually need a specialization which is aware of the actual schema in there.

[simonschoe]

@dolfim-ibm thanks for your reply! I understand you correctly that .txt is already supported right now?

With regard to the other types: any chance that it will be on near-future roadmap?

[dolfim-ibm]

For the moment we will rely on a community contribution for those.

I was just checking .txt. It is not natively there, but the markdown backend should be able to handle it.

[simonschoe]

As a quick-win, would it be possible to add plain txt files to the mimetype allowlist for the md backend?