Key value pairs? Forms?

[ericfeunekes]

I'm curious if you have thoughts on how to use docling for things like completed government forms? E.g extract all key value pairs from a page.

It seems like a difficult problem because a table recognition model could easily get confused between tables and KV pairs. But I'm bringing it up because the extensibility of this library seems like it offers a great opportunity to build something like this that actually works, particularly with the JSON output.

I'm not aware of any specific model that can do it just yet, but even something like a moderately powerful VLM could be inserted in the pipeline somewhere to predict the KV pairs elements.

So have you thought about how to integrate this? How would you build it into the pipeline somewhere as a prediction, even using a powerful closed source model as a proof of concept initially?

[VickySekhon]

This is a great idea, I think that Docling could definitely tackle this issue by creating a specific pipeline targeting government forms as you mentioned. The technology already exists, I believe it's simply a matter of tailoring the output.

[ericfeunekes]

Yeah. It's not even necessarily government forms, but forms generally. So more like key value pair extraction.

I think the important part is differentiating between tables and KV pairs, and then figuring out what the output format would be.

Should this be an issue? It'd be a major help for some stuff I'm working on so happy to test some things out.

[PeterStaar-IBM]

@VickySekhon @ericfeunekes Yes, we are actively working on a form model for exactly this reason. Please stay tuned!

If you want to collaborate with us (eg share some of the forms you are interested in ), please let us know. We would be happy to do so.

[ericfeunekes]

I have thousands of different forms that I'm looking to run this on. How do you want to connect.

[PeterStaar-IBM]

contact us here: [email protected]

[ericfeunekes]

Here are some forms you can try from the Canadian tax authority. You can get French and English by clicking the language in the top right

https://www.canada.ca/en/revenue-agency/services/forms-publications/forms.html

[PeterStaar-IBM]

@ericfeunekes Actually, I am downloading all the forms you listed above and trying to do some regression tests now. I have also my team-members working on propagating the key-value and form regions.

So, not forgotten at all and working behind the scenes to get a good first version!

[mcalello]

Greetings @PeterStaar-IBM and Team! We are very excited about Docling, but our use case will also be working with the US government IRS Forms, and aiming to extract text in a consistent and reliable manner. I created a few samples, after downloading from the IRS website, and entering fake names and only a fake social security number. In one case, docling is able to extract the names and fake social, but they ended up concatenated together. In the other case, which tests OCR, docling extracts the names, but not the fake social security numbers.

I'm happy to share all relates files, but hoping that I'm doing something in error.

from docling.document_converter import DocumentConverter
import sys
import json

filename = ""
if len(sys.argv) > 1:
    filename = sys.argv[1]
    # print(f"Processing file: {filename}")
else:
    print("Please provide a filename as an argument")
    sys.exit(1)


source = filename  # PDF path or URL
converter = DocumentConverter()
result = converter.convert(source)
result_dict = result.document.export_to_dict()
print(json.dumps(result_dict, indent=4))

[PeterStaar-IBM]

@mcalello I actually downloaded all tax-forms and we are now actively working on accurate form understanding. It is however very tedious. Would love for us to collaborate if possible!

[mcalello]

How can I help?

For this month, I'm mainly focused on the most prevalent forms - starting with IRS Form 1040. Does docling need to be trained on this form to properly extract values?

[PeterStaar-IBM]

@mcalello the best help is to gather and collect as diverse (tax-)forms as possible. I you want, we can do this together, just reach out to me!

[Dr-Zorg]

Dear @PeterStaar-IBM and team. I am also very interested in this conversation, although from a very different sector and use-case.
We are seeking ways to extract value pairs from leprosy clinical forms even on primary healthcare (PHC) level. This would enable us to improve the leprosy national case registers and information systems and even intervene at the patient engagement on the PHC level. There are quite a few endemic countries where we can see the application for this. The forms are usually standardized in country, but varies a lot between countries.
Would love to unpack this more, and share some specific examples of forms if you are open to this conversation.

Many thanks for your good work and also for sharing

[PeterStaar-IBM]

@Dr-Zorg We are prepping for large scale annotation of forms, primarily for taxes. If you have forms you can share, please let me know.

[Dr-Zorg]

Dear @PeterStaar-IBM, thanks for your prompt reply.
Yes I do have the forms used in some of the countries we work. Would be happy to share with you.
What channel should I use to share them with you ?

[PeterStaar-IBM]

yes, please write an email to [email protected]. We will follow up!