LLM powered Importer workflow

[joepio]

Atomic Data has its merits over other forms of data:

• Strict Schema using URLs
• Available on the web if needed
• Browseable (links to related information)
• Powerful authorization controls in AtomicServer
• Searchable / filterable in AtomicServer
• Can be easily converted to JSON, RDF and more

So there are plenty of reasons to want to make your data atomic.

But how do you get your existing data into AtomicServer? The current answer is "convert it to JSON-AD", and that works, but it's a pretty big hassle for non-developers. We have virtually no intergrations, no importer tools, nothing. So how do we solve this major task? Surely we can provide easy APIs for creating data (like #976) but there may be something more powerful...

Importer plug-ins

Instead of relying on some external system to put stuff into AtomicServer, AtomicServer can fetch stuff from the outside, using some other existing format and transform it to JSON-AD. But writing these plugins still requires a developer... Or does it?

We could use LLMs to do the hard part. LLMs can write the mapping code, the fetching code, the transform code. It can make the whole ETL. It can iterate, try things, fail, fix the problem. Having a feedback loop here is essential. Let's look at some ways to approach this

• Full cloud docker container. This is what Gemini, Claude, Codex use. Most LLM chat interfaces now have some sort of env where they can play around with some code, experiment. It costs a lot, it's slow, it's powerful.
• JS. The browser can do JS. But no way to sandbox, AFAIK. I don't think this is secure enough.
• WASM + Python (Pyodide). We run python on the client, in WASM(time). We make our own component to view / edit the code. We run the python code in pyodide and provide output to our LLM. LLMs are good at python, and it doesn't require any compilation step. But it does mean we have to load maybe 100+MB of libraries.
• JupyterLite. A full-on python notebook. I dont think we need all of this GUI stuff.