Zotero Document Loader Integration

[KamdynS]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Create a document-loader module for downloading documents from Zotero libraries. Pyzotero module has capabilities for this, and it can allow academic users to create RAGs from their saved research very easily.

Motivation

I use Langchain daily at work, but most of our RAG use-cases are for academic users. As such, I've created a script for downloading documents from Zotero and loading them into document loaders to make a RAG from user's own personal Zotero library. I would love if this functionality could just be integrated into the platform.

Proposal (If applicable)

I've already written a module that does this. Integrating it into langchain would take some work, but I'm willing to help contribute if you would like.

[nickdickinson]

Hi @KamdynS

I'm actually doing this right now with pyzotero and I was intending to build the same for llamahub. Currently I load some metadata and citation as well as a documents.

Am I correct that these can be used both for langchain and llama-index? Is there a useful way to combine the work / efforts?

There is also the Paper-QA project: https://github.com/whitead/paper-qa by @whitead

Seems like we would have a pretty solid team if we would combine efforts. I imagine most code could be shared even if there are some framework specific hooks.

[KamdynS]

Would love to collaborate. Current flow that I have is getting a user's personal or group key, filtering through every item, appending YouTube links to a .txt file, and downloading any other documents into storage. I did the integration with LangChain and llama-index separately, but could easily envision combining all into one nice module. I've never contributed to open source before, so I'm not quite sure what the workflow looks like to be quite honest. Willing to share code and/or put in a pull request. Whatever seems best! @nickdickinson

[nickdickinson]

Great! My script takes a library id (personal or group) and then goes through all resources and their attachments and puts them into a local sqlite database with a very simple schema. I make sure it only syncs whatever is required to update and I maintain version numbers etc. I'm not attached to keeping the sqlite dependency but I think it is rather easy to reuse and doesn't require the local zotero installation. It sounds like your workflows adds a number of very useful components: especially getting the different personal/group ids and handling some content types differently. We could have the loading of metadata and download available as a separate standalone package. This would then create a db file and the directory with files. With a db file, I don't think you would need the separate txt file.

May a simplistic diagram:
zotero-local-sync (metadata and files) library --> (1 or 2 doc loader modules for llamahub) --> whatever embedding / RAG pipeline you have in place

I haven't contributed yet to a python library but I do maintain a couple of open source R packages and I'm willing to look into it. I know that LlamaHub also has some requirements for contributions and I'm sure Langchain has some policies as well. I'd be happy to setup a repo.

[wilenius]

I'm very interested in this – let me know if I can help in any way (testing or documentation, for example).

[nickdickinson]

Hi @wilenius, great, thanks for reviving this. Are you using either langchain or llamaindex? I'm using the latter.

[wilenius]

Well, neither, at the moment. =)

Just ended up here after looking what options there are for RAG & Zotero.

Have I understood correctly that it makes more sense to use llamaindex with large Zotero databases?