Updating the vector store

[randoentity]

The current vector store implementation via the openai api is a bit dated and seems to be behind an account wall. I'm lukewarm about using RAG to assist in coding tasks, but with better and smaller models coming out I want to believe there's a way to get this working for some use cases. What do you think?

The easiest path to get this working quickly is to use ChromaDB/VectorDB and looking at superboogav2 for inspiration.

SillyTavern has recently pivoted away from ChromaDB and is using Vectra instead. (Not a good source, but: https://old.reddit.com/r/SillyTavernAI/comments/16iclpd/any_advantage_of_vector_storage_compared_to/k0k6enn/) However, that's a nodejs library. A Python alternative seems to be VectorDB, but development there seems to have stalled (did huge context windows kill it?).

So:

• RAG for coding, yay or nay?
• ChromaDB, VectorDB (both) or an alternative?

I'm happy to work on this, but it likely won't be done before the singularity. I'm also keen on making it generic so it plugs into other neovim LLM plugins to make RAG more widely available for vim-kind.

[gsuuon]

Hey! Yeah feel free to play around with it and PR if you land on anything. The RAG stuff that's in now is just from my initial (unfinished) experimentation. I think a separate RAG plugin would be great since it'd be generally useful outside of an LLM context as well. I checked out some other dbs, but I wanted to keep this plugin at as few external dependencies as possible, and I figure for tiny data sets we shouldn't have too many perf issues.

So, RAG for coding: yay especially for local stuff. External vector store: sure, as long as installation is easy. Python is already a necessary dependency. Preferably, we don't add much more than that.

[life00]

Having a RAG integration would be very cool. I have no experience in any AI development but I was searching for a simple way of doing it with ollama as a backend. Such stuff is mostly implemented in the GUI AI clients and there is not much simple examples. The best tutorial I found was https://github.com/ollama/ollama/tree/main/examples/langchain-python-rag-privategpt which I am planning to try reproduce. Maybe you'll find it interesting.