Technical limitation with Vector Store databases? Do I need to use fine-tuning?

[sergiogcb]

Hey guys.
I was able to set the following project:
I ingested all the information from my Confluence website and then created a vector store database in Pinecone/Vectara.
Then, I proceeded to make questions and answers about that documentation and it worked partially.

The thing is... The documentation from our Confluence is about our app. And it has different pages about how to use it correctly. Our goal is to implement a bot that gives a first answer to our customers based in our public documentation from our app.

The problem is that at the moment, some answers use incorrect syntax from our own documents. And this is to the fact that it searches using the similarity search and just takes 3-4 documents. If I really want to make it learn how it works our code in order to build good solutions (some of the solution it gives are correct, as we have a lot of use cases in our documentation), will I require to use fine-tuning? Can't I just bump the entire vector store database into the model and that proceeds to make his answer without making a search before? Or will that increase a lot the price?
I had already an error saying that the model exceeded the 4096 tokens limit (We're using Chatgpt-3.5 turbo), so I was just asking myself if we should consider to combine Langchain and fine tuning so this model is aware how coding in our app works, so even if it gives a wrong answer at least the syntax is correct and to reduce hallucinations.

If I had to go for fine-tuning, which approach would you suggest?

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out and providing a detailed explanation of your use case. It seems like you're doing some interesting work with LangChain and language models.

To address your concerns, let's break down the issues:

1. Incorrect Syntax in Responses: This is likely due to the fact that the model is generating responses based on the similarity search in the vector store database, which might not always yield the most syntactically correct results. Fine-tuning could indeed help in this case, as it would allow the model to better understand the specific syntax and language used in your app's documentation.

2. Bypassing Similarity Search: Directly feeding the entire vector store database into the model without a preceding search might not be feasible. The model needs to have some form of selection mechanism (like similarity search) to choose relevant documents from the database to generate responses. Without this, the model might struggle to generate coherent and relevant responses due to the sheer volume of information.

3. Exceeding Token Limit: The token limit is a constraint of the underlying language model (ChatGPT-3.5 Turbo in your case). If your documents are too large, you might need to break them down into smaller chunks before feeding them into the model.

4. Fine-tuning Approach: If you decide to go for fine-tuning, you would need to create a dataset that represents the kind of questions and answers you expect the model to handle. This dataset should ideally include examples of correct syntax and usage of your app's language. Once you have this dataset, you can use it to fine-tune the model. This should help the model to better understand the specific language and syntax used in your app's documentation, and thus generate more accurate responses.

In terms of reducing hallucinations, fine-tuning with a well-prepared dataset can indeed help. However, it's also important to note that hallucinations can sometimes be a result of the model not having enough relevant information to generate a response. So, ensuring that your vector store database is comprehensive and well-structured is also crucial.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/tests/unit_tests/document_loaders/test_confluence.py
• libs/langchain/langchain/document_loaders/confluence.py
• libs/langchain/langchain/vectorstores/singlestoredb.py
• libs/langchain/langchain/vectorstores/dingo.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.