Self Query Help

[Dawsonabel]

Self query question

I’ve been reading through everything i can find regarding self query (and also tinkering with it myself), and from what i can tell, i am required to use vectorStore.

Am i misunderstanding the purpose or capabilities of the self query?

Id rather not update all my documents every time i query using the metadata as a filter. That strikes me as a waste of embedding and speed.

I’m hoping someone here can point to a misunderstanding i’m having, because i’m stumped at the moment. The only way i’ve been successful in using this is following it exactly as the documentation shows.

For reference, im using pinecone/js, here’s the documentation:

https://js.langchain.com/docs/modules/data_connection/retrievers/self_query/pinecone-self-query

edit: For clarification, im talking about this snippet:
const selfQueryRetriever = await SelfQueryRetriever.fromLLM({
llm,
vectorStore,
documentContents,
attributeInfo,

[dosubot[bot]]

🤖

Based on your question, it seems like you're trying to understand the purpose and capabilities of the self query feature in the LangChain Javascript framework, specifically in relation to the use of vectorStore, and whether it necessitates updating all documents with each query using metadata as a filter.

The self query feature in the LangChain Javascript framework is designed to allow users to query a set of documents based on their metadata. This is done by converting natural language queries into a format that can be understood by the vector store, which in this case is VectaraStore.

The vector store is where the embeddings of the documents are stored. These embeddings are used to retrieve relevant documents based on the user's query. The SelfQueryRetriever class is used to handle this process. It takes in the language model (llm), the vector store (vectorStore), the document contents (documentContents), the attribute information (attributeInfo), and a structured query translator (structuredQueryTranslator).

The structuredQueryTranslator is responsible for translating the user's query into a format that the vector store can understand. In this case, a VectaraTranslator is used, but users can create their own translator by extending the BaseTranslator abstract class.

The getRelevantDocuments method of the SelfQueryRetriever class is used to retrieve the relevant documents based on the user's query. The method takes in a natural language query and returns the relevant documents.

As for the question about whether it necessitates updating all documents with each query using metadata as a filter, the code does not suggest that all documents need to be updated with each query. The documents are loaded into the vector store once, and then queries are made against this store. The vector store uses the document embeddings and the translated query to retrieve the relevant documents. The documents themselves are not updated or modified during this process.

Here is the relevant code:

const vectorStore = await VectaraStore.fromDocuments(
  docs,
  new FakeEmbeddings(),
  config
);

const llm = new OpenAI();
const documentContents = "Brief summary of a movie";
const selfQueryRetriever = await SelfQueryRetriever.fromLLM({
  llm,
  vectorStore,
  documentContents,
  attributeInfo,
  structuredQueryTranslator: new VectaraTranslator(),
});

const query1 = await selfQueryRetriever.getRelevantDocuments(
  "What are some movies about dinosaurs"
);

In this code, the documents are loaded into the vector store using VectaraStore.fromDocuments. Then, the SelfQueryRetriever is used to retrieve the relevant documents based on the user's query. The documents themselves are not updated or modified during this process.

You can find more information about this in the examples/src/retrievers/vectara_self_query.ts file in the langchainjs repository.

I hope this helps clarify the purpose and capabilities of the self query feature in the LangChain Javascript framework. If you have any more questions, feel free to ask.

Sources

• examples/src/retrievers/vectara_self_query.ts
• langchain-core/src/vectorstores.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Dawsonabel]

What i'm not understanding is, if i'm using a database like pinecone, it still appears like the vectoreStore will store the embeddings in the database every time SelfQueryRetriever is invoked.

Can you explain to me why it is not?

Additionally, to clarify, i am trying to query my pinecone database every time, i do not store them locally.