Missing metadata when querying with ConversationalRetrievalQAChain

[marcdelalonde]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

const questionGenerator = new LLMChain({
  llm: new OpenAI({ temperature: 0 }),
  prompt: CONDENSE_PROMPT,
});

const docChain = loadQAChain(
  new OpenAI({ temperature: 0, modelName: "gpt-4-turbo" }),
  {
    type: "stuff",
    prompt: QA_PROMPT,
  }
);

return new ConversationalRetrievalQAChain({
  retriever: vectorstore.asRetriever(),
  combineDocumentsChain: docChain,
  questionGeneratorChain: questionGenerator,
  returnSourceDocuments: true,
  k: 1, // number of source documents to return
});

Description

Hello,

Langchain only return loc.lines, sourceURL and title, while there is a lot more metadata in pinecone.

How to get all the metadata when querying?

Thank you

System Info

node v21.6.1

[dosubot[bot]]

To retrieve the full range of metadata from Pinecone when using ConversationalRetrievalQAChain in LangChain, follow these steps:

1. Index Additional Metadata: Ensure that the metadata you're interested in is being indexed by your Pinecone vector store. This may require adjusting your data ingestion process to include these additional metadata fields.

2. Modify Retrieval Code: Extend or modify the relevant classes (such as SelfQueryRetriever or PineconeTranslator) that interact with Pinecone to extract and return the desired metadata. This could involve changes to methods like getRelevantDocuments to include additional metadata in their responses.

3. Process Metadata in Results: Adjust your code to process the query results to handle and display the additional metadata fields. This involves parsing the results to extract the additional metadata from the Pinecone response.

Ensure that your modifications are aligned with how your application is designed to ingest, retrieve, and display data. For specific implementation details, refer to the relevant parts of your codebase that deal with data retrieval and processing.

For more guidance, you might want to look into the following parts of the LangChain documentation and source code:

• The implementation of data retrieval and processing methods in your application.
• The langchain/src/retrievers/self_query/pinecone.ts and libs/langchain-community/src/vectorstores/pinecone.ts for insights on how to interact with Pinecone.
• The langchain/src/chains/conversational_retrieval_chain.ts for how the conversational retrieval chain is implemented.

This approach will help you access and utilize the full metadata available in Pinecone for your application.

Sources

• langchain/src/retrievers/self_query/pinecone.ts
• libs/langchain-community/src/vectorstores/pinecone.ts
• langchain/src/chains/conversational_retrieval_chain.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.