Searching and storing metadata with the VectorStoreRetrieverMemory memory module

[frankolson]

This is my example code for using a vector store as memory and providing metadata filters:

import { ConversationChain } from "langchain/chains"
import { VectorStoreRetrieverMemory } from "langchain/memory"
import { Chroma } from "langchain/vectorstores/chroma"
import { OpenAIEmbeddings } from "langchain/embeddings/openai"
import { ChatOpenAI } from "langchain/chat_models/openai"
import { PromptTemplate } from "langchain/prompts"

const DEFAULT_TEMPLATE = `The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Relevant pieces of previous conversations (You do not need to use these if not relevant):
{history}

Human: {input}
AI:`

export async function createChatChain(excludedIds: number[]) {
  const llm = new ChatOpenAI({})
  const prompt = new PromptTemplate({
    template: DEFAULT_TEMPLATE,
    inputVariables: ["input", "history"],
  })
  const vectorStore = new Chroma(new OpenAIEmbeddings(), {
    collectionName: "memory",
    url: process.env.CHROMA_URL,
  })
  const filters = excludedIds.length > 0
    ? { "$or": excludedIds.map(id => ({ id: { "$ne": id } })) }
    : {}
  const memory = new VectorStoreRetrieverMemory({
    vectorStoreRetriever: vectorStore.asRetriever(5, filters),
    memoryKey: "history",
    inputKey: "input",
  })
  
  return new ConversationChain({ llm, prompt, memory, verbose: true })
}

In the code I am using excludedIds to filter out certain messages, but it could also be used to scope loops to just those with a specific userId. What I want to do is look up messages with these filters, but then also save the current message with it's own metadata (eg. message IDs, userIDs, etc.). However, the VectorStoreRetrieverMemory will save conversations using the internal saveContext function which doesn't provide a way for adding documents with metadata (source link):

// /langchain/src/memory/vector_store.ts

// ...

  async saveContext(
    inputValues: InputValues,
    outputValues: OutputValues
  ): Promise<void> {
    const text = Object.entries(inputValues)
      .filter(([k]) => k !== this.memoryKey)
      .concat(Object.entries(outputValues))
      .map(([k, v]) => `${k}: ${v}`)
      .join("\n");
    await this.vectorStoreRetriever.addDocuments([
      new Document({ pageContent: text }),
    ]);
  }
 
// ...

How do I use the save metadata while using the VectorStoreRetrieverMemory memory module or prevent it from saving (making it exclusively a lookup module) so that I can implement my own custom saving method?

[dosubot[bot]]

🤖

Hello @frankolson,

I'm here to assist you with your question about the VectorStoreRetrieverMemory module in the langchainjs framework.

To save metadata or prevent the module from saving so you can implement a custom saving method, you can extend the VectorStoreRetrieverMemory class and override the saveContext method. Here's an example:

class CustomVectorStoreRetrieverMemory extends VectorStoreRetrieverMemory {
  async saveContext(inputValues: InputValues, outputValues: OutputValues): Promise<void> {
    // Your custom saving logic here
  }
}

This will allow you to control how the context is saved, including whether or not metadata is saved. If you want to prevent saving entirely, you could leave the saveContext method empty:

class CustomVectorStoreRetrieverMemory extends VectorStoreRetrieverMemory {
  async saveContext(inputValues: InputValues, outputValues: OutputValues): Promise<void> {
    // Do nothing
  }
}

This would effectively prevent the context from being saved.

The metadata that needs to be saved along with the context is structured as a string. This string is constructed by joining the key-value pairs of the input and output values (excluding the memory key), with each pair formatted as key: value and separated by a newline (\n).

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Best,
Dosu

Sources

• langchain/src/vectorstores/base.ts
• examples/src/memory/vector_store.ts
• langchain/src/memory/vector_store.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[frankolson]

Okay yah, your second option of extending the VectorStoreRetrieverMemory class and overriding the saveContext function in order to prevent saving makes sense. But how would overriding the class allow you to save metadata as suggested in your first option? Where would the metadata used to when saving the documents be acquired from? Could you give an example of how this might be implemented assuming I wanted to save the following metadata (the exact values would be dynamic)?:

{ userId: 1234, humanMessageId: 2345, aiMessageId: 3456 }

[MJDeligan]

I'm also missing this feature. As you've shown the filtering can be done on the vectorstore itself, but the storing cannot.

There's really 2 options to have this work:

1. Have the VectorStoreRetrieverMemory accept a metadata param which automatically adds the specified metadata to the documents. (This can be an object or for more complex use cases, a function which receives some context as input)
2. Implement this behaviour on the vectorstores themselves, where an instance property is used and added on top of the metadata provided with the documents. This would of course require every single vector store to implement this, if it wants to benefit from the behaviour. This is maybe more suitable for a quick patch of your integration of choice.

The first options shouldn't take much to implement, but I don't see a way to easily filter results from an existing retriever, so if that should be possible VectorStoreRetrieverMemory, it would probably be morre complex. Maybe @jacoblee93 wants to share his thoughts on this.

[jacoblee93]

We've sort of gone away from these memory classes in favor of just manually manipulating ChatMessageHistory classes since they don't play nicely with LCEL:

https://js.langchain.com/docs/modules/memory/chat_messages/

But if you're up for opening a quick PR I think it could be nice!

[MJDeligan]

I see! Is there an example for an alternative to emulate the vector search using ChatHistories? As far as I can tell we can't really control the retrieval of messages at the query level (using RunnableWithHistory), since the getMessages method never gets passed any argument.

I can certainly open a little PR to implement this behaviour.

[jacoblee93]

It would be something like adding the message to the history and then replacing it with the result of the vector search on the most recent message content - will see if I can get an example together tomorrow.

[MJDeligan]

That would be great, thank you!

[MJDeligan]

Hey @jacoblee93 , did you create the example? I would love to see your approach!