How to create a method to remove document from FaissStore?

[DennisMinn]

I want to create a method that takes in the id and deletes the respective Document within the FaissStore.

// Before
Map(2) {
  '8fc00bbd-0d8f-4713-a1b7-02f800ae311e' => { pageContent: 'Hello World' },
  '3cb586d1-ce66-469a-ba28-344766ecc905' => { pageContent: 'foobar' }
}

// Delete record
myFaissStore.removeDocument('8fc00bbd-0d8f-4713-a1b7-02f800ae311e');

// After
Map(2) {
  '3cb586d1-ce66-469a-ba28-344766ecc905' => { pageContent: 'foobar' }
}

What I've tried

I can manually do this by calling myFaissStore.docstore._docs.delete('8fc00bbd-0d8f-4713-a1b7-02f800ae311e'); but the vector store still thinks the information is present. When I call vectorStore.similaritySearch("Hello World", 1); I get the following error.

/home/dminn/projects/stream-guard-bot/node_modules/langchain/dist/stores/doc/in_memory.cjs:50
            throw new Error(`ID ${search} not found.`);
                  ^

Error: ID 29e2508e-eda6-4cfe-b1b7-84a36b46b12a not found.
    at SynchronousInMemoryDocstore.search (/home/dminn/projects/stream-guard-bot/node_modules/langchain/dist/stores/doc/in_memory.cjs:50:19)
    at /home/dminn/projects/stream-guard-bot/node_modules/langchain/dist/vectorstores/faiss.cjs:128:35
    at Array.map (<anonymous>)
    at FaissStore.similaritySearchVectorWithScore (/home/dminn/projects/stream-guard-bot/node_modules/langchain/dist/vectorstores/faiss.cjs:126:30)
    at FaissStore.similaritySearch (/home/dminn/projects/stream-guard-bot/node_modules/langchain/dist/vectorstores/base.cjs:95:36)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[dosubot[bot]]

🤖

Hello,

From your description, it seems like you're trying to remove a document from the FaissStore and also update the vector store accordingly. Currently, the LangChain library does not provide a built-in method to remove a document from the FaissStore.

However, you can extend the FaissStore class and add a removeDocument method. This method should not only remove the document from the docstore but also update the _index and _mapping properties of the FaissStore to reflect the removal of the document.

Here's a rough idea of how you might implement this:

class MyFaissStore extends FaissStore {
  removeDocument(id: string) {
    // Find the index of the document in the _mapping
    const index = Object.values(this._mapping).indexOf(id);
    if (index === -1) {
      throw new Error(`Document with ID ${id} not found.`);
    }

    // Remove the document from the docstore
    this.docstore._docs.delete(id);

    // Remove the document from the _mapping
    delete this._mapping[index];

    // Rebuild the _index without the removed document
    const vectors = Object.values(this._mapping).map(id => this.docstore._docs.get(id).vector);
    this._index = new faiss.IndexFlatL2(vectors[0].length);
    this._index.add(vectors);

    // Update the _mapping to reflect the new _index
    this._mapping = Object.values(this._mapping).reduce((mapping, id, i) => {
      mapping[i] = id;
      return mapping;
    }, {});
  }
}

Please note that this is a simplified example and might not cover all edge cases. For example, it assumes that each document in the docstore has a vector property, which might not be the case in your application. You might need to adjust this code to fit your specific needs.

Also, please be aware that this operation can be expensive in terms of time and memory, especially if you have a large number of documents in your FaissStore, because it involves rebuilding the _index from scratch.

I hope this helps! Let me know if you have any other questions.

Sources

• langchain/src/vectorstores/faiss.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.