Faiss dont update the number of embeddings

[ghost]

Hi im trying to learn how to use Faiss and Generative QA

im saving the data like that based on the tutorial12

from haystack.document_stores import FAISSDocumentStore
from haystack.nodes import DensePassageRetriever
from haystack.utils import convert_files_to_docs, fetch_archive_from_http, clean_wiki_text

document_store = FAISSDocumentStore(embedding_dim=128, faiss_index_factory_str="Flat")

# Let's first get some files that we want to use
doc_dir = "data/tutorial12"
s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt12.zip"
fetch_archive_from_http(url=s3_url, output_dir=doc_dir)

# Convert files to dicts
docs = convert_files_to_docs(dir_path=doc_dir, clean_func=clean_wiki_text, split_paragraphs=True)

# Now, let's write the dicts containing documents to our DB.
document_store.write_documents(docs)

retriever = DensePassageRetriever(
    document_store=document_store,
    query_embedding_model="vblagoje/dpr-question_encoder-single-lfqa-wiki",
    passage_embedding_model="vblagoje/dpr-ctx_encoder-single-lfqa-wiki",
)

document_store.update_embeddings(retriever=retriever)

document_store.save("data/faiss_data.faiss", config_path="data/faiss_config")
retriever.save("data/retriever.pt")

But if i run it twice it give me this error

ValueError: The number of documents present in the SQL database (2358) does not match the number of embeddings in FAISS (0). Make sure your FAISS configuration file correctly points to the same database that was used when creating the original index

i dont know how to resolve it and if i add new doc in the future i will need start from scratch and it will take too much time.
Thanks for your reply.

[julian-risch]

Hi @azetoy did you have a look at this related and resolved issue already? #1903

[ghost]

@julian-risch thanks for your help it helped a lot i have one more question in faiss if i add a docs and write it in document store will it reload from the start or just update the last document ?

[julian-risch]

Great to hear that! You would need to use document_store.update_embeddings(retriever=retriever, update_existing_embeddings=False) to prevent the document store from calculating all embeddings from the start and instead only calculate the embeddings for the newly added documents.

[sonal-511]

hi @julian-risch ,
I ran the faiss document store to create embeddings for 1M documents first, afterwards some new documents came up hence I had to update the documents in the store so I used the doc_store.update_embeddings(retriever ,update_existsing_embeddings = False) but this processes stopped in between.I also tried running the update embedding again but it runs successfully without doing anything. when I try loading the doc store by giving index and config file it says the number of documents in sql does not match with faiss index , sql database contains the updated number of documents but the faiss index still contains the old number of documents.

[anakin87]

Similar Discord thread: https://discord.com/channels/993534733298450452/1052575068972142602