Help retrieving FAISS index and documents

[gamcoh]

Hi,

I saved my index using this command on a Google Colab:

document_store.save("/content/drive/MyDrive/index.faiss")

Now when I reconnected to the Colab, I lost the sqlite db and I couldn't retrieve the index:

ValueError: The number of documents in the SQL database (0) doesn't match the number of embeddings in FAISS (125653). Make sure your FAISS configuration file points to the same database that you used when you saved the original index.

I have the docs it's not a problem, the issue is that I don't want to re-embed the docs because I use OpenAI and it costs.

Is there a way to retrieve the index and use it?

[anakin87]

Hello!
In my opinion, you should have kept the .db (and .json) files as well. I don't know if you can do anything now to recover them...

[gamcoh]

I have the json.

From my understanding, in the db there's only the documents and their metadata, isn't there a way to re-create this using the documents directly? And then apply their embedding using the faiss index?

[sfx101]

@gamcoh did you try this already?

# Load the saved index into a new DocumentStore instance:
new_document_store = FAISSDocumentStore(faiss_index_path="my_faiss_index.faiss")

# or use the `load()` method to create a new DocumentStore instance: 
new_document_store = FAISSDocumentStore.load(index_path="my_faiss_index.faiss")

# Also, provide `config_path` parameter if you set it when calling the `save()` method: 
new_document_store = FAISSDocumentStore.load(index_path="data/my_index.faiss", config_path="data/my_config.json")

[gamcoh]

Yes I did, that's what gotten me the error

[mgardner-chegg]

I am experiencing exactly the same issue. @gamcoh did you resolve it?

[gamcoh]

I didn't unfortunately