Store on disk?

[richarddd]

Hello,

How can i persist the Embedder and then load from a file for faster access? I don't want to load my entire corpus each time.

Similarly can i do this with SearchEngine?

Thanks!

[Michael-JB]

Hey,

Although there's no built-in way to persist the embedder, the process of 'fitting' a corpus is effectively just computing the avgdl. So, you can construct an embedder once, take note of this avgdl, and then instantiate future embedders with this value directly.

As for persisting the search engine, there's currently no way to do this. Also, it operates entirely in-memory. If your corpus is large enough that persistence is a feature you need, have you considered using a vector database? The embedder from this crate produces sparse embeddings that you can easily store in most vector databases.

-Michael

[Adarsh999bh]

But in order to store data in VectorDB, embedding length should be of fixed size right.? Unlike here, embeddings are generated of different length.

[Michael-JB]

Dense vectors should have the same dimensionality to allow comparison, however this crate generates sparse vectors where the number of elements that are "set" can vary between embeddings. So to store in a vector database, you'd need to use a vector database that supports sparse vectors (e.g., Qdrant).

[Adarsh999bh]

Ok thanks

[Adarsh999bh]

Can I serialize the embedder and store it in a file ?

[Michael-JB]

Hi Adarsh,

Please see my earlier reply on this: #13 (comment)

Constructing an embedder with an avgdl is very cheap.

[Adarsh999bh]

What about SearchEengine .? Can I serialize and store that.? I'm asking because I don't wanna include a seperate vector database just to store sparse vectors !

[Michael-JB]

The Search Engine is not serialisable at the moment per my message above. However, you can persist the documents themselves and reconstruct the search engine. Without storing the embeddings yourself, there's no way to avoid recomputing the embeddings at present. As an optimisation, you could also persist the avgdl and construct the search engine with that (this saves repetition of the fitting process).

An alternative approach which I haven't explored would be to use the in-memory version of Qdrant: a cursory look at their documentation suggests that their in-memory implementation supports loading data from a file.

[Adarsh999bh]

How searching for sparse vectors in Qdrant different than searching using SearchEngine in bm25?
Is scoring mechanism different in bm25 ?

[Adarsh999bh]

Hi @Michael-JB , I just want to know that if searching mechanism in Qdrant is different than searching using SearchEngine in BM25.?

[Adarsh999bh]

Hi @Michael-JB , If I store embeddings in Qdrant, how can I load it back in bm25 .?

[DavZim]

I would also be very interested in this. I maintain the rbm25 R package that uses this crate under the hood (https://github.com/DavZim/rbm25/) and I would love to add functionality that persists a SearchEngine to disk for later use.

I looked through the code and documentation but I couldn't figure out how to extract the avgdl from a fitted SearchEngine or how to instantiate a SearchEngine efficiently with a known avgdl and a given corpus.

[JosiahParry]

Hi there, I am interested in using the bm25 search engine for a lightweight documentation site built with leptos. It would be ideal to be able to build the search engine during build.rs and store that only in the wasm binary instead of all of the documents and then build it at runtime. Having the ability to serialize it would make the binary smaller iiuc and have no cold start cost.

Could there be an optional bincode feature?

[Michael-JB]

Hi all, thanks for sharing this feedback! This wasn't a use-case I expected, but it's cool to hear that you think it would be valuable. I'll take a look at the code in the coming days/weeks and have a think about how best to go about persistence.

[Michael-JB]

Small update here. I've had a think about how best to approach this, and I concluded that making the search engine Serializable+Deserializable via the serde crate would be best. This way, the crate can remain agnostic to the persistence file format (e.g., you could serialize to json or binary). Would that work for your use-cases?

I put together an initial implementation, though I'd like serde to be an optional dependency as I expect most people won't need this feature. Unfortunately, making this optional is not as trivial as I had hoped. See comments in the diff for the details. As a result, I'm not sure where to go from here. To support this feature, either I add serde as a permanent dependency, or I remove the ability to bring-your-own-type for the embedding space (breaking change). I'll think through the options and keep you all updated.