Prefixes for embedding models (possibly also others)

[schubidoo]

Issue Description

Some embedding models require prefixes for using them, particularly Nomic (which is one of the models on the embedding model list in aichat). For other models it is recommended to use them. However, when setting Nomic as the embedder, no such prefixes are sent, at least according to the --loglevel debug output. It seems that only the chunks are sent for embedding, and only the query for requests later, both without the prefix.

Nomic expects:

"search_document:" as prefix during the creation of embedding vectors, and
"search_query:" during the retrieval process

It also supports "clustering" and "classification" as prefixes.

Not sure how critical this is, but in the documentation on huggingface (https://huggingface.co/nomic-ai/nomic-embed-text-v1.5), it says:

Important: the text prompt must include a task instruction prefix, instructing the model which task is being performed.

For example, if you are implementing a RAG application, you embed your documents as search_document: and embed your user queries as search_query: .

I classified this as a bug based on that description, but it is certainly not a critically breaking one...

[splitbrain]

Are you using those models with the ollama interface? As far as I understand, Ollama will automatically handle this prefixing when used with the OpenAI compatible interface. Which makes sense since OpenAI does not require any prefixing for their models either.

[schubidoo]

That could be. In this specific case, the machine that I can use for that model to embed is interfaced with vLLM (but through OpenAI compatible interface).

Although this would be a bit surprising - Not sure how the interface would be able to distinguish between an embedding request and a search request in order to decide which prefix to add? The API call itself is the same after all, as far as I understand.

[splitbrain]

Hmm, that's a good point.

The Ollama documentation does not mention anything about prefixes on their model page: https://ollama.com/library/nomic-embed-text:v1.5 Their samples do not use any prefixes either. But that of course doesn't mean that one shouldn't use them.

I couldn't find any definite answer to that. I guess it would be interesting to see the difference in how the model performs with and without the prefixes for similarity searches.

[splitbrain]

This video seems to answer the question: https://www.youtube.com/watch?v=76EIC_RaDNw

Prefixes seem to mattter and need to be passed, even when using an OpenAI-compatible API. I guess it would make most sense to have the prefixes defined in the models.json file

[splitbrain]

This is now implemented in the early access version available to our partners