How to use Ollama with this package?

[AbrJA]

Hi @svilupp I hope you are doing well,

Thank you for all the work you did with this package!

I just have a quick question, I'm trying to hack the interface to integrate it with a vector database such as Qdrant. But I want to use Ollama embedding model, is it possible? I saw a parameter called model in some functions get_embeddings but I think it is related to OpenAI models because I cannot pass an schema.

If you can give me some hint to achieve it, I will appreciate it.

Greetings,

[svilupp]

Hi there,

The easiest way is to:

1, register a model in PromptingTools so it knows which schema it belongs to (eg, https://siml.earth/PromptingTools.jl/dev/examples/working_with_ollama#Schema-Changes-/-Custom-models)

2, either change your embedding model everywhere (PT.MODEL_EMBEDDING="your new alias")
or selectively pass it as a "model" kwarg to get embeddings (see the interface docs how send kwargs to some pipeline step, eg, https://juliagenai.github.io/RAGTools.jl/stable/interface/#Passing-Keyword-Arguments )

but remember you must use the same model to prepare your index and then to embed incoming queries in production

Paste your minimal snippet here if you can't figure it out!

[AbrJA]

Thank you very much, I was missing the first step

[KajWiik]

Along similar lines: I have a problem setting Groq as the model provider. I succeeded running

julia> ai"What is the capital of France?"
[ Info: Tokens: 56 @ Cost: $0.0 in 1.6 seconds
AIMessage("The capital of France is Paris.")

without specifying a model by using PT.set_preferences!() but still

index = build_index(sentences);
ERROR: ArgumentError: api_key cannot be empty

I guess there are more steps involved?

[svilupp]

I'd suggest the same:

1, make sure your model is registered with the right schema (groq in this case, make sure you register both something for chatting and something for embeddings!)

2, either change the model used directly or via kwargs arguments

Remember, there are two important model names:
PT.MODEL_CHAT
PT.MODEL_EMBEDDING

They each handle different model type / different workload.
A, Check what they are for you
B, Make sure you set up their API keys etc.

Specifically, you fail on "build_index" - only AI in there is embeddings, so it tells me that you don't have API key set for openAI default model (which you are most likely using). So set that up first (if Groq has any or add the OpenAI key, which is simplest)

Your Groq model is relevant only later for "generation" of the answer from the relevant document chunks (see the docs for pipeline stages)

[KajWiik]

Aha, no I do not have OpenAI key, I did not understand that these are totally separate processes (I am a newcomer to AI...). After sending the question, I noticed that most probably Groq does not support embedding so I think I have to get an OpenAI key or use local Ollama.

Many thanks for quick answer and very useful packages!