Khoj is reverting to downloading a model even if I add my own llama-cpp hosted server

[DeutscheGabanna]

Server

• Cloud (https://app.khoj.dev)
• Self-Hosted Docker
• Self-Hosted Python package
• Self-Hosted source code

Clients

• Web browser
• Desktop/mobile app
• Obsidian
• Emacs
• WhatsApp

OS

• Windows
• macOS
• Linux
• Android
• iOS

Khoj version

latest

Describe the bug

Khoj tries to download a model from Hugginface with this local configuration, instead of using the local API.

Current Behavior

This is the bielik configuration:

I used admin API key because I had no clue what to put here, since it is a local self-hosted llama-cpp-server.

I confirmed it works by doing:

curl -X POST http://localhost:8080/completion \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Napisz puuuuu",
    "max_tokens": 32,
    "temperature": 0.2
  }'

{"index":0,"content":"uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuu","tokens":[],"id_slot":3,"stop":true,"model":"Bielik-4.5B-v3.0-Instruct.Q8_0.gguf","tokens_predicted":32,"tokens_evaluated":8,"generation_settings":{"seed":4294967295,"temperature":0.20000000298023224,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":40,"top_p":0.949999988079071,"min_p":0.05000000074505806,"top_n_sigma":-1.0,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"dry_multiplier":0.0,"dry_base":1.75,"dry_allowed_length":2,"dry_penalty_last_n":4096,"dry_sequence_breakers":["\n",":","\"","*"],"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"stop":[],"max_tokens":32,"n_predict":32,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"logit_bias":[],"n_probs":0,"min_keep":0,"grammar":"","grammar_lazy":false,"grammar_triggers":[],"preserved_tokens":[],"chat_format":"Content-only","reasoning_format":"deepseek","reasoning_in_content":false,"thinking_forced_open":false,"samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"speculative.n_max":16,"speculative.n_min":0,"speculative.p_min":0.75,"timings_per_token":false,"post_sampling_probs":false,"lora":[]},"prompt":"<s>Napisz puuuuu","has_new_line":false,"truncated":false,"stop_type":"limit","stopping_word":"","tokens_cached":39,"timings":{"cache_n":1,"prompt_n":7,"prompt_ms":275.595,"prompt_per_token_ms":39.37071428571429,"prompt_per_second":25.399589978047494,"predicted_n":32,"predicted_ms":5506.741,"predicted_per_token_ms":172.08565625,"predicted_per_second":5.8110595722588005}}%   

Yet Khoj tries to download the online model from HF as evident from the logs:

ValueError: No file found in                         
[server]   |                            speakleash/Bielik-4.5B-v3.0-Instruct-                
[server]   |                            GGUF that match *Q4_K_M.gguf 

If I save the same local model with a different name, without the slash /, it also doesn't work with a different error.

ValueError: not enough values to                     
[server]   |                            unpack (expected 2, got 1)     

I tried again with this name, since it is a perfect match: speakleash/Bielik-4.5B-v3.0-Instruct.Q8_0.gguf

FileNotFoundError:                                   
[server]   |                            speakleash/Bielik-4.5B-v3.0-Instruct.                
[server]   |                            Q8_0.gguf (repository not found)  

Expected Behavior

Khoj should use the localhost:8080 as API instead of downloading the HF model.

I followed the manual very closely:

Install any Openai API compatible local ai model server like llama-cpp-server, Ollama, vLLM etc.
Add an ai model api on the admin panel
Set the api url field to the url of your local ai model provider like http://localhost:11434/v1/ for Ollama
Restart the Khoj server to load models available on your local ai model provider
If that doesn't work, you'll need to manually add available chat model in the admin panel.
Set the newly added chat model as your preferred model in your User chat settings
Start chatting with your local AI!

Reproduction Steps

Install on arch using podman compose up. Run a separate llama-cpp-server instance (I installed this through AUR) using model https://huggingface.co/speakleash/Bielik-4.5B-v3.0-Instruct-GGUF - llama-cpp-server -m local_path_model.gguf. Copy the open port and add the model to the Khoj configuration.

Possible Workaround

No response

Additional Information

No response

Link to Discord or Github discussion

No response

[DeutscheGabanna]

Initially I thought maybe my URL was not right. After all, I entered localhost:8080 in the Khoj config, but I tested localhost:8080/completion in my curl test.

So I tried this URL too - localhost:8080/v1/models which responds positively:

{"models":[{"name":"Bielik-4.5B-v3.0-Instruct.Q8_0.gguf","model":"Bielik-4.5B-v3.0-Instruct.Q8_0.gguf","modified_at":"","size":"","digest":"","type":"model","description":"","tags":[""],"capabilities":["completion"],"parameters":"","details":{"parent_model":"","format":"gguf","family":"","families":[""],"parameter_size":"","quantization_level":""}}],"object":"list","data":[{"id":"Bielik-4.5B-v3.0-Instruct.Q8_0.gguf","object":"model","created":1770489078,"owned_by":"llamacpp","meta":{"vocab_type":1,"n_vocab":32000,"n_ctx_train":8192,"n_embd":2048,"n_params":4757260288,"size":5060370432}}]}

But the error is still suggesting Khoj tries to download the model straight from HF:

FileNotFoundError:                                   
[server]   |                            speakleash/Bielik-4.5B-v3.0-Instruct.                
[server]   |                            Q8_0.gguf (repository not found)   

[DeutscheGabanna]

Ah, just in case you bring up the documentation that says to use localhost:port/v1/. I tested that too, and there's a 404 HTTP response.

[yang1002378395-cmyk]

分析

这个问题是因为 ChatModel API 和 SearchModelConfig embeddings 是独立配置的。

你配置的 localhost:8080 是 ChatModel 的 API，但 bi_encoder 字段（speakleash/Bielik-4.5B-v3.0-Instruct）是用于搜索/索引的 embeddings 模型。

问题根因

1. bi_encoder 字段含有 /，被 SentenceTransformer 识别为 HuggingFace repo
2. 当 embeddings_inference_endpoint 未配置时，Khoj 会尝试本地下载模型
3. 这是两个独立的系统：Chat（对话）和 Search（搜索索引）

解决方案

有两个选择：

选项 1：使用远程 embeddings API
在 Admin Panel 配置 embeddings_inference_endpoint，指向支持 embeddings 的服务（如 OpenAI、本地 TEI 等）

选项 2：使用本地 embeddings 模型
确保 bi_encoder 是有效的本地路径或 HuggingFace repo（需要网络访问 HF）

相关代码

• src/khoj/processor/embeddings.py:41-43 - 当无 endpoint 时下载模型
• src/khoj/database/models/__init__.py:558 - bi_encoder 字段定义

建议

这不是 bug，但文档可以更清晰地说明 Chat 和 Search 配置的独立性。如果你需要进一步帮助配置，请提供更多关于你的 embeddings 需求的信息。

[debanjum]

@DeutscheGabanna, Based on the logs shared mentioning speakleash/Bielik-4.5B-v3.0-Instruct... Khoj seems to try download the model you specified from HuggingFace to use its tokenizer. The tokenizer is used to count tokens for context truncation. Using the model specific tokenizer gives more accurate token counting.

This is by design, though maybe it can be improved.