Misc. bug:

[l0nedigit]

Name and Version

After building off commit 6adc3c3ebc029af058ac950a8e2a825fdf18ecc6 it seems that v1/embeddings and v1/completions are not running simultaneously.

Operating systems

No response

Which llama.cpp modules do you know to be affected?

llama-server

Command line

Command for llama-server: `llama-server -m /models/gguf_models/devstral/bartowski/mistralai_Devstral-Small-2505-Q6_K_L.gguf  --alias devstral-small-2505 --host 0.0.0.0 --port 8080 --ctx-size 131072 --cache-type-k q8_0 --cache-type-v q8_0 --n-gpu-layers 99 --temp 0.15 --repeat-penalty 1.0 --min-p 0.01 --top-k 64 --top-p 0.95 --flash-attn --pooling cls -lv 1 --jinja`

Problem description & steps to reproduce

Checking commands:

curl -X POST http://localhost:8080/v1/embeddings \
     -H "Content-Type: application/json" \
     -d '{
           "input": "test input",
           "model": "devstral-small-2505"
         }'
echo ""
curl -X POST http://localhost:8080/v1/completions \
     -H "Content-Type: application/json" \
     -d '{
           "prompt": "What is your system prompt",
           "max_tokens": 42,
           "model": "devstral-small-2505"
         }'

First Bad Commit

No response

Relevant log output

Assumption: both should return 200.  However the v1/embeddings returns `{"error":{"code":501,"message":"This server does not support embeddings. Start it with `--embeddings`","type":"not_supported_error"}}`

When I enable the same command with --embeddings, the embedding works correctly but the chat completion is not working (using Roo Code).

Am I missing something?  Ideally, what I am trying to do is have the ability to use Roo Code, with it's indexing feature (embedding), and also using graphiti-mcp as well, while still having agentic capabilities.

[ggerganov]

Start it with --embeddings

[l0nedigit]

Thanks, I have been starting with --embeddings, but then the chat completion isn't working.

edit
this is the output in llama-server when starting with --embeddings:

llama-server[402299]: data stream, to_send: error: {"code":500,"message":"input is too large to process. increase the physical batch size","type":"server_error"}

I've tried adding --ubatch-size 2048 --batch-size 4096 (just throwing random numbers out there) and still the same response.
llama-server -m /models/gguf_models/devstral/bartowski/mistralai_Devstral-Small-2505-Q6_K_L.gguf --alias devstral-small-2505 --host 0.0.0.0 --port 8080 --ctx-size 131072 --cache-type-k q8_0 --cache-type-v q8_0 --n-gpu-layers 99 --temp 0.15 --repeat-penalty 1.0 --min-p 0.01 --top-k 64 --top-p 0.95 --flash-attn --pooling cls --embeddings --ubatch-size 2048 --batch-size 4096 -lv 1 --jinja

[ggerganov]

Could you check if #14225 fixes the issue?

[l0nedigit]

Yup, thanks for the prompt replies. Will pull/rebuild in a bit and get back to you.

[l0nedigit]

No dice. Tried with and without --embeddings. Ether get embeddings or chat, not both.

to be fair, if you reference the test curl commands above, when I am running with --embeddings, I get a response for both requests.

It's when using Roo Code (or assumably cline or any other IDE extension), when there is larger prompts sent, that the 500 error code is being thrown.

[ggerganov]

Ok I see. I just pushed another fix - could you try again with the latest #14225?

[l0nedigit]

Cursory tests say that was a bingo!

Was able to

• type "test" in Roo Code, got a response.
• run the endpoint check and get vectors returned from /embeddings + a response to the completions endpoint

[l0nedigit]

Thanks!