Don't bundle all responses together in llama-server in the logs

[simonw]

This will need a change to LLM core too. It's not great that loading different models still results in everything ending up in the same llama-server model ID in the logs database.

[simonw]

I decided to see what endpoints are available on the server:

cd /tmp
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp/tools/server
files-to-prompt . -e cpp -c | llm -m gemini-2.5-flash-preview-04-17 \
  -s 'describe all HTTP endpoints'

This did a great job: https://gist.github.com/simonw/35fc90b5d0cae9f868d96a41c389d681

1. GET /

   • Method: GET
   • Description: Serves the static web UI files from the directory specified by --public-path. If --public-path is not set, it serves the embedded index.html.gz (requiring gzip support in the client browser). Includes headers for COEP and COOP for compatibility with things like Pyodide.
   • Requires: Web UI enabled (--webui).

2. GET /health

   • Method: GET
   • Description: Returns a simple JSON response {"status": "ok"} if the server is running and the model is loaded. If the server is still loading the model, it returns a 503 Service Unavailable error, potentially with a loading HTML page depending on the Accept header.
   • Authentication: Public endpoint, does not require an API key.

3. GET /metrics

   • Method: GET
   • Description: Provides server metrics in Prometheus text format (text/plain). Includes various counters and gauges related to token processing, KV cache usage, and request handling.
   • Requires: Metrics endpoint enabled (--metrics).

4. GET /props

   • Method: GET
   • Description: Returns a JSON object describing various server properties, including default generation settings, total number of slots, model path, supported modalities (vision), chat template source, BOS/EOS tokens, and build information.
   • Authentication: Public endpoint, does not require an API key.

5. POST /props

   • Method: POST
   • Description: Allows changing certain global server properties. The code snippet shows this handler exists but doesn't fully implement extensive property changes, returning {"success": true}.
   • Requires: Props endpoint enabled (--props).

6. POST /api/show

   • Method: POST
   • Description: Returns information about the model template and context length.

7. GET /models

   • Method: GET
   • Description: Returns a JSON list of available models. Currently lists the main loaded model with basic metadata (vocab size, context, embeddings, params, size) in an OAI-compatible format.
   • Authentication: Public endpoint, does not require an API key.

8. GET /v1/models

   • Method: GET
   • Description: Alias for /models, providing an OAI-compatible endpoint for listing models.
   • Authentication: Public endpoint, does not require an API key.

9. GET /lora-adapters

   • Method: GET
   • Description: Lists the LoRA adapters currently loaded by the server, including their ID, path, and scale.

10. POST /lora-adapters

    • Method: POST
    • Description: Allows applying new LoRA adapters to the server contexts. The request body is expected to be a JSON array specifying the adapters to load (e.g., [{"path": "lora1.bin", "scale": 1.0}, {"path": "lora2.bin", "scale": 0.5}]). This is a hot-swap mechanism.

11. GET /slots

    • Method: GET
    • Description: Returns detailed JSON data about all configured inference slots, including their ID, associated task, current state, parameters, prompt, and next token prediction status. Can optionally accept a fail_on_no_slot query parameter which causes it to return a 503 if no idle slots are available.
    • Requires: Slots endpoint enabled (--slots).

12. POST /slots/:id_slot

    • Method: POST
    • Description: Performs an action on a specific slot identified by :id_slot in the URL path. The action is specified by the action query parameter (?action=save, ?action=restore, ?action=erase).
      • ?action=save: Saves the state of the specified slot to a file. Requires a filename in the JSON request body.
      • ?action=restore: Restores the state of the specified slot from a file. Requires a filename in the JSON request body.
      • ?action=erase: Erases the token cache and state for the specified slot.
    • Requires: Slot save path configured (--slot-save-path).

Inference Endpoints:

13. POST /completion (Legacy)

    • Method: POST
    • Description: Handles text completion requests. Accepts a JSON body with prompt and generation parameters. Returns the completed text. Can handle streaming via the stream parameter. Uses a custom, non-OAI compatible response format.

14. POST /completions

    • Method: POST
    • Description: Alias for /completion. Handles text completion requests with a custom, non-OAI compatible response format.

15. POST /v1/completions

    • Method: POST
    • Description: Handles text completion requests in an OpenAI-compatible format. Accepts JSON body with prompt and standard OAI completion parameters (model, prompt, max_tokens, temperature, stream, logprobs, etc.). Can handle streaming.

16. POST /chat/completions

    • Method: POST
    • Description: Handles chat completion requests. Accepts a JSON body typically containing a list of messages and generation parameters. Applies a chat template to format the prompt. Can handle streaming. Uses a custom, non-OAI compatible chat response format. Can process multimodal inputs if --mmproj is used.

17. POST /v1/chat/completions

    • Method: POST
    • Description: Handles chat completion requests in an OpenAI-compatible format. Accepts JSON body with messages and standard OAI chat parameters (model, messages, max_tokens, temperature, stream, logprobs, tool_choice, tools etc.). Applies a chat template. Can handle streaming. Can process multimodal inputs if --mmproj is used.

18. POST /infill

    • Method: POST
    • Description: Handles text infill requests (Fill-in-the-Middle). Requires the model to have FIM tokens (--vocab-type spm). Accepts JSON body with input_prefix and input_suffix, optionally prompt and input_extra.

19. POST /embedding (Legacy)

    • Method: POST
    • Description: Handles text embedding requests. Accepts JSON body with text input (content). Returns the computed embeddings. Uses a custom, non-OAI compatible response format.
    • Requires: Embedding mode enabled (--embedding).

20. POST /embeddings

    • Method: POST
    • Description: Alias for /embedding. Handles text embedding requests with a custom, non-OAI compatible response format.
    • Requires: Embedding mode enabled (--embedding).

21. POST /v1/embeddings

    • Method: POST
    • Description: Handles text embedding requests in an OpenAI-compatible format. Accepts JSON body with text input (input) and optional encoding_format (float or base64). Returns the computed embeddings.
    • Requires: Embedding mode enabled (--embedding) and a pooling type other than 'none'.

22. POST /rerank

    • Method: POST
    • Description: Handles reranking requests. Accepts JSON body with a query string and a list of documents (or texts for TEI compatibility). Reranks the documents based on relevance to the query using the model.
    • Requires: Reranking mode enabled (--reranking) and embedding mode disabled (--no-embedding).

23. POST /reranking

    • Method: POST
    • Description: Alias for /rerank. Handles reranking requests.
    • Requires: Reranking mode enabled (--reranking) and embedding mode disabled (--no-embedding).

24. POST /v1/rerank

    • Method: POST
    • Description: Alias for /rerank. Handles reranking requests, potentially with support for TEI or Jina API formats based on input fields.
    • Requires: Reranking mode enabled (--reranking) and embedding mode disabled (--no-embedding).

25. POST /v1/reranking

    • Method: POST
    • Description: Alias for /v1/rerank. Handles reranking requests.
    • Requires: Reranking mode enabled (--reranking) and embedding mode disabled (--no-embedding).

26. POST /tokenize

    • Method: POST
    • Description: Tokenizes the input content (string) from the JSON request body. Accepts optional add_special (bool) and with_pieces (bool) parameters. Returns a JSON array of token IDs, or objects including IDs and piece strings if with_pieces is true.

27. POST /detokenize

    • Method: POST
    • Description: Detokenizes a list of tokens (array of integers) from the JSON request body. Returns the resulting text string in a JSON object.

[simonw]

That turned out not to be necessary: the README documents all of this stuff here https://github.com/ggml-org/llama.cpp/tree/master/tools/server#api-endpoints

[simonw]

I dumped out the JSON context to see if there is anything useful in there but sadly there is not. This was while running:

llama-server --jinja -hf unsloth/gemma-3-12b-it-qat-GGUF:Q4_K_M

In no-streaming mode (because I was testing tools):

{'choices': [{'finish_reason': 'stop',
              'index': 0,
              'logprobs': None,
              'message': {'annotations': None,
                          'audio': None,
                          'content': '```json\n'
                                     '{\n'
                                     '  "local_time": '
                                     '"2024-05-03T19:12:48-07:00",\n'
                                     '  "utc_time": "2024-05-04T02:12:48Z"\n'
                                     '}\n'
                                     '```',
                          'function_call': None,
                          'refusal': None,
                          'role': 'assistant',
                          'tool_calls': None}}],
 'created': 1748414034,
 'id': 'chatcmpl-rNakMWTjcZrx8PlVts8oy0KHmUwKtum6',
 'model': 'llama-server',
 'object': 'chat.completion',
 'service_tier': None,
 'system_fingerprint': 'b5410-3e0be1ca',
 'timings': {'predicted_ms': 2483.352,
             'predicted_n': 84,
             'predicted_per_second': 33.82524909879872,
             'predicted_per_token_ms': 29.563714285714283,
             'prompt_ms': 767.649,
             'prompt_n': 209,
             'prompt_per_second': 272.25984792528874,
             'prompt_per_token_ms': 3.672961722488038},
 'usage': {'completion_tokens': 84,
           'completion_tokens_details': None,
           'prompt_tokens': 209,
           'prompt_tokens_details': None,
           'total_tokens': 293}}

[simonw]

Hitting http://localhost:8080/models gives me something useful:

{
  "object": "list",
  "data": [
    {
      "id": "/Users/simon/Library/Caches/llama.cpp/unsloth_gemma-3-12b-it-qat-GGUF_gemma-3-12b-it-qat-Q4_K_M.gguf",
      "object": "model",
      "created": 1748414223,
      "owned_by": "llamacpp",
      "meta": {
        "vocab_type": 1,
        "n_vocab": 262208,
        "n_ctx_train": 131072,
        "n_embd": 3840,
        "n_params": 11766034176,
        "size": 7294226304
      }
    }
  ]
}

I could use the filename of that id field as a model ID.

[simonw]

OK, I have a fix in a branch but it can't land until there's an alpha out with the new LLM feature for it to test against.

I manually tested it and it works:

% llm -m llama-server hello        
Hello there! How can I help you today? 😊

% llm -m llama-server hello --async
Hello there! How can I help you today? 😊
% llm logs -n 2
# 2025-05-28T14:42:52    conversation: 01jwbnh71vjvcta5fqsfyma79c id: 01jwbnh6jqnvfssnxtk9j0w8rf

Model: **llama-server** (resolved: **unsloth_gemma-3-12b-it-qat-GGUF_gemma-3-12b-it-qat-Q4_K_M.gguf**)

## Prompt

hello

## Response

Hello there! How can I help you today? 😊


# 2025-05-28T14:42:55    conversation: 01jwbnha7ep7xjk9gqm4efkjpm id: 01jwbnh9ryxnq512ts45g7av9z

Model: **llama-server** (resolved: **unsloth_gemma-3-12b-it-qat-GGUF_gemma-3-12b-it-qat-Q4_K_M.gguf**)

## Prompt

hello

## Response

Hello there! How can I help you today? 😊