I am trying (and failing) to make olla work with groq

[Sophist-UK]

proxy:
  engine: "sherpa"
  profile: "auto"
  stream_buffer_size: 2048


logging:
  level: "debug"
  format: "text"
  output: "stdout"

discovery:
  model_discovery:
    enabled: true
    interval: 10m
    timeout: 30s
    concurrent_workers: 5

  static:
    endpoints:
      - name: ollama
        type: ollama
        url: "http://ollama:11434"
        model_url: "/api/tags"
        health_check_url: "/"
        check_interval: 300s
        check_timeout: 5s

      - name: groq
        type: openai-compatible
        url: "https://api.groq.com/openai/v1"
        preserve_path: true
        model_url: "/models"
        health_check_url: "/models"
        check_interval: 300s
        check_timeout: 5s
        headers:
          Authorization: "Bearer ${GROQ_API_KEY}"

The ollama endpoint is registering just fine, but the groq endpoint is erroring out.

Here are the olla log entries:

{
  "timestamp":"2026-02-10 16:46:55",
  "level":"WARN",
  "msg":"Endpoint discovered offline: groq",
  "status":"unhealthy",
  "latency":209037080,
  "next_check_in":300000000000
}
{
  "timestamp":"2026-02-10 16:46:55",
  "level":"WARN",
  "msg":"Endpoint discovered offline:",
  "endpoint_name":"groq",
  "status":"unhealthy",
  "latency":209037080,
  "next_check_in":300000000000,
  "endpoint_url":"https://api.groq.com/openai/v1",
  "status_code":401,
  "error_type":0
}
{
  "timestamp":"2026-02-10 16:46:55",
  "level":"INFO",
  "msg":"Endpoint registered",
  "name":"groq",
  "url":"https://api.groq.com/openai/v1",
  "priority":0
}

Here is the result of /internal/status/endpoints:

{
  "timestamp":"2026-02-10T17:05:24.781543945Z",
  "endpoints": [
    {
      "name":"ollama",
      "type":"ollama",
      "status":"healthy",
      "last_model_sync":"8m ago",
      "health_check":"3m ago",
      "response_time":"12ms",
      "success_rate":"N/A",
      "priority":0,
      "model_count":1,
      "request_count":0
    },
    {
      "name":"groq",
      "type":"openai-compatible",
      "status":"unhealthy",
      "health_check":"59s ago",
      "response_time":"1.8s",
      "success_rate":"N/A",
      "issues":"unavailable",
      "priority":0,
      "model_count":0,
      "request_count":0
    }
  ],
  "total_count":2,
  "healthy_count":1,
  "routable_count":1
}

[thushan]

Hey,

Olla is designed for local backends and not remote ones (like supporting OpenAI or Groq etc). This was one of the original design goals. By default we have mechanisms to strip auth headers etc so we don't leak any keys etc too.

We could add this feature but it will take some time to mature and will have to be in beta for a while.

If you open an Issue and write down your specific requirements and potential ways users would be able to use it, we'll look into it (or another user can).

[Sophist-UK]

Ah - looks like I should switch to LiteLLM instead - the olla documentation does say "Use LiteLLM when integrating multiple cloud providers".

But then again, the documentation also says to use LiteLLM to support translation of APIs e.g. between openai and anthropic yet I believe that olla now does some of that.

However thanks for considering and answering - and I wouldn't want to push you into something that is outside the primary scope and leads to bloat.

[thushan]

LiteLLM is the best approach, we have a few folks using it for OpenAI and Bedrock. The work required to get stability is quite high and as you'd see from the LiteLLM codebase, it was quite a challenge to support everything.

Good point, we have to update the documentation. Also in the works is to route directly to those that support Anthropic endpoints natively (instead of Olla doing it - like vllm and lately Ollama).