2.3.5 Satellite: LiteLLM

LiteLLM

Handle: litellm
URL: http://localhost:33841/

LLM API Proxy/Gateway.

LiteLLM is very useful for setups where the target LLM backend is either:

• Not supported by Harbor directly
• Doesn't have an OpenAI-compatible API that can be plugged into Open WebUI directly (for example, text-generation-inference)

litellm is also a way to use API-based LLM providers with Harbor.

Starting

# [Optional] Pull the litellm images
# ahead of starting the service
harbor pull litellm

# Start the service
harbor up litellm

You'll likely want to start it with at least one of the compatible LLM backends, or pointing to an external API ahead of time, see below.

Harbor's LiteLLM service is configured with a DB and an API key, so that you can access LiteLLM UI. The UI is available on the /ui endpoint of the service.

# Open LiteLLM API docs
harbor open litellm
# Open LiteLLM UI directly
harbor litellm ui

You can login with the default credentials: admin / admin. To adjust, use the harbor CLI:

# Set the new credentials
harbor litellm username paole
harbor litellm password $(tr -dc 'A-Za-z0-9!?%=' < /dev/urandom | head -c 10)

Configuration

Harbor runs LiteLLM in the proxy mode. In order to configure it, you'll need to edit services/litellm/litellm.config.yaml according to the documentation.

For example:

model_list:
  # What LiteLLM client will see
  - model_name: sllama
    litellm_params:
      # What LiteLLM will send to downstream API
      model: huggingface/repo/model
      # This can be pointed to one of the compatible Harbor
      # backends or to the external API compatible with the LiteLLM
      api_base: http://tgi:80
  - model_name: llamaster
    litellm_params:
      model: bedrock/meta.llama3-1-405b-instruct-v1:0
      aws_region_name: us-west-2

When run together with certain Harbor backends, LiteLLM picks up a matching config fragment automatically (merged at startup from services/litellm/litellm.<service>.yaml): dmr, mlx, omlx, npcsh, tgi, vllm, optillm, langfuse, and llamacpp. For llamacpp the fragment uses wildcard routing — the llama.cpp router discovers models dynamically, so request them through LiteLLM as llamacpp/<router model id>:

harbor up llamacpp litellm
curl "$(harbor url litellm)/v1/chat/completions" \
  -H "Authorization: Bearer $(harbor config get litellm.master.key)" \
  -H 'Content-Type: application/json' \
  -d '{"model":"llamacpp/<router model id>","messages":[{"role":"user","content":"Hello"}]}'

The optillm fragment uses the same wildcard scheme — request optillm/<model> and LiteLLM forwards <model> to optillm (which passes it through to its own backend; approach prefixes like none-<model> work too):

harbor up ollama optillm litellm
curl "$(harbor url litellm)/v1/chat/completions" \
  -H "Authorization: Bearer $(harbor config get litellm.master.key)" \
  -H 'Content-Type: application/json' \
  -d '{"model":"optillm/none-qwen3:0.6b","messages":[{"role":"user","content":"Hello"}]}'

• After changing the litellm.config.yaml - you will need to restart the service harbor restart litellm
• URLs must be internal to Docker network, can be obtained with harbor url -i <service>
• You can use harbor env litellm <name> <value> to set environment variables to reference in the config.yaml file

Following options can be set via harbor config:

# The port on the host where LiteLLM endpoint will be available
HARBOR_LITELLM_HOST_PORT      33841

# The port on the host where LiteLLM DB is available
HARBOR_LITELLM_DB_HOST_PORT   33842

# Master key for LiteLLM API access
HARBOR_LITELLM_MASTER_KEY     sk-litellm

# Credentials for LiteLLM UI
HARBOR_LITELLM_UI_USERNAME    admin
HARBOR_LITELLM_UI_PASSWORD    admin

Please see official LiteLLM documentation for plenty of additional examples.