2.3.5 Satellite: LiteLLM

🚅 LiteLLM

Handle: litellm
URL: http://localhost:33841/

LLM API Proxy/Gateway.

LiteLLM is very useful for setups where the target LLM backend is either:

• Not supported by Harbor directly
• Doesn't have an OpenAI-compatible API that can be plugged into Open WebUI directly (for example, text-generation-inference)

litellm is also a way to use API-based LLM providers with Harbor.

Starting

# [Optional] Pull the litellm images
# ahead of starting the service
harbor pull litellm

# Start the service
harbor up litellm

You'll likely want to start it with at least one of the compatible LLM backends, or pointing to an external API ahead of time, see below.

Harbor's LiteLLM service is configured with a DB and an API key, so that you can access LiteLLM UI. The UI is available on the /ui endpoint of the service.

# Open LiteLLM API docs
harbor open litellm
# Open LiteLLM UI directly
harbor litellm ui

You can login with the default credentials: admin / admin. To adjust either modify the .env file or use the harbor CLI:

# Set the new credentials
harbor litellm username paole
harbor litellm password $(tr -dc 'A-Za-z0-9!?%=' < /dev/urandom | head -c 10)

Configuration

Harbor runs LiteLLM in the proxy mode. In order to configure it, you'll need to edit ./litellm/config.yaml file according to the documentation.

For example:

model_list:
  # What LiteLLM client will see
  - model_name: sllama
    litellm_params:
      # What LiteLLM will send to downstream API
      model: huggingface/repo/model
      # This can be pointed to one of the compatible Harbor
      # backends or to the external API compatible with the LiteLLM
      api_base: http://tgi:80
  - model_name: llamaster
    litellm_params:
      model: bedrock/meta.llama3-1-405b-instruct-v1:0
      aws_region_name: us-west-2

• After changing the config.yaml - you will need to restart the service harbor restart litellm
• URLs must be internal to Docker network, can be obtained with harbor url -i <service>
• You can use harbor env litellm <name> <value> to set environment variables to reference in the config.yaml file

Please see official LiteLLM documentation for plenty of additional examples.