|
| 1 | +--- |
| 2 | +sidebar_label: LiteLLM |
| 3 | +--- |
| 4 | + |
| 5 | +# Using LiteLLM With Roo Code |
| 6 | + |
| 7 | +LiteLLM is a versatile tool that provides a unified interface to over 100 Large Language Models (LLMs) by offering an OpenAI-compatible API. This allows you to run a local server that can proxy requests to various model providers or serve local models, all accessible through a consistent API endpoint. |
| 8 | + |
| 9 | +**Website:** [https://litellm.ai/](https://litellm.ai/) (Main project) & [https://docs.litellm.ai/](https://docs.litellm.ai/) (Documentation) |
| 10 | + |
| 11 | +## Key Benefits |
| 12 | + |
| 13 | +* **Unified API:** Access a wide range of LLMs (from OpenAI, Anthropic, Cohere, HuggingFace, etc.) through a single, OpenAI-compatible API. |
| 14 | +* **Local Deployment:** Run your own LiteLLM server locally, giving you more control over model access and potentially reducing latency. |
| 15 | +* **Simplified Configuration:** Manage credentials and model configurations in one place (your LiteLLM server) and let Roo Code connect to it. |
| 16 | +* **Cost Management:** LiteLLM offers features for tracking costs across different models and providers. |
| 17 | + |
| 18 | +## Setting Up Your LiteLLM Server |
| 19 | + |
| 20 | +To use LiteLLM with Roo Code, you first need to set up and run a LiteLLM server. |
| 21 | + |
| 22 | +1. **Installation:** Follow the official [LiteLLM installation guide](https://docs.litellm.ai/docs/proxy_server) to install LiteLLM and its dependencies. |
| 23 | +2. **Configuration:** Configure your LiteLLM server with the models you want to use. This typically involves setting API keys for the underlying providers (e.g., OpenAI, Anthropic) in your LiteLLM server's configuration. |
| 24 | +3. **Start the Server:** Run your LiteLLM server. By default, it usually starts on `http://localhost:4000`. |
| 25 | + * You can also configure an API key for your LiteLLM server itself for added security. |
| 26 | + |
| 27 | +Refer to the [LiteLLM documentation](https://docs.litellm.ai/docs/) for detailed instructions on server setup, model configuration, and advanced features. |
| 28 | + |
| 29 | +## Configuration in Roo Code |
| 30 | + |
| 31 | +Once your LiteLLM server is running: |
| 32 | + |
| 33 | +1. **Open Roo Code Settings:** Click the gear icon (<Codicon name="gear" />) in the Roo Code panel. |
| 34 | +2. **Select Provider:** Choose "LiteLLM" from the "API Provider" dropdown. |
| 35 | +3. **Enter Base URL:** |
| 36 | + * Input the URL of your LiteLLM server. |
| 37 | + * Defaults to `http://localhost:4000` if left blank. |
| 38 | +4. **Enter API Key (Optional):** |
| 39 | + * If you've configured an API key for your LiteLLM server, enter it here. |
| 40 | + * If your LiteLLM server doesn't require an API key, Roo Code will use a default dummy key (`"dummy-key"`), which should work fine. |
| 41 | +5. **Select Model:** |
| 42 | + * Roo Code will attempt to fetch the list of available models from your LiteLLM server by querying the `${baseUrl}/v1/model/info` endpoint. |
| 43 | + * The models displayed in the dropdown are sourced from this endpoint. |
| 44 | + * If no model is selected, Roo Code defaults to `anthropic/claude-3-7-sonnet-20250219` (this is `litellmDefaultModelId`). Ensure this model (or your desired default) is configured and available on your LiteLLM server. |
| 45 | + |
| 46 | +<img src="/img/litellm/litellm.png" alt="Roo Code LiteLLM Provider Settings" width="600" /> |
| 47 | + |
| 48 | +## How Roo Code Fetches and Interprets Model Information |
| 49 | + |
| 50 | +When you configure the LiteLLM provider, Roo Code interacts with your LiteLLM server to get details about the available models: |
| 51 | + |
| 52 | +* **Model Discovery:** Roo Code makes a GET request to `${baseUrl}/v1/model/info` on your LiteLLM server. If an API key is provided in Roo Code's settings, it's included in the `Authorization: Bearer ${apiKey}` header. |
| 53 | +* **Model Properties:** For each model reported by your LiteLLM server, Roo Code extracts and interprets the following: |
| 54 | + * `model_name`: The identifier for the model. |
| 55 | + * `maxTokens`: Maximum output tokens. Defaults to `8192` if not specified by LiteLLM. |
| 56 | + * `contextWindow`: Maximum context tokens. Defaults to `200000` if not specified by LiteLLM. |
| 57 | + * `supportsImages`: Determined from `model_info.supports_vision` provided by LiteLLM. |
| 58 | + * `supportsPromptCache`: Determined from `model_info.supports_prompt_caching` provided by LiteLLM. |
| 59 | + * `inputPrice` / `outputPrice`: Calculated from `model_info.input_cost_per_token` and `model_info.output_cost_per_token` from LiteLLM. |
| 60 | + * `supportsComputerUse`: This flag is set to `true` if the underlying model identifier (from `litellm_params.model`, e.g., `openrouter/anthropic/claude-3.5-sonnet`) matches one of the Anthropic models predefined in Roo Code as suitable for "computer use" (see `COMPUTER_USE_MODELS` in technical details). |
| 61 | + |
| 62 | +Roo Code uses default values for some of these properties if they are not explicitly provided by your LiteLLM server's `/model/info` endpoint for a given model. The defaults are: |
| 63 | +* `maxTokens`: 8192 |
| 64 | +* `contextWindow`: 200,000 |
| 65 | +* `supportsImages`: `true` |
| 66 | +* `supportsComputerUse`: `true` (for the default model ID) |
| 67 | +* `supportsPromptCache`: `true` |
| 68 | +* `inputPrice`: 3.0 (µUSD per 1k tokens) |
| 69 | +* `outputPrice`: 15.0 (µUSD per 1k tokens) |
| 70 | + |
| 71 | +## Tips and Notes |
| 72 | + |
| 73 | +* **LiteLLM Server is Key:** The primary configuration for models, API keys for downstream providers (like OpenAI, Anthropic), and other advanced features are managed on your LiteLLM server. Roo Code acts as a client to this server. |
| 74 | +* **Model Availability:** The models available in Roo Code's "Model" dropdown depend entirely on what your LiteLLM server exposes through its `/v1/model/info` endpoint. |
| 75 | +* **Network Accessibility:** Ensure your LiteLLM server is running and accessible from the machine where VS Code and Roo Code are running (e.g., check firewall rules if not on `localhost`). |
| 76 | +* **Troubleshooting:** If models aren't appearing or requests fail: |
| 77 | + * Verify your LiteLLM server is running and configured correctly. |
| 78 | + * Check the LiteLLM server logs for errors. |
| 79 | + * Ensure the Base URL in Roo Code settings matches your LiteLLM server's address. |
| 80 | + * Confirm any API key required by your LiteLLM server is correctly entered in Roo Code. |
| 81 | +* **Computer Use Models:** The `supportsComputerUse` flag in Roo Code is primarily relevant for certain Anthropic models known to perform well with tool-use and function-calling tasks. If you are routing other models through LiteLLM, this flag might not be automatically set unless the underlying model ID matches the specific Anthropic ones Roo Code recognizes. |
| 82 | + |
| 83 | +By leveraging LiteLLM, you can significantly expand the range of models accessible to Roo Code while centralizing their management. |
0 commit comments