Merge pull request #14769 from TeddyAmkie/doc-updates-sept-2025

krrishdholakia · web-flow · commit 666bcac2af11 · 2025-09-22T22:41:10.000-07:00
Doc updates sept 2025
diff --git a/docs/my-website/docs/completion/usage.md b/docs/my-website/docs/completion/usage.md
@@ -26,6 +26,7 @@ response = completion(
 
 print(response.usage)
 ```
+> **Note:** LiteLLM supports endpoint bridging—if a model does not natively support a requested endpoint, LiteLLM will automatically route the call to the correct supported endpoint (such as bridging `/chat/completions` to `/responses` or vice versa) based on the model's `mode`set in `model_prices_and_context_window`.
 
 ## Streaming Usage
 
diff --git a/docs/my-website/docs/enterprise.md b/docs/my-website/docs/enterprise.md
@@ -1,6 +1,11 @@
 import Image from '@theme/IdealImage';
 
 # Enterprise
+
+:::info
+✨ SSO is free for up to 5 users. After that, an enterprise license is required. [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
+:::
+
 For companies that need SSO, user management and professional support for LiteLLM Proxy
 
 :::info
diff --git a/docs/my-website/docs/fine_tuning.md b/docs/my-website/docs/fine_tuning.md
@@ -13,6 +13,8 @@ This is an Enterprise only endpoint [Get Started with Enterprise here](https://c
 | Feature | Supported | Notes | 
 |-------|-------|-------|
 | Supported Providers | OpenAI, Azure OpenAI, Vertex AI | - |
+
+#### ⚡️See an exhaustive list of supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
 | Cost Tracking | 🟡 | [Let us know if you need this](https://github.com/BerriAI/litellm/issues) |
 | Logging | ✅ | Works across all logging integrations |
 
diff --git a/docs/my-website/docs/getting_started.md b/docs/my-website/docs/getting_started.md
@@ -32,7 +32,8 @@ Next Steps 👉 [Call all supported models - e.g. Claude-2, Llama2-70b, etc.](./
 More details 👉
 
 - [Completion() function details](./completion/)
-- [All supported models / providers on LiteLLM](./providers/)
+- [Overview of supported models / providers on LiteLLM](./providers/)
+- [Search all models / providers](https://models.litellm.ai/)
 - [Build your own OpenAI proxy](https://github.com/BerriAI/liteLLM-proxy/tree/main)
 
 ## streaming
diff --git a/docs/my-website/docs/image_edits.md b/docs/my-website/docs/image_edits.md
@@ -18,6 +18,9 @@ LiteLLM provides image editing functionality that maps to OpenAI's `/images/edit
 | Supported LiteLLM Proxy Versions | 1.71.1+ | |
 | Supported LLM providers | **OpenAI** | Currently only `openai` is supported |
 
+ #### ⚡️See all supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
+
+
 ## Usage
 
 ### LiteLLM Python SDK
diff --git a/docs/my-website/docs/image_generation.md b/docs/my-website/docs/image_generation.md
@@ -279,6 +279,8 @@ print(f"response: {response}")
 
 ## Supported Providers
 
+#### ⚡️See all supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
+
 | Provider | Documentation Link |
 |----------|-------------------|
 | OpenAI | [OpenAI Image Generation →](./providers/openai) |
diff --git a/docs/my-website/docs/index.md b/docs/my-website/docs/index.md
@@ -524,6 +524,15 @@ try:
 except OpenAIError as e:
     print(e)
 ```
+### See How LiteLLM Transforms Your Requests
+
+Want to understand how LiteLLM parses and normalizes your LLM API requests? Use the `/utils/transform_request` endpoint to see exactly how your request is transformed internally.
+
+You can try it out now directly on our Demo App!
+Go to the [LiteLLM API docs for transform_request](https://litellm-api.up.railway.app/#/llm%20utils/transform_request_utils_transform_request_post)
+
+LiteLLM will show you the normalized, provider-agnostic version of your request. This is useful for debugging, learning, and understanding how LiteLLM handles different providers and options.
+
 
 ### Logging Observability - Log LLM Input/Output ([Docs](https://docs.litellm.ai/docs/observability/callbacks))
 LiteLLM exposes pre defined callbacks to send data to Lunary, MLflow, Langfuse, Helicone, Promptlayer, Traceloop, Slack
diff --git a/docs/my-website/docs/moderation.md b/docs/my-website/docs/moderation.md
@@ -130,6 +130,8 @@ Here's the exact json output and type you can expect from all moderation calls:
 
 ## **Supported Providers**
 
+#### ⚡️See all supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
+
 | Provider    |
 |-------------|
 | OpenAI      |  
diff --git a/docs/my-website/docs/observability/callbacks.md b/docs/my-website/docs/observability/callbacks.md
@@ -5,13 +5,15 @@
 liteLLM provides `input_callbacks`, `success_callbacks` and `failure_callbacks`, making it easy for you to send data to a particular provider depending on the status of your responses.
 
 :::tip
-**New to LiteLLM Callbacks?** Check out our comprehensive [Callback Management Guide](./callback_management.md) to understand when to use different callback hooks like `async_log_success_event` vs `async_post_call_success_hook`.
+**New to LiteLLM Callbacks?**
+
+- For proxy/server logging and observability, see the [Proxy Logging Guide](https://docs.litellm.ai/docs/proxy/logging).
+- To write your own callback logic, see the [Custom Callbacks Guide](https://docs.litellm.ai/docs/observability/custom_callback).
 :::
 
-liteLLM supports:
 
-- [Custom Callback Functions](https://docs.litellm.ai/docs/observability/custom_callback)
-- [Callback Management Guide](./callback_management.md) - **Comprehensive guide for choosing the right hooks**
+### Supported Callback Integrations
+
 - [Lunary](https://lunary.ai/docs)
 - [Langfuse](https://langfuse.com/docs)
 - [LangSmith](https://www.langchain.com/langsmith)
@@ -21,9 +23,20 @@ liteLLM supports:
 - [Sentry](https://docs.sentry.io/platforms/python/)
 - [PostHog](https://posthog.com/docs/libraries/python)
 - [Slack](https://slack.dev/bolt-python/concepts)
+- [Arize](https://docs.arize.com/)
+- [PromptLayer](https://docs.promptlayer.com/)
 
 This is **not** an extensive list. Please check the dropdown for all logging integrations.
 
+### Related Cookbooks
+Try out our cookbooks for code snippets and interactive demos:
+
+- [Langfuse Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/logging_observability/LiteLLM_Langfuse.ipynb)
+- [Lunary Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/logging_observability/LiteLLM_Lunary.ipynb)
+- [Arize Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/logging_observability/LiteLLM_Arize.ipynb)
+- [Proxy + Langfuse Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/logging_observability/LiteLLM_Proxy_Langfuse.ipynb)
+- [PromptLayer Callback Example (Colab)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/LiteLLM_PromptLayer.ipynb)
+
 ### Quick Start
 
 ```python
diff --git a/docs/my-website/docs/observability/custom_callback.md b/docs/my-website/docs/observability/custom_callback.md
@@ -67,6 +67,23 @@ asyncio.run(completion())
 - `async_post_call_success_hook` - Access user data + modify responses
 - `async_pre_call_hook` - Modify requests before sending
 
+### Example: Modifying the Response in async_post_call_success_hook
+
+You can use `async_post_call_success_hook` to add custom headers or metadata to the response before it is returned to the client. For example:
+
+```python
+async def async_post_call_success_hook(data, user_api_key_dict, response):
+    # Add a custom header to the response
+    additional_headers = getattr(response, "_hidden_params", {}).get("additional_headers", {}) or {}
+    additional_headers["x-litellm-custom-header"] = "my-value"
+    if not hasattr(response, "_hidden_params"):
+        response._hidden_params = {}
+    response._hidden_params["additional_headers"] = additional_headers
+    return response
+```
+
+This allows you to inject custom metadata or headers into the response for downstream consumers. You can use this pattern to pass information to clients, proxies, or observability tools.
+
 ## Callback Functions
 If you just want to log on a specific event (e.g. on input) - you can use callback functions. 
 
diff --git a/docs/my-website/docs/providers/bedrock.md b/docs/my-website/docs/providers/bedrock.md
@@ -2340,6 +2340,39 @@ response = completion(
 
 Make the bedrock completion call
 
+---
+
+### Required AWS IAM Policy for AssumeRole
+
+To use `aws_role_name` (STS AssumeRole) with LiteLLM, your IAM user or role **must** have permission to call `sts:AssumeRole` on the target role. If you see an error like:
+
+```
+An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::...:assumed-role/litellm-ecs-task-role/... is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::...:role/Enterprise/BedrockCrossAccountConsumer
+```
+
+This means the IAM identity running LiteLLM does **not** have permission to assume the target role. You must update your IAM policy to allow this action.
+
+#### Example IAM Policy
+
+Replace `<TARGET_ROLE_ARN>` with the ARN of the role you want to assume (e.g., `arn:aws:iam::123456789012:role/Enterprise/BedrockCrossAccountConsumer`).
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": "sts:AssumeRole",
+      "Resource": "<TARGET_ROLE_ARN>"
+    }
+  ]
+}
+```
+
+**Note:** The target role itself must also trust the calling IAM identity (via its trust policy) for AssumeRole to succeed. See [AWS AssumeRole docs](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-api.html) for more details.
+
+---
+
 <Tabs>
 <TabItem value="sdk" label="SDK">
 
diff --git a/docs/my-website/docs/providers/vertex.md b/docs/my-website/docs/providers/vertex.md
@@ -196,7 +196,19 @@ model_list:
       vertex_location: "us-central1"
       vertex_credentials: "/path/to/service_account.json" # [OPTIONAL] Do this OR `!gcloud auth application-default login` - run this to add vertex credentials to your env
 ```
-
+or
+```yaml
+model_list:
+ - model_name: gemini-pro
+    litellm_params:
+      model: vertex_ai/gemini-1.5-pro
+      litellm_credential_name: vertex-global
+      vertex_project: project-name-here
+      vertex_location: global
+      base_model: gemini
+      model_info:
+        provider: Vertex
+```
 2. Start Proxy 
 
 ```
diff --git a/docs/my-website/docs/proxy/caching.md b/docs/my-website/docs/proxy/caching.md
@@ -958,6 +958,19 @@ curl http://localhost:4000/v1/chat/completions \
 
 </Tabs>
 
+
+## Redis max_connections
+
+You can set the `max_connections` parameter in your `cache_params` for Redis. This is passed directly to the Redis client and controls the maximum number of simultaneous connections in the pool. If you see errors like `No connection available`, try increasing this value:
+
+```yaml
+litellm_settings:
+  cache: true
+  cache_params:
+    type: redis
+    max_connections: 100
+```
+
 ## Supported `cache_params` on proxy config.yaml
 
 ```yaml
@@ -966,6 +979,7 @@ cache_params:
   ttl: Optional[float]
   default_in_memory_ttl: Optional[float]
   default_in_redis_ttl: Optional[float]
+  max_connections: Optional[Int]
 
   # Type of cache (options: "local", "redis", "s3")
   type: s3
diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md
@@ -50,6 +50,7 @@ litellm_settings:
     port: 6379  # The port number for the Redis cache. Required if type is "redis".
     password: "your_password"  # The password for the Redis cache. Required if type is "redis".
     namespace: "litellm.caching.caching" # namespace for redis cache
+    max_connections: 100  # [OPTIONAL] Set Maximum number of Redis connections. Passed directly to redis-py. 
   
     # Optional - Redis Cluster Settings
     redis_startup_nodes: [{"host": "127.0.0.1", "port": "7001"}] 
diff --git a/docs/my-website/docs/proxy/custom_sso.md b/docs/my-website/docs/proxy/custom_sso.md
@@ -1,9 +1,7 @@
 # ✨ Event Hooks for SSO Login
 
 :::info
-
-✨ This is an Enterprise only feature [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
-
+✨ SSO is free for up to 5 users. After that, an enterprise license is required. [Get Started with Enterprise here](https://www.litellm.ai/enterprise)
 :::
 
 ## Overview
diff --git a/docs/my-website/docs/proxy/db_deadlocks.md b/docs/my-website/docs/proxy/db_deadlocks.md
@@ -84,3 +84,29 @@ LiteLLM emits the following prometheus metrics to monitor the health/status of t
 | `litellm_in_memory_spend_update_queue_size`         | In-memory aggregate spend values for keys, users, teams, team members, etc.| In-Memory    |
 | `litellm_redis_spend_update_queue_size`             | Redis aggregate spend values for keys, users, teams, etc.                  | Redis        |
 
+
+## Troubleshooting: Redis Connection Errors
+
+You may see errors like:
+
+```
+LiteLLM Redis Caching: async async_increment() - Got exception from REDIS No connection available., Writing value=21
+LiteLLM Redis Caching: async set_cache_pipeline() - Got exception from REDIS No connection available., Writing value=None
+```
+ 
+This means all available Redis connections are in use, and LiteLLM cannot obtain a new connection from the pool. This can happen under high load or with many concurrent proxy requests.
+
+**Solution:**
+
+- Increase the `max_connections` parameter in your Redis config section in `proxy_config.yaml` to allow more simultaneous connections. For example:
+
+```yaml
+litellm_settings:
+  cache: True
+  cache_params:
+    type: redis
+    max_connections: 100  # Increase as needed for your traffic
+```
+
+Adjust this value based on your expected concurrency and Redis server capacity.
+
diff --git a/docs/my-website/docs/proxy/guardrails/bedrock.md b/docs/my-website/docs/proxy/guardrails/bedrock.md
@@ -4,6 +4,10 @@ import TabItem from '@theme/TabItem';
 
 # Bedrock Guardrails
 
+:::tip ⚡️
+If you haven't set up or authenticated your Bedrock provider yet, see the [Bedrock Provider Setup & Authentication Guide](../../providers/bedrock.md).
+:::
+
 LiteLLM supports Bedrock guardrails via the [Bedrock ApplyGuardrail API](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_ApplyGuardrail.html). 
 
 ## Quick Start
diff --git a/docs/my-website/docs/proxy/load_balancing.md b/docs/my-website/docs/proxy/load_balancing.md
@@ -172,6 +172,9 @@ router_settings:
   redis_host: <your redis host>
   redis_password: <your redis password>
   redis_port: 1992
+  cache_params:
+    type: redis
+    max_connections: 100  # maximum Redis connections in the pool; tune based on expected concurrency/load
 ```
 
 ## Router settings on config - routing_strategy, model_group_alias
diff --git a/docs/my-website/docs/proxy/self_serve.md b/docs/my-website/docs/proxy/self_serve.md
@@ -227,7 +227,7 @@ export PROXY_LOGOUT_URL="https://www.google.com"
 <Image img={require('../../img/ui_logout.png')}  style={{ width: '400px', height: 'auto' }} />
 
 
-### Set max budget for internal users 
+### Set default max budget for internal users 
 
 Automatically apply budget per internal user when they sign up. By default the table will be checked every 10 minutes, for users to reset. To modify this, [see this](./users.md#reset-budgets)
 
@@ -239,6 +239,10 @@ litellm_settings:
 
 This sets a max budget of $10 USD for internal users when they sign up. 
 
+You can also manage these settings visually in the UI:
+
+<Image img={require('../../img/default_user_settings_admin_ui.png')}  style={{ width: '700px', height: 'auto' }} />
+
 This budget only applies to personal keys created by that user - seen under `Default Team` on the UI. 
 
 <Image img={require('../../img/max_budget_for_internal_users.png')}  style={{ width: '500px', height: 'auto' }} />
diff --git a/docs/my-website/docs/proxy_api.md b/docs/my-website/docs/proxy_api.md
@@ -27,7 +27,7 @@ Email us @ krrish@berri.ai
 ## Supported Models for LiteLLM Key
 These are the models that currently work with the "sk-litellm-.." keys.
 
-For a complete list of models/providers that you can call with LiteLLM, [check out our provider list](./providers/)
+For a complete list of models/providers that you can call with LiteLLM, [check out our provider list](./providers/) or check out [models.litellm.ai](https://models.litellm.ai/)
 
 * OpenAI models - [OpenAI docs](./providers/openai.md)
     * gpt-4
diff --git a/docs/my-website/docs/rerank.md b/docs/my-website/docs/rerank.md
@@ -109,6 +109,8 @@ curl http://0.0.0.0:4000/rerank \
 
 ## **Supported Providers**
 
+#### ⚡️See all supported models and providers at [models.litellm.ai](https://models.litellm.ai/)
+
 | Provider    | Link to Usage      |
 |-------------|--------------------|
 | Cohere (v1 + v2 clients)      |   [Usage](#quick-start)                 |
diff --git a/docs/my-website/docs/response_api.md b/docs/my-website/docs/response_api.md
diff --git a/docs/my-website/img/default_user_settings_admin_ui.png b/docs/my-website/img/default_user_settings_admin_ui.png
diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js
diff --git a/litellm/proxy/guardrails/guardrail_hooks/model_armor/model_armor.py b/litellm/proxy/guardrails/guardrail_hooks/model_armor/model_armor.py
diff --git a/security.md b/security.md
