Skip to content

Model Catalog Migration 9/9: Product Pages #638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
roh26it wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Model Catalog Migration 9/9: Product Pages #638

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 18 additions & 19 deletions product/administration/configure-virtual-key-access-permissions.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "Configure Virtual Key Access Permission for Workspaces"
title: "Configure Provider Access Permissions"
---

<Check>
Expand All @@ -8,50 +8,49 @@ title: "Configure Virtual Key Access Permission for Workspaces"

## Overview

Virtual Key Management in Portkey allows Organization administrators to define who can view and manage virtual keys within workspaces. This feature provides granular control over access to virtual keys, which are critical for managing connections to external AI providers and services.
Provider Management in Portkey allows Organization administrators to define who can view and manage providers within workspaces. This feature provides granular control over access to providers, which are critical for managing connections to external AI services.

## Accessing Virtual Key Management Permissions
## Accessing Provider Management Permissions

1. Navigate to **Admin Settings** in the Portkey dashboard
2. Select the **Security** tab from the left sidebar
3. Locate the **Virtual Key Management** section
3. Locate the **Provider Management** section

## Permission Settings

The Virtual Key Management section provides three distinct permission options:
The Provider Management section provides three distinct permission options:

| Permission | Description |
|------------|-------------|
| **View Virtual Keys (Workspace Managers)** | Enable workspace managers to view all virtual keys within their workspace. |
| **Manage Virtual Keys (Workspace Managers)** | Allow workspace managers to create, update, and delete virtual keys within their workspace. |
| **View Virtual Keys (Workspace Members)** | Enable workspace members to view virtual keys within their workspace. Note: Members cannot create, modify or delete virtual keys by default. |
| **View Providers (Workspace Managers)** | Enable workspace managers to view all providers within their workspace. |
| **Manage Providers (Workspace Managers)** | Allow workspace managers to create, update, and delete providers within their workspace. |
| **View Providers (Workspace Members)** | Enable workspace members to view providers within their workspace. Note: Members cannot create, modify or delete providers by default. |

<Frame caption="Virtual Key Management settings">
<Frame caption="Provider Management settings">
<img src="/images/product/virtual-key-management.png" />
</Frame>

## Understanding Virtual Keys
## Understanding Providers in Model Catalog

Virtual keys in Portkey securely store provider credentials and enable:
Providers in Portkey's [Model Catalog](/product/model-catalog) securely store your API credentials and enable:

- Centralized management of AI provider keys
- Abstraction of actual provider keys from end users
- Centralized management of AI provider credentials
- Abstraction of actual API keys from end users
- Definition of routing rules, fallbacks, and other advanced features
- Application of usage limits and tracking across providers

By controlling who can view and manage these virtual keys, organizations can maintain security while enabling appropriate access for different team roles.

By controlling who can view and manage providers, organizations can maintain security while enabling appropriate access for different team roles.

## Related Features

<Card title="Access Control Management" href="/product/enterprise-offering/access-control-management">
Learn about Portkey's access control features including user roles and organization hierarchy
</Card>

<Card title="Virtual Keys" href="/product/ai-gateway/virtual-keys">
Understand how virtual keys work and how to configure them effectively
<Card title="Model Catalog" href="/product/model-catalog">
Understand how to add and manage providers in the Model Catalog
</Card>

<Card title="Budget Limits" href="/product/ai-gateway/virtual-keys/budget-limits">
Learn how to set budget limits on virtual keys to control spending
<Card title="Budget Limits" href="/product/model-catalog/integrations#3-budget-%26-rate-limits">
Learn how to set budget limits on providers to control spending
</Card>
4 changes: 2 additions & 2 deletions product/ai-gateway.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,11 @@ description: The world's fastest AI Gateway with advanced routing & integrated G
Easily handle unresponsive LLM requests
</Card>

<Card title="Budget Limits" href="/product/ai-gateway/virtual-keys/budget-limits">
<Card title="Budget Limits" href="/product/model-catalog/integrations#3-budget-%26-rate-limits">
Set usage limits based on costs incurred or tokens used
</Card>

<Card title="Rate Limits" href="/product/ai-gateway/virtual-keys/rate-limits">
<Card title="Rate Limits" href="/product/model-catalog/integrations#3-budget-%26-rate-limits">
Set hourly, daily, or per minute rate limits on requests or tokens sent
</Card>
</CardGroup>
Expand Down
163 changes: 53 additions & 110 deletions product/ai-gateway/automatic-retries.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,160 +1,103 @@
---
title: "Automatic Retries"
description: "LLM APIs often have inexplicable failures. With Portkey, you can rescue a substantial number of your requests with our in-built automatic retries feature. "
description: Automatically retry failed LLM requests with exponential backoff.
---

<Info>
This feature is available on all Portkey [plans](https://portkey.ai/pricing).
Available on all Portkey [plans](https://portkey.ai/pricing).
</Info>

* Automatic retries are triggered **up to 5 times**
* Retries can also be triggered only on **specific error codes**
* Each subsequent retry attempt follows **exponential backoff strategy** to prevent network overload
* Optionally respects provider's `Retry-After` headers for rate-limited requests
- Up to **5 retry attempts**
- Trigger on **specific error codes**
- **Exponential backoff** to prevent overload
- Optionally respect provider's `Retry-After` headers

## Enabling Retries
## Examples

To enable retry, just add the `retry` param to your [config object](/api-reference/config-object).
<CodeGroup>

### Retry with 5 attempts

```JSON
```json Basic (5 attempts)
{
"retry": {
"attempts": 5
},
"provider":"@virtual-key-xxx"
"retry": { "attempts": 5 },
"override_params": { "model": "@openai-prod/gpt-4o" }
}
```

### Retry only on specific error codes

By default, Portkey triggers retries on the following error codes: **\[429, 500, 502, 503, 504\]**

You can change this behaviour by setting the optional `on_status_codes` param in your retry config and manually inputting the error codes on which retry will be triggered.


```JSON
```json Specific Error Codes
{
"retry": {
"attempts": 3,
"on_status_codes": [ 408, 429, 401 ]
},
"provider":"@virtual-key-xxx"
"retry": { "attempts": 3, "on_status_codes": [429, 503] },
"override_params": { "model": "@openai-prod/gpt-4o" }
}
```

<Info>
If the `on_status_codes` param is present, retries will be triggered **only** on the error codes specified in that Config and not on Portkey's default error codes for retries (i.e. \[429, 500, 502, 503, 504\])
</Info>

### Respecting provider's retry headers

Portkey can respect the provider's `retry-after-ms`, `x-ms-retry-after-ms` and `retry-after`response headers when encountering rate limits. This enables more intelligent retry timing based on the provider's response headers rather than using the default exponential backoff strategy.

To enable this feature, add the `use_retry_after_headers` parameter to your retry config. By default this behaviour is disabled, and `use_retry_after_headers` is set to `false`.

```JSON
```json Respect Retry-After Headers
{
"retry": {
"attempts": 3,
"on_status_codes": [ 429 ],
"use_retry_after_headers": true
},
"provider":"@virtual-key-xxx"
"retry": { "attempts": 3, "on_status_codes": [429], "use_retry_after_headers": true },
"override_params": { "model": "@openai-prod/gpt-4o" }
}
```

<Info>
When `use_retry_after_headers` is set to `true` and the provider includes `Retry-After` or `Retry-After-ms` headers in their response, Portkey will use these values to determine the wait time before the next retry attempt, overriding the exponential backoff strategy.

If the provider doesn't include these headers in the response, Portkey will fall back to the standard exponential backoff strategy.

The cumulative retry wait time for a single request is capped at 60 seconds. For example, if the first retry has a wait time of 20 seconds, and the second retry response includes a Retry-After value of 50 seconds, the request will fail since the total wait time (20+50=70) exceeds the 60-second cap. Similarly, if any single Retry-After value exceeds 60 seconds, the request will fail immediately.

</Info>

### Exponential backoff strategy

When not using provider retry headers (or when they're not available), Portkey triggers retries following this exponential backoff strategy:

| Attempt | Time out between requests |
| ----------------- | ------------------------- |
| Initial Call | Immediately |
| Retry 1st attempt | 1 second |
| Retry 2nd attempt | 2 seconds |
| Retry 3rd attempt | 4 seconds |
| Retry 4th attempt | 8 seconds |
| Retry 5th attempt | 16 seconds |



</CodeGroup>

<Info>
This feature is available on all Portkey [plans](https://portkey.ai/pricing).
The `@provider-slug/model-name` format automatically routes to the correct provider. Set up providers in [Model Catalog](https://app.portkey.ai/model-catalog).
</Info>

* Automatic retries are triggered **up to 5 times**
* Retries can also be triggered only on **specific error codes**
* And each subsequent retry attempt follows **exponential backoff strategy** to prevent network overload

## Enabling Retries
## Retry on Specific Error Codes

To enable retry, just add the `retry` param to your [config object](/api-reference/config-object).
Default retry codes: **[429, 500, 502, 503, 504]**

### Retry with 5 attempts
Override with `on_status_codes`:

```JSON
```json
{
"retry": {
"attempts": 5
},
"provider":"@virtual-key-xxx"
"retry": { "attempts": 3, "on_status_codes": [408, 429, 500] },
"override_params": { "model": "@openai-prod/gpt-4o" }
}
```

### Retry only on specific error codes

By default, Portkey triggers retries on the following error codes: **\[429, 500, 502, 503, 504\]**
<Note>
When `on_status_codes` is set, retries trigger **only** on those codes—not the defaults.
</Note>

You can change this behaviour by setting the optional `on_status_codes` param in your retry config and manually inputting the error codes on which rety will be triggered.
## Respect Provider Retry Headers

Enable `use_retry_after_headers` to use the provider's `retry-after-ms`, `x-ms-retry-after-ms`, or `retry-after` headers instead of exponential backoff.

```JSON
```json
{
"retry": {
"attempts": 3,
"on_status_codes": [ 408, 429, 401 ]
},
"provider":"@virtual-key-xxx"
"retry": { "attempts": 3, "on_status_codes": [429], "use_retry_after_headers": true },
"override_params": { "model": "@openai-prod/gpt-4o" }
}
```

<Info>
If the `on_status_codes` param is present, retries will be triggered **only** on the error codes specified in that Config and not on Portkey's default error codes for retries (i.e. \[429, 500, 502, 503, 504\])
- Falls back to exponential backoff if headers aren't present
- Cumulative retry wait capped at **60 seconds**
- Single `Retry-After` value > 60s fails immediately
</Info>

### Exponential backoff strategy

Here's how Portkey triggers retries following exponential backoff:
## Exponential Backoff

| Attempt | Time out between requests |
| ----------------- | ------------------------- |
| Initial Call | Immediately |
| Retry 1st attempt | 1 second |
| Retry 2nd attempt | 2 seconds |
| Retry 3rd attempt | 4 seconds |
| Retry 4th attempt | 8 seconds |
| Retry 5th attempt | 16 seconds |
| Attempt | Wait Time |
|---------|-----------|
| Initial | Immediate |
| 1st retry | 1 second |
| 2nd retry | 2 seconds |
| 3rd retry | 4 seconds |
| 4th retry | 8 seconds |
| 5th retry | 16 seconds |

### Understanding the Retry Attempt Header
## Retry Attempt Header

In the response from Portkey, you can find the `x-portkey-retry-attempt-count` header which provides information about retry attempts:
Check `x-portkey-retry-attempt-count` in responses:

- If the value is `-1`: This means that Portkey exhausted all the retry attempts and the request was unsuccessful
- If the value is `0`: This means that there were no retries configured
- If the value is `>0`: This means that Portkey attempted retries and this is the exact number at which the request was successful
| Value | Meaning |
|-------|---------|
| `-1` | All retries exhausted, request failed |
| `0` | No retries configured |
| `>0` | Successful on this retry attempt |

<Note>
Currently, Portkey does not log all the retry attempts individually in the logs dashboard. Instead, the response times from all retry attempts are summed up in the single log entry.
Retry attempts aren't logged individually. Response times are summed in a single log entry.
</Note>
2 changes: 1 addition & 1 deletion product/ai-gateway/batches.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,7 @@ Portkey Files are files uploaded to Portkey that are then automatically uploaded
| ---------------------------------- | --------------------------------------------------------------------------------------------------------- |
| **Batch Job** | A collection of completion requests executed asynchronously. |
| **Portkey File** (`input_file_id`) | Files uploaded to Portkey that are automatically uploaded to the provider for batch processing. Useful for reusing the same file across multiple batch completions. |
| **Virtual Key** | A logical provider credential stored in Portkey; referenced by ID, not secret. |
| **Provider Slug** | A unique identifier for your AI provider (e.g., `@openai-prod`). Set up in [Model Catalog](https://app.portkey.ai/model-catalog). |
| **Completion Window** | Time frame in which the job must finish. `immediate` → handled by Portkey; `24h` → delegated to provider. |

---
Expand Down
Loading