-
-
-
-
-
-
-
-
-
+This creates a "Portkey Provider" that you can then use in any of your Portkey requests without having to send auth details for that LLM provider again.
-
-
+1. Navigate to [Integrations](https://app.portkey.ai/integrations) in Portkey dashboard and create a new Integration.
+2. Provision this Integration for each department with their budget limits and rate limits
+3. Configure model access if required.
-
-
-
-
-
+1. Navigate to [Integrations](https://app.portkey.ai/integrations) in Portkey dashboard and create a new Integration
+2. Provision the integration to relevant workspaces with their own budgets
+
-
-
-
-
+### Step 1: Setting Up Department-Specific Controls
+1. Navigate to [Integrations](https://app.portkey.ai/integrations) in Portkey dashboard and create one.
+2. Provision the integration to relevant Workspaces and add budget/rate limits per workspace as required.
@@ -21,7 +21,7 @@ Create a new virtual key on Portkey, select **Bedrock** as the provider and **AW
## Create an AWS Role for Portkey to Assume
-This role you create will be used by Porktey to execute InvokeModel commands on Bedrock models in your AWS account. The setup process will establish a minimal-permission ("least privilege") role and set it up to allow Porktey to assume this role.
+This role you create will be used by Portkey to execute InvokeModel commands on Bedrock models in your AWS account. The setup process will establish a minimal-permission ("least privilege") role and set it up to allow Portkey to assume this role.
### Create a permission policy in your AWS account using the following JSON
@@ -116,11 +116,11 @@ If you set an external ID, add it to the condition as shown below.
-## Configure the virtual key with the role ARN
+## Configure the integration with the role ARN
Once the role is created, copy the role ARN and paste it into the Bedrock integrations modal in Portkey along with the external ID if you set one and the AWS region you are using.
- You're all set! You can now use the virtual key to invoke Bedrock models.
+ You're all set! You can now use the providers inherited from your integration to invoke Bedrock models.
diff --git a/product/ai-gateway/virtual-keys/budget-limits.mdx b/product/ai-gateway/virtual-keys/budget-limits.mdx
index 459913e7..286cb07e 100644
--- a/product/ai-gateway/virtual-keys/budget-limits.mdx
+++ b/product/ai-gateway/virtual-keys/budget-limits.mdx
@@ -1,21 +1,21 @@
---
title: "Budget Limits"
-description: "Budget Limits lets you set cost limits on virtual keys"
+description: "Budget Limits lets you set cost limits on providers/integrations"
---
-Budget Limits lets you set cost or token limits on virtual keys
+Budget Limits lets you set cost or token limits on providers/integrations
-
+And you can set both these limtis for each workspace.
> #### Key Considerations
>
@@ -35,7 +33,7 @@ Set a maximum number of tokens that can be consumed, allowing you to control usa
> * Budget limits apply until exhausted or reset
> * Budget limits are applied only to requests made after the limit is set; they do not apply retroactively
> * Once set, budget limits **cannot be edited** by any organization member
-> * Budget limits work for **all providers** available on Portkey and apply to **all organization members** who use the virtual key
+> * Budget limit feature is available for **all LLMs** available on Portkey and will apply to **all members/workspaces** who have permission to use it.
## Alert Thresholds
@@ -65,15 +63,15 @@ You can configure budget limits to automatically reset at regular intervals:
## Editing Budget Limits
-If you need to change or update a budget limit, you can **duplicate** the existing virtual key and create a new one with the desired limit.
+If you need to change or update a budget limit, you can **duplicate** the existing provider and create a new one with the desired limit.
## Monitoring Your Spending and Usage
-You can track your spending and token usage for any specific virtual key by navigating to the Analytics tab and filtering by the **desired key** and **timeframe**.
+You can track your spending and token usage for any specific provider by navigating to the Analytics tab and filtering by the **desired provider** and **timeframe**.
## Pricing Support and Limitations
-Budget limits currently apply to all providers and models for which Portkey has pricing support. If a specific request log shows `0 cents` in the COST column, it means that Portkey does not currently track pricing for that model, and it will not count towards the virtual key's budget limit.
+Budget limits currently apply to all providers and models for which Portkey has pricing support. If a specific request log shows `0 cents` in the COST column, it means that Portkey does not currently track pricing for that model, and it will not count towards the providers's budget limit.
For token-based budgets, Portkey tracks both input and output tokens across all supported models.
diff --git a/product/ai-gateway/virtual-keys/rate-limits.mdx b/product/ai-gateway/virtual-keys/rate-limits.mdx
index 1e5b6083..07f202e7 100644
--- a/product/ai-gateway/virtual-keys/rate-limits.mdx
+++ b/product/ai-gateway/virtual-keys/rate-limits.mdx
@@ -1,21 +1,21 @@
---
title: "Rate Limits"
-description: "Set Rate Limts to your virtual keys"
+description: "Set Rate Limts to your Integrations/Providers"
---
-Rate Limits lets you set request or token consumption limits on virtual keys
+Rate Limits lets you set request or token consumption limits on providers/integrations
@@ -29,8 +29,8 @@ With the ability to create and manage multiple organizations, you can tailor acc
Portkey offers a comprehensive Role-Based Access Control (RBAC) system that allows you to define and assign user roles with granular permissions. By default, Portkey provides three roles: `Owner`, `Admin`, and `Member`, each with a predefined set of permissions across various features.
* `Owners` have complete control over the organization, including user management, billing, and all platform features.
-* `Admins` have elevated privileges, allowing them to manage users, prompts, configs, guardrails, virtual keys, and API keys.
-* `Members` have access to essential features like logs, analytics, prompts, configs, and virtual keys, with limited permissions.
+* `Admins` have elevated privileges, allowing them to manage users, prompts, configs, guardrails, integrations, providers, and API keys.
+* `Members` have access to essential features like logs, analytics, prompts, configs, and integrations, providers, with limited permissions.
| Feature | Owner Role | Admin Role | Member Role |
| ------------------ | ------------------------------------------- | ------------------------------------------- | -------------------------- |
@@ -38,7 +38,9 @@ Portkey offers a comprehensive Role-Based Access Control (RBAC) system that allo
| Prompts | List, View, Create, Update, Delete, Publish | List, View, Create, Update, Delete, Publish | List, View, Create, Update |
| Configs | List, View, Create, Update, Delete | List, View, Create, Update, Delete | List, View, Create |
| Guardrails | List, View, Create, Update, Delete | List, View, Create, Update, Delete | List, View, Create, Update |
-| Virtual Keys | List, Create, Edit, Duplicate, Delete, Copy | List, Create, Edit, Duplicate, Delete, Copy | List, Copy |
+| Integrations | List, Create, Edit, Duplicate, Delete, Copy | List, Create, Edit, Duplicate, Delete, Copy | List, Copy |
+| Model Catalog: Providers | List, Create, Edit, Duplicate, Delete, Copy | List, Create, Edit, Duplicate, Delete, Copy | List, Copy |
+| Model Catalog: Models | List, Create, Edit, Duplicate, Delete, Copy | List, Create, Edit, Duplicate, Delete, Copy | List, Copy |
| Team | Add users, assign roles | Add users, assign roles | \- |
| Organisation | Update | Update | \- |
| API Keys | Create, Edit, Delete, Update, Rotate | Create, Edit, Delete, Update, Rotate | \- |
@@ -52,7 +54,7 @@ You can easily add team members to your organization and assign them appropriate
## 3\. Secure and Customizable API Key Management
-Portkey provides a secure and flexible API key management system that allows you to create and manage multiple API keys with fine-grained permissions. Each API key can be customized to grant specific access levels to different features, such as metrics, completions, prompts, configs, guardrails, virtual keys, team management, and API key management.
+Portkey provides a secure and flexible API key management system that allows you to create and manage multiple API keys with fine-grained permissions. Each API key can be customized to grant specific access levels to different features, such as metrics, completions, prompts, configs, guardrails, integrations, providers, team management, and API key management.
| Feature | Permissions | Default |
| --------------------------- | ----------------------------- | -------- |
@@ -61,7 +63,9 @@ Portkey provides a secure and flexible API key management system that allows you
| Prompts | Disabled, Read, Write, Delete | Read |
| Configs | Disabled, Read, Write, Delete | Disabled |
| Guardrails | Disabled, Read, Write, Delete | Disabled |
-| Virtual Keys | Disabled, Read, Write, Delete | Disabled |
+| Integrations | Disabled, Read, Write, Delete | Disabled |
+| Model Catalog: Providers | Disabled, Read, Write, Delete | Disabled |
+| Model Catalog: Models | Disabled, Read, Write, Delete | Disabled |
| Users (Team Management) | Disabled, Read, Write, Delete | Disabled |
By default, a new organization is provisioned with a master API key that has all permissions enabled. Owners and admins can edit and manage these keys, as well as create new API keys with tailored permissions. This granular control enables you to enforce the principle of least privilege, ensuring that each API key has access only to the necessary resources.
@@ -70,7 +74,7 @@ Portkey's API key management system provides a secure and auditable way to contr
## Audit Logs
-Portkey maintains detailed audit logs that capture all administrative activities across the platform. These logs provide visibility into actions related to prompts, configs, guardrails, virtual keys, team management, organization updates, and API key modifications.
+Portkey maintains detailed audit logs that capture all administrative activities across the platform. These logs provide visibility into actions related to prompts, configs, guardrails, integrations, providers, team management, organization updates, and API key modifications.
Each log entry includes information about the user, the action performed, the affected resource, and a timestamp. This ensures traceability and accountability, helping teams monitor changes and investigate any unauthorized activity.
diff --git a/product/enterprise-offering/audit-logs.mdx b/product/enterprise-offering/audit-logs.mdx
index 7da163d5..d637ef7f 100644
--- a/product/enterprise-offering/audit-logs.mdx
+++ b/product/enterprise-offering/audit-logs.mdx
@@ -64,7 +64,8 @@ Portkey provides powerful filtering options to help you find specific audit even
- **Resource Type**: Filter by type of resource affected:
- Workspaces
- API Keys
- - Virtual Keys
+ - Integrations
+ - Model Catalog
- Configs
- Prompts
- Guardrails
diff --git a/product/enterprise-offering/cloud-marketplace/aws.mdx b/product/enterprise-offering/cloud-marketplace/aws.mdx
index 6398b7c8..21e87bc1 100644
--- a/product/enterprise-offering/cloud-marketplace/aws.mdx
+++ b/product/enterprise-offering/cloud-marketplace/aws.mdx
@@ -18,7 +18,7 @@ description: This enterprise-focused document provides comprehensive instruction
| --------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| AI Gateway | Deploy as a Docker container in your Kubernetes cluster using Helm Charts | AWS NodeGroup t4g.medium instance, with at least 4GiB of memory and two vCPUs For high reliability, deploy across multiple Availability Zones. |
| Logs store | AWS S3 | Each log document is \~10kb in size (uncompressed) |
-| Cache (Prompts, Configs & Virtual Keys) | Elasticache or self-hosted Redis | Deploy in the same VPC as the Portkey Gateway. |
+| Cache (Prompts, Configs & Providers) | Elasticache or self-hosted Redis | Deploy in the same VPC as the Portkey Gateway. |
## Helm Chart
diff --git a/product/enterprise-offering/kms.mdx b/product/enterprise-offering/kms.mdx
index 7daf2e33..323911d9 100644
--- a/product/enterprise-offering/kms.mdx
+++ b/product/enterprise-offering/kms.mdx
@@ -36,7 +36,7 @@ sequenceDiagram
### Encrypted Fields
- Configs
- Full template
-- Virtual Keys
+- LLM Integrations
- Auth Key
- Auth Configuration
- Prompts
diff --git a/product/enterprise-offering/org-management.mdx b/product/enterprise-offering/org-management.mdx
index 39a09775..872a0fdf 100644
--- a/product/enterprise-offering/org-management.mdx
+++ b/product/enterprise-offering/org-management.mdx
@@ -17,7 +17,7 @@ This hierarchy allows for efficient management of resources and access control a
**Workspaces** is currently a feature only enabled on the **Enterprise Plans**. If you're looking to add this feature to your organisation, please reach out to us on our [Discord Community](https://portkey.ai/community) or via email on [support@portkey.ai](mailto:support@portkey.ai)
-Organizations contain User Invites & Users, Admin API Keys, and Workspaces. Workspaces, in turn, have their own Team structure (with Managers and Members), Workspace API Keys, and various features like Virtual Keys, Configs, Prompts, and more.
+Organizations contain User Invites & Users, Admin API Keys, and Workspaces. Workspaces, in turn, have their own Team structure (with Managers and Members), Workspace API Keys, and various features like Integrations, Model Catalog, Configs, Prompts, and more.
This structure enables you to:
diff --git a/product/enterprise-offering/org-management/jwt.mdx b/product/enterprise-offering/org-management/jwt.mdx
index 046c305b..4f64bb62 100644
--- a/product/enterprise-offering/org-management/jwt.mdx
+++ b/product/enterprise-offering/org-management/jwt.mdx
@@ -123,27 +123,24 @@ Once the JWT is validated, the server checks for the required **scope**. Scopes
- 
-Resources that must be deleted before removing a workspace like - Prompts, Prompt partials, Virtual keys, Configs, Guardrails and more.
+Resources that must be deleted before removing a workspace like - Prompts, Prompt partials, Providers, Configs, Guardrails and more.
Once all resources have been removed, enter the workspace name in the confirmation field to proceed with deletion.
diff --git a/product/enterprise-offering/private-cloud-deployments/architecture.mdx b/product/enterprise-offering/private-cloud-deployments/architecture.mdx
index 8bc9108e..0d3a30bd 100644
--- a/product/enterprise-offering/private-cloud-deployments/architecture.mdx
+++ b/product/enterprise-offering/private-cloud-deployments/architecture.mdx
@@ -58,7 +58,7 @@ The Control Plane is fully managed by Portkey and provides the administrative la
The AI Gateway periodically synchronizes with the Control Plane:
- **Frequency**: 30-second heartbeat intervals
- - **Data Retrieved**: Prompt templates, routing configs, virtual keys, API keys
+ - **Data Retrieved**: Prompt templates, routing configs, integrations, providers, API keys
- **Process**: Data is fetched, decrypted locally, and stored in the Gateway cache
- **Resilience**: Gateway operates independently between syncs using cached configs
@@ -96,7 +96,7 @@ Portkey AI Gateway is deployed as containerized workloads using Helm charts for
| Component | Description | Configuration Options |
|:----------|:------------|:----------------------|
| **AI Gateway** | Core container running the routing logic | Deployed as stateless containers that can scale horizontally |
-| **Cache System** | Stores routing configs, virtual keys, and more | Redis (in-cluster, AWS ElastiCache, or custom endpoint) |
+| **Cache System** | Stores routing configs, integrations, providers, and more | Redis (in-cluster, AWS ElastiCache, or custom endpoint) |
| **Log Storage** | Persistence for request/response data | Multiple options (see below) |
### Storage Options
diff --git a/product/enterprise-offering/private-cloud-deployments/aws.mdx b/product/enterprise-offering/private-cloud-deployments/aws.mdx
index 063d92bb..08957f6d 100644
--- a/product/enterprise-offering/private-cloud-deployments/aws.mdx
+++ b/product/enterprise-offering/private-cloud-deployments/aws.mdx
@@ -11,7 +11,7 @@ description: This enterprise-focused document provides comprehensive instruction
| --------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| AI Gateway | Deploy as a Docker container in your Kubernetes cluster using Helm Charts | AWS EC2 t4g.medium instance, with at least 4GiB of memory and two vCPUs For high reliability, deploy across multiple Availability Zones. |
| Logs Store (optional) | Hosted MongoDB, Document DB or AWS S3 | Each log document is \~10kb in size (uncompressed) |
-| Cache (Prompts, Configs & Virtual Keys) | Elasticache or self-hosted Redis | Deploy in the same VPC as the Portkey Gateway. |
+| Cache (Prompts, Configs & Providers) | Elasticache or self-hosted Redis | Deploy in the same VPC as the Portkey Gateway. |
## Deployment Steps
diff --git a/product/enterprise-offering/private-cloud-deployments/azure.mdx b/product/enterprise-offering/private-cloud-deployments/azure.mdx
index eb011cd0..56f0cee2 100644
--- a/product/enterprise-offering/private-cloud-deployments/azure.mdx
+++ b/product/enterprise-offering/private-cloud-deployments/azure.mdx
@@ -27,7 +27,7 @@ description: This enterprise-focused document provides comprehensive instruction
* **Options:** Azure Cosmos DB, Azure Blob Storage.
* **Sizing:** Each log document is \~10kb in size (uncompressed).
-### Component: Cache (Prompts, Configs & Virtual Keys)
+### Component: Cache (Prompts, Configs & Providers)
* **Options:** Azure Cache for Redis or self-hosted Redis.
* **Deployment:** Deploy in the same VNet as the Portkey Gateway.
diff --git a/product/enterprise-offering/private-cloud-deployments/gcp.mdx b/product/enterprise-offering/private-cloud-deployments/gcp.mdx
index f083bbb5..e904cebb 100644
--- a/product/enterprise-offering/private-cloud-deployments/gcp.mdx
+++ b/product/enterprise-offering/private-cloud-deployments/gcp.mdx
@@ -18,7 +18,7 @@ It includes specific recommendations for component sizing, high availability, di
* **Options:** Hosted MongoDB, Google Cloud Storage (GCS), or Google Firestore.
* **Sizing:** Each log document is \~10kb in size (uncompressed).
-### Component: Cache (Prompts, Configs & Virtual Keys)
+### Component: Cache (Prompts, Configs & Providers)
* **Options:** Google Memorystore for Redis or self-hosted Redis.
* **Deployment:** Deploy in the same VPC as the Portkey Gateway.
diff --git a/product/enterprise-offering/security-portkey.mdx b/product/enterprise-offering/security-portkey.mdx
index a761620b..27a9e529 100644
--- a/product/enterprise-offering/security-portkey.mdx
+++ b/product/enterprise-offering/security-portkey.mdx
@@ -12,7 +12,7 @@ At Portkey AI, we understand the critical importance of security in today's digi
Portkey AI ensures secure API access through token-based authentication mechanisms, supporting Single Sign-On (SSO) via OIDC on enterprise plans.
-We also implement [Virtual Keys](/product/ai-gateway/virtual-keys), which provide an added layer of security by securely storing provider API keys within a controlled and monitored environment.
+We also implement key encryption, which provide an added layer of security by securely storing provider API keys within a controlled and monitored environment.
This multi-tier authentication strategy is crucial for protecting against unauthorized access and ensuring the integrity of user interactions.
diff --git a/product/guardrails.mdx b/product/guardrails.mdx
index 64e22d2a..85259636 100644
--- a/product/guardrails.mdx
+++ b/product/guardrails.mdx
@@ -134,7 +134,7 @@ Portkey now offers a more intuitive way to add guardrails to your configurations
"cache": {
"mode": "simple"
},
- "virtual_key": "openai-xxx",
+ "provider":"@openai-xxx",
"input_guardrails": ["guardrails-id-xxx", "guardrails-id-yyy"],
"output_guardrails": ["guardrails-id-xxx", "guardrails-id-yyy"]
}
@@ -158,7 +158,7 @@ You can also continue to use the original hook-based approach:
"cache": {
"mode": "simple"
},
- "virtual_key": "openai-xxx",
+ "provider":"@openai-xxx",
"before_request_hooks": [{
"id": "input-guardrail-id-xx"
}],
@@ -204,8 +204,8 @@ For requests where `async= FALSE`:
"on_status_codes": [246, 446]
},
"targets": [
- {"virtual_key": "openai-key-xxx"},
- {"virtual_key": "anthropic-key-xxx"}
+ {"provider":"@openai-key-xxx"},
+ {"provider":"@anthropic-key-xxx"}
],
"input_guardrails": ["guardrails-id-xxx"]
}
diff --git a/product/integrations.mdx b/product/integrations.mdx
index 83201dcf..047c164c 100644
--- a/product/integrations.mdx
+++ b/product/integrations.mdx
@@ -34,6 +34,30 @@ If you are an existing Portkey user, this step is similar to creating a Virtual
* **Credentials:** Securely enter your API keys or other authentication details. These are encrypted and will not be visible after saving.
3. Click **Next**.
+##### **Special Case: Connecting to Azure OpenAI**
+
+Portkey allows you to manage multiple Azure deployments under a single integration. This is useful when different models or versions are in separate deployments.
+
+1. When connecting an Azure integration, you can add multiple deployments.
+2. Assign a unique **`alias`** to each deployment (e.g., `gpt-4o-deploy`, `gpt-35-deploy`).
+3. Set one deployment as the default.
+
+To use a specific deployment, developers will pass its `alias` as the `model` in their LLM request. If no alias is provided, the default deployment is used.
+
+
+ 
+
+
+##### **Special Case: Connecting to Self-Hosted LLMs**
+
+You can manage your privately hosted models alongside commercial providers.
+
+1. When creating a new integration, enable the **"Local/Privately hosted provider"** toggle.
+2. Select the API specification your LLM implements (e.g., OpenAI).
+3. Enter your model's base URL in the **"Custom Host"** field and add any required authentication headers.
+
+This allows you to use your self-hosted models with all Portkey features. For more details, see [Bring Your Own LLM](/product/ai-gateway/byollm).
+
#### **Step 2: Provision to Workspaces**
Here, you decide which teams get access to this provider and under what conditions.
@@ -53,4 +77,34 @@ This is where you enforce model governance and control costs.
3. **(Optional) For Dynamic Models:** If you're using a provider like Fireworks AI with many community models, you can toggle on **"Automatically enable new models"**. For providers like OpenAI or Azure, we recommend an explicit allow-list for better cost control.
4. Click **Create Integration**.
-**That's it!** You have successfully created and provisioned a centrally managed integration. It will now appear in your **`Connected`** tab. The workspaces you provisioned will see this as an available "AI Provider" in their Model Catalog, with access only to the models you specified and constrained by the budgets you set.
\ No newline at end of file
+**That's it!** You have successfully created and provisioned a centrally managed integration. It will now appear in your **`Connected`** tab. The workspaces you provisioned will see this as an available "AI Provider" in their Model Catalog, with access only to the models you specified and constrained by the budgets you set.
+
+### **Setting Budgets and Rate Limits**
+
+When provisioning an integration to a workspace, you can enforce powerful governance rules by setting budget and rate limits. This gives you granular control over costs and usage patterns.
+
+
+
+
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+
+### 6\. [Prompt Management](/product/prompt-library)
+
+Use Portkey as a centralized hub to store, version, and experiment with your agent's prompts across multiple LLMs. Easily modify your prompts and run A/B tests without worrying about the breaking prod.
+
+### 7\. [Continuous Improvement](/product/observability/feedback)
+
+Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests, and then using that feedback to make your prompts AND LLMs themselves better.
+
+### 8\. [Security & Compliance](/product/enterprise-offering/security-portkey)
+
+Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
diff --git a/virtual_key_old/integrations/agents/autogen.mdx b/virtual_key_old/integrations/agents/autogen.mdx
new file mode 100644
index 00000000..5e6fb1b0
--- /dev/null
+++ b/virtual_key_old/integrations/agents/autogen.mdx
@@ -0,0 +1,214 @@
+---
+title: "Autogen"
+description: "Use Portkey with Autogen to take your AI Agents to production"
+---
+
+## Getting Started
+
+### 1\. Install the required packages:
+
+```sh
+pip install -qU pyautogen portkey-ai
+```
+
+### **2\.** Configure your Autogen configs:
+
+```py
+from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
+
+config = [
+ {
+ "api_key": "OPENAI_API_KEY",
+ "model": "gpt-3.5-turbo",
+ "base_url": PORTKEY_GATEWAY_URL,
+ "api_type": "openai",
+ "default_headers": createHeaders(
+ api_key ="PORTKEY_API_KEY", #Replace with Your Portkey API key
+ provider = "openai",
+ )
+ }
+]
+```
+
+## Integration Guide
+
+Here's a simple Google Colab notebook that demonstrates Autogen with Portkey integration
+
+[](https://dub.sh/Autogen-docs)
+
+## Make your agents Production-ready with Portkey
+
+Portkey makes your Autogen agents reliable, robust, and production-grade with its observability suite and AI Gateway. Seamlessly integrate 200+ LLMs with your Autogen agents using Portkey. Implement fallbacks, gain granular insights into agent performance and costs, and continuously optimize your AI operationsโall with just 2 lines of code.
+
+Let's dive deep! Let's go through each of the use cases!
+
+### 1.[ Interoperability](/product/ai-gateway/universal-api)
+
+Easily switch between 200+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `provider ` and `API key` in the `ChatOpenAI` object.
+
+
+
+
+
+### 5\. [Continuous Improvement](/product/observability/feedback)
+
+Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
+
+### 6\. [Caching](/product/ai-gateway/cache-simple-and-semantic)
+
+Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
+
+
+
+```json
+{
+ "cache": {
+ "mode": "semantic" // Choose between "simple" or "semantic"
+ }
+}
+```
+
+### 7\. [Security & Compliance](/product/enterprise-offering/security-portkey)
+
+Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
+
+
+## [Portkey Config](/product/ai-gateway/configs)
+
+Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
+
+For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
diff --git a/virtual_key_old/integrations/agents/bring-your-own-agents.mdx b/virtual_key_old/integrations/agents/bring-your-own-agents.mdx
new file mode 100644
index 00000000..0611978b
--- /dev/null
+++ b/virtual_key_old/integrations/agents/bring-your-own-agents.mdx
@@ -0,0 +1,191 @@
+---
+title: "Bring Your own Agents"
+description: "You can also use Portkey if you are doing custom agent orchestration!"
+---
+
+## Getting Started
+
+### 1\. Install the required packages:
+
+```sh
+pip install portkey-ai openai
+```
+
+### **2\.** Configure your OpenAI object:
+
+
+```py
+client = OpenAI(
+ api_key="OPENAI_API_KEY",
+ base_url=PORTKEY_GATEWAY_URL,
+ default_headers=createHeaders(
+ provider="openai",
+ api_key="PORTKEY_API_KEY",
+ virtual_key="openai-latest-a4a53d"
+ )
+)
+```
+
+## Integrate Portkey with your custom Agents
+
+This notebook demonstrates integrating a ReAct agent with Portkey
+
+[](https://dub.sh/ReAct-agent)
+
+## Make your agents Production-ready with Portkey
+
+Portkey makes your agents reliable, robust, and production-grade with its observability suite and AI Gateway. Seamlessly integrate 200+ LLMs with your custom agents using Portkey. Implement fallbacks, gain granular insights into agent performance and costs, and continuously optimize your AI operationsโall with just 2 lines of code.
+
+Let's dive deep! Let's go through each of the use cases!
+
+### 1\. [Interoperability](/product/ai-gateway/universal-api)
+
+Easily switch between 200+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `provider ` and `API key` in the `ChatOpenAI` object.
+
+
+
+### 5\. [Continuous Improvement](/product/observability/feedback)
+
+Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
+
+### 6\. [Caching](/product/ai-gateway/cache-simple-and-semantic)
+
+Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
+
+
+
+```py
+{
+ "cache": {
+ "mode": "semantic" // Choose between "simple" or "semantic"
+ }
+}
+```
+
+### 7.[ Security & Compliance](/product/enterprise-offering/security-portkey)
+
+Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
+
+***
+
+## [Portkey Config](/product/ai-gateway/configs)
+
+Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
+
+For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
diff --git a/virtual_key_old/integrations/agents/control-flow.mdx b/virtual_key_old/integrations/agents/control-flow.mdx
new file mode 100644
index 00000000..56748ab9
--- /dev/null
+++ b/virtual_key_old/integrations/agents/control-flow.mdx
@@ -0,0 +1,199 @@
+---
+title: "Control Flow"
+description: "Use Portkey with Control Flow to take your AI Agents to production"
+---
+
+## Getting Started
+
+### 1\. Install the required packages:
+
+```sh
+pip install -qU portkey-ai controlflow
+```
+
+### **2\.** Configure your Control FLow LLM objects:
+
+```py
+import controlflow as cf
+from langchain_openai import ChatOpenAI
+from portkey_ai import createHeaders, PORTKEY_GATEWAY_URL
+
+llm = ChatOpenAI(
+ api_key="OpenAI_API_Key",
+ base_url=PORTKEY_GATEWAY_URL,
+ default_headers=createHeaders(
+ provider="openai", #choose your provider
+ api_key="PORTKEY_API_KEY"
+ )
+)
+```
+
+## Integration Guide
+
+Here's a simple Google Colab notebook that demonstrates Control Flow with Portkey integration
+
+[](https://dub.sh/Control-Flow-docs)
+
+
+## Make your agents Production-ready with Portkey
+
+Portkey makes your Control Flow agents reliable, robust, and production-grade with its observability suite and AI Gateway. Seamlessly integrate 200+ LLMs with your Control Flow agents using Portkey. Implement fallbacks, gain granular insights into agent performance and costs, and continuously optimize your AI operationsโall with just 2 lines of code.
+
+Let's dive deep! Let's go through each of the use cases!
+
+### 1\. [Interoperability](/product/ai-gateway/universal-api)
+
+Easily switch between 200+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `provider ` and `API key` in the `ChatOpenAI` object.
+
+
+
+### 5\. [Continuous Improvement](/product/observability/feedback)
+
+Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
+
+### 6\. [Caching](/product/ai-gateway/cache-simple-and-semantic)
+
+Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
+
+```py
+{
+ "cache": {
+ "mode": "semantic" // Choose between "simple" or "semantic"
+ }
+}
+```
+
+### 7\. [Security & Compliance](/product/enterprise-offering/security-portkey)
+
+Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
+
+---
+
+## Portkey Config
+
+Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
+
+For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
diff --git a/virtual_key_old/integrations/agents/crewai.mdx b/virtual_key_old/integrations/agents/crewai.mdx
new file mode 100644
index 00000000..263a86d9
--- /dev/null
+++ b/virtual_key_old/integrations/agents/crewai.mdx
@@ -0,0 +1,822 @@
+---
+title: "CrewAI"
+description: "Use Portkey with CrewAI to take your AI Agents to production"
+---
+
+## Introduction
+
+CrewAI is a framework for orchestrating role-playing, autonomous AI agents designed to solve complex, open-ended tasks through collaboration. It provides a robust structure for agents to work together, leverage tools, and exchange insights to accomplish sophisticated objectives.
+
+Portkey enhances CrewAI with production-readiness features, turning your experimental agent crews into robust systems by providing:
+
+- **Complete observability** of every agent step, tool use, and interaction
+- **Built-in reliability** with fallbacks, retries, and load balancing
+- **Cost tracking and optimization** to manage your AI spend
+- **Access to 200+ LLMs** through a single integration
+- **Guardrails** to keep agent behavior safe and compliant
+- **Version-controlled prompts** for consistent agent performance
+
+
+
+
+Traces provide a hierarchical view of your crew's execution, showing the sequence of LLM calls, tool invocations, and state transitions.
+
+```python
+# Add trace_id to enable hierarchical tracing in Portkey
+portkey_llm = LLM(
+ model="gpt-4o",
+ base_url=PORTKEY_GATEWAY_URL,
+ api_key="dummy",
+ extra_headers=createHeaders(
+ api_key="YOUR_PORTKEY_API_KEY",
+ virtual_key="YOUR_OPENAI_VIRTUAL_KEY",
+ trace_id="unique-session-id" # Add unique trace ID
+ )
+)
+```
+ 
+
+
+Portkey logs every interaction with LLMs, including:
+
+- Complete request and response payloads
+- Latency and token usage metrics
+- Cost calculations
+- Tool calls and function executions
+
+All logs can be filtered by metadata, trace IDs, models, and more, making it easy to debug specific crew runs.
+ 
+
+
+Portkey provides built-in dashboards that help you:
+
+- Track cost and token usage across all crew runs
+- Analyze performance metrics like latency and success rates
+- Identify bottlenecks in your agent workflows
+- Compare different crew configurations and LLMs
+
+You can filter and segment all metrics by custom metadata to analyze specific crew types, user groups, or use cases.
+ 
+
+
+Add custom metadata to your CrewAI LLM configuration to enable powerful filtering and segmentation:
+
+```python
+portkey_llm = LLM(
+ model="gpt-4o",
+ base_url=PORTKEY_GATEWAY_URL,
+ api_key="dummy",
+ extra_headers=createHeaders(
+ api_key="YOUR_PORTKEY_API_KEY",
+ virtual_key="YOUR_OPENAI_VIRTUAL_KEY",
+ metadata={
+ "crew_type": "research_crew",
+ "environment": "production",
+ "_user": "user_123", # Special _user field for user analytics
+ "request_source": "mobile_app"
+ }
+ )
+)
+```
+
+This metadata can be used to filter logs, traces, and metrics on the Portkey dashboard, allowing you to analyze specific crew runs, users, or environments.
+ 
+
+
+This enables:
+- Per-user cost tracking and budgeting
+- Personalized user analytics
+- Team or organization-level metrics
+- Environment-specific monitoring (staging vs. production)
+
+
+
+
+
+
+
+
+
+
+ Official CrewAI documentation
+Get personalized guidance on implementing this integration
+
+
+
+### 5\. [Traces](/product/observability/traces)
+
+With traces, you can see each agent run granularly on Portkey. Tracing your Langchain agent runs helps in debugging, performance optimzation, and visualizing how exactly your agents are running.
+
+### Using Traces in Langchain Agents
+
+#### Step 1: Import & Initialize the Portkey Langchain Callback Handler
+
+
+
+```py
+from portkey_ai.langchain import LangchainCallbackHandler
+
+portkey_handler = LangchainCallbackHandler(
+ api_key="YOUR_PORTKEY_API_KEY",
+ metadata={
+ "session_id": "session_1", # Use consistent metadata across your application
+ "agent_id": "research_agent_1", # Specific to the current agent
+ }
+)
+
+```
+
+#### Step 2: Configure Your LLM with the Portkey Callback
+
+
+
+```py
+from langchain.chat_models import ChatOpenAI
+
+llm = ChatOpenAI(
+ api_key="YOUR_OPENAI_API_KEY_HERE",
+ callbacks=[portkey_handler],
+ # ... other parameters
+)
+```
+
+With Portkey tracing, you can encapsulate the complete execution of your agent workflow.
+
+ 
+
+
+### 6\. Guardrails
+
+LLMs are brittle - not just in API uptimes or their inexplicable `400`/`500` errors, but also in their core behavior. You can get a response with a `200` status code that completely errors out for your app's pipeline due to mismatched output. With Portkey's Guardrails, we now help you enforce LLM behavior in real-time with our _Guardrails on the Gateway_ pattern.
+
+Using Portkey's Guardrail platform, you can now verify your LLM inputs AND outputs to be adhering to your specifed checks; and since Guardrails are built on top of our [Gateway](https://github.com/portkey-ai/gateway), you can orchestrate your request exactly the way you want - with actions ranging from _denying the request_, _logging the guardrail result_, _creating an evals dataset_, _falling back to another LLM or prompt_, _retrying the request_, and more.
+
+### 7\. [Continuous Improvement](/product/observability/feedback)
+
+Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
+
+### 8\. [Caching](/product/ai-gateway/cache-simple-and-semantic)
+
+Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
+
+
+
+```json
+{
+ "cache": {
+ "mode": "semantic" // Choose between "simple" or "semantic"
+ }
+}
+```
+
+### 9\. [Security & Compliance](/product/enterprise-offering/security-portkey)
+
+Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
+
+---
+
+## [Portkey Config](/product/ai-gateway/configs)
+
+Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
+
+For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
diff --git a/virtual_key_old/integrations/agents/langgraph.mdx b/virtual_key_old/integrations/agents/langgraph.mdx
new file mode 100644
index 00000000..b7f32c65
--- /dev/null
+++ b/virtual_key_old/integrations/agents/langgraph.mdx
@@ -0,0 +1,1022 @@
+---
+title: "LangGraph"
+description: "Use Portkey with LangGraph to take your AI agent workflows to production"
+---
+
+## Introduction
+
+LangGraph is a library for building stateful, multi-actor applications with LLMs, designed to make developing complex agent workflows easier. It provides a flexible framework to create directed graphs where nodes process information and edges define the flow between them.
+
+Portkey enhances LangGraph with production-readiness features, turning your experimental agent workflows into robust systems by providing:
+
+- **Complete observability** of every agent step, tool use, and state transition
+- **Built-in reliability** with fallbacks, retries, and load balancing
+- **Cost tracking and optimization** to manage your AI spend
+- **Access to 200+ LLMs** through a single integration
+- **Guardrails** to keep agent behavior safe and compliant
+- **Version-controlled prompts** for consistent agent performance
+
+
+
+
+Traces provide a hierarchical view of your agent's execution, showing the sequence of LLM calls, tool invocations, and state transitions.
+
+```python
+# Add trace_id to enable hierarchical tracing in Portkey
+llm = ChatOpenAI(
+ api_key="dummy",
+ base_url="https://api.portkey.ai/v1",
+ default_headers=createHeaders(
+ api_key="YOUR_PORTKEY_API_KEY",
+ virtual_key="YOUR_LLM_PROVIDER_VIRTUAL_KEY",
+ trace_id="unique-session-id", # Add unique trace ID
+ metadata={"request_type": "user_query"}
+ )
+)
+```
+
+LangGraph also offers its own tracing via LangSmith, which can be used alongside Portkey for even more detailed workflow insights.
+
+
+
+Portkey logs every interaction with LLMs, including:
+
+- Complete request and response payloads
+- Latency and token usage metrics
+- Cost calculations
+- Tool calls and function executions
+
+All logs can be filtered by metadata, trace IDs, models, and more, making it easy to debug specific agent runs.
+
+
+
+Portkey provides built-in dashboards that help you:
+
+- Track cost and token usage across all agent runs
+- Analyze performance metrics like latency and success rates
+- Identify bottlenecks in your agent workflows
+- Compare different agent configurations and LLMs
+
+You can filter and segment all metrics by custom metadata to analyze specific agent types, user groups, or use cases.
+
+
+
+Add custom metadata to your LangGraph agent calls to enable powerful filtering and segmentation:
+
+```python
+llm = ChatOpenAI(
+ api_key="dummy",
+ base_url="https://api.portkey.ai/v1",
+ default_headers=createHeaders(
+ api_key="YOUR_PORTKEY_API_KEY",
+ virtual_key="YOUR_LLM_PROVIDER_VIRTUAL_KEY",
+ metadata={
+ "agent_type": "search_agent",
+ "environment": "production",
+ "_user": "user_123", # Special _user field for user analytics
+ "graph_id": "complex_workflow"
+ }
+ )
+)
+```
+
+This metadata can be used to filter logs, traces, and metrics on the Portkey dashboard, allowing you to analyze specific agent runs, users, or environments.
+
+
+
+This enables:
+- Per-user cost tracking and budgeting
+- Personalized user analytics
+- Team or organization-level metrics
+- Environment-specific monitoring (staging vs. production)
+
+
+
+
+
+
+
+
+
+
+ Official LangGraph documentation
+Official Portkey documentation
+Get personalized guidance on implementing this integration
+
+
+
+
+
+
+
+
+
+
+
+
+ 
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
+- Complete request and response tracking
+- Metadata tags for filtering
+- Cost attribution and much more...
+
+
+
+
+
+
+
+
+### 3. Unified Access to 1600+ LLMs
+
+You can easily switch between 1600+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
+
+
+
+### 5\. [Traces](/product/observability/traces)
+
+With traces, you can see each agent run granularly on Portkey. Tracing your LlamaIndex agent runs helps in debugging, performance optimzation, and visualizing how exactly your agents are running.
+
+### Using Traces in LlamaIndex Agents
+
+#### Step 1: Import & Initialize the Portkey LlamaIndex Callback Handler
+
+
+
+```py
+from portkey_ai.llamaindex import LlamaIndexCallbackHandler
+
+portkey_handler = LlamaIndexCallbackHandler(
+ api_key="YOUR_PORTKEY_API_KEY",
+ metadata={
+ "session_id": "session_1", # Use consistent metadata across your application
+ "agent_id": "research_agent_1", # Specific to the current agent
+ }
+)
+```
+
+#### Step 2: Configure Your LLM with the Portkey Callback
+
+
+
+```py
+from llama_index.llms.openai import OpenAI
+
+llm = OpenAI(
+ api_key="YOUR_OPENAI_API_KEY_HERE",
+ callbacks=[portkey_handler], # Replace with your OpenAI API key
+ # ... other parameters
+)
+```
+
+With Portkey tracing, you can encapsulate the complete execution of your agent workflow.
+
+### 6\. [Continuous Improvement](/product/observability/feedback)
+
+Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
+
+### 7\. [Caching](/product/ai-gateway/cache-simple-and-semantic)
+
+Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
+
+
+
+```py
+{
+ "cache": {
+ "mode": "semantic" // Choose between "simple" or "semantic"
+ }
+}
+```
+
+### 8\. [Security & Compliance](/product/enterprise-offering/security-portkey)
+
+Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
+
+---
+
+## [Portkey Config](/product/ai-gateway/configs)
+
+Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
+
+For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
diff --git a/virtual_key_old/integrations/agents/openai-agents-ts.mdx b/virtual_key_old/integrations/agents/openai-agents-ts.mdx
new file mode 100644
index 00000000..dba87cb7
--- /dev/null
+++ b/virtual_key_old/integrations/agents/openai-agents-ts.mdx
@@ -0,0 +1,923 @@
+---
+title: "OpenAI Agents SDK (TypeScript)"
+description: "Use Portkey with OpenAI Agents SDK to take your AI Agents to production"
+---
+
+## Introduction
+OpenAI Agents SDK enables the development of complex AI agents with tools, planning, and memory capabilities. Portkey enhances OpenAI Agents with observability, reliability, and production-readiness features.
+
+Portkey turns your experimental OpenAI Agents into production-ready systems by providing:
+
+- **Complete observability** of every agent step, tool use, and interaction
+- **Built-in reliability** with fallbacks, retries, and load balancing
+- **Cost tracking and optimization** to manage your AI spend
+- **Access to 200+ LLMs** through a single integration
+- **Guardrails** to keep agent behavior safe and compliant
+- **Version-controlled prompts** for consistent agent performance
+
+
+
+
+Traces provide a hierarchical view of your agent's execution, showing the sequence of LLM calls, tool invocations, and state transitions.
+
+```typescript
+// Add tracing to your OpenAI Agents
+const portkey = new OpenAI({
+ baseURL: PORTKEY_GATEWAY_URL,
+ apiKey: process.env.PORTKEY_API_KEY!,
+ defaultHeaders: createHeaders({
+ traceId: "unique_execution_trace_id", // Add unique trace ID
+ virtualKey: "YOUR_OPENAI_VIRTUAL_KEY"
+ })
+});
+setDefaultOpenAIClient(portkey);
+```
+
+
+
+
+ Portkey logs every interaction with LLMs, including:
+
+ - Complete request and response payloads
+ - Latency and token usage metrics
+ - Cost calculations
+ - Tool calls and function executions
+
+ All logs can be filtered by metadata, trace IDs, models, and more, making it easy to debug specific agent runs.
+
+
+
+ Portkey provides built-in dashboards that help you:
+
+ - Track cost and token usage across all agent runs
+ - Analyze performance metrics like latency and success rates
+ - Identify bottlenecks in your agent workflows
+ - Compare different agent configurations and LLMs
+
+ You can filter and segment all metrics by custom metadata to analyze specific agent types, user groups, or use cases.
+
+
+
+Add custom metadata to your OpenAI agent calls to enable powerful filtering and segmentation:
+
+```typescript
+// Add metadata to your OpenAI Agents
+const portkey = new OpenAI({
+ baseURL: PORTKEY_GATEWAY_URL,
+ apiKey: process.env.PORTKEY_API_KEY!,
+ defaultHeaders: createHeaders({
+ metadata: {"agent_type": "research_agent"}, // Add custom metadata
+ virtualKey: "YOUR_OPENAI_VIRTUAL_KEY"
+ })
+});
+setDefaultOpenAIClient(portkey);
+```
+
+This metadata can be used to filter logs, traces, and metrics on the Portkey dashboard, allowing you to analyze specific agent runs, users, or environments.
+
+
+
+This enables:
+- Per-user cost tracking and budgeting
+- Personalized user analytics
+- Team or organization-level metrics
+- Environment-specific monitoring (staging vs. production)
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Official OpenAI Agents SDK documentation
+Example implementations for various use cases
+Get personalized guidance on implementing this integration
+
+
+
+
+ Traces provide a hierarchical view of your agent's execution, showing the sequence of LLM calls, tool invocations, and state transitions.
+
+ ```python
+ # Add tracing to your OpenAI Agents
+ portkey = AsyncOpenAI(
+ base_url=PORTKEY_GATEWAY_URL,
+ api_key=os.environ["PORTKEY_API_KEY"],
+ default_headers=createHeaders(
+ trace_id="unique_execution_trace_id", # Add unique trace ID
+ virtual_key="YOUR_OPENAI_VIRTUAL_KEY"
+ )
+ )
+ set_default_openai_client(portkey)
+ ```
+
+
+
+
+ Portkey logs every interaction with LLMs, including:
+
+ - Complete request and response payloads
+ - Latency and token usage metrics
+ - Cost calculations
+ - Tool calls and function executions
+
+ All logs can be filtered by metadata, trace IDs, models, and more, making it easy to debug specific agent runs.
+
+
+
+ Portkey provides built-in dashboards that help you:
+
+ - Track cost and token usage across all agent runs
+ - Analyze performance metrics like latency and success rates
+ - Identify bottlenecks in your agent workflows
+ - Compare different agent configurations and LLMs
+
+ You can filter and segment all metrics by custom metadata to analyze specific agent types, user groups, or use cases.
+
+
+
+
+
+ Add custom metadata to your OpenAI agent calls to enable powerful filtering and segmentation:
+ ```python
+
+ # Add tracing to your OpenAI Agents
+ portkey = AsyncOpenAI(
+ base_url=PORTKEY_GATEWAY_URL,
+ api_key=os.environ["PORTKEY_API_KEY"],
+ default_headers=createHeaders(
+ metadata={"agent_type": "research_agent"}, # Add custom metadata
+ virtual_key="YOUR_OPENAI_VIRTUAL_KEY"
+ )
+ )
+ set_default_openai_client(portkey)
+ ```
+
+ This metadata can be used to filter logs, traces, and metrics on the Portkey dashboard, allowing you to analyze specific agent runs, users, or environments.
+
+
+
+
+This enables:
+- Per-user cost tracking and budgeting
+- Personalized user analytics
+- Team or organization-level metrics
+- Environment-specific monitoring (staging vs. production)
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Official OpenAI Agents SDK documentation
+Example implementations for various use cases
+Get personalized guidance on implementing this integration
+Call various LLMs like Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, and AWS Bedrock with minimal code changes.
+Speed up agent responses and save costs by storing past responses in the Portkey cache. Choose between Simple and Semantic cache modes.
+Set up fallbacks between different LLMs, load balance requests across multiple instances, set automatic retries, and request timeouts.
+Get comprehensive logs of agent interactions, including cost, tokens used, response time, and function calls. Send custom metadata for better analytics.
+Access detailed logs of agent executions, function calls, and interactions. Debug and optimize your agents effectively.
+Implement budget limits, role-based access control, and audit trails for your agent operations.
+Capture and analyze user feedback to improve agent performance over time.
+
+
+
+
+#### Send Custom Metadata with your requests
+Add trace IDs to track specific workflows:
+
+
+
+```python
+portkey = Portkey(
+ api_key="YOUR_PORTKEY_API_KEY",
+ virtual_key="YOUR_VIRTUAL_KEY",
+ trace_id="weather_workflow_123",
+ metadata={
+ "agent": "weather_agent",
+ "environment": "production"
+ }
+)
+```
+
+
+
+
+
+## 5. [Logs and Traces](/product/observability/logs)
+
+Logs are essential for understanding agent behavior, diagnosing issues, and improving performance. They provide a detailed record of agent activities and tool use, which is crucial for debugging and optimizing processes.
+
+
+Access a dedicated section to view records of agent executions, including parameters, outcomes, function calls, and errors. Filter logs based on multiple parameters such as trace ID, model, tokens used, and metadata.
+
+
+
+
+
+
+
+## 6. [Security & Compliance - Enterprise-Ready Controls](/product/enterprise-offering/security-portkey)
+
+When deploying agents in production, security is crucial. Portkey provides enterprise-grade security features:
+
+
+
+
+
+
+
diff --git a/virtual_key_old/integrations/agents/phidata.mdx b/virtual_key_old/integrations/agents/phidata.mdx
new file mode 100644
index 00000000..d8d83937
--- /dev/null
+++ b/virtual_key_old/integrations/agents/phidata.mdx
@@ -0,0 +1,180 @@
+---
+title: "Phidata"
+description: "Use Portkey with Phidata to take your AI Agents to production"
+---
+
+## Getting started
+
+### 1\. Install the required packages:
+
+```sh
+pip install phidata portkey-ai
+```
+
+### **2\.** Configure your Phidata LLM objects:
+
+```py
+from phi.llm.openai import OpenAIChat
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
+
+llm = OpenAIChat(
+ base_url=PORTKEY_GATEWAY_URL,
+ api_key="OPENAI_API_KEY", #Replace with Your OpenAI Key
+ default_headers=createHeaders(
+ provider="openai",
+ api_key=PORTKEY_API_KEY # Replace with your Portkey API key
+ )
+)
+```
+
+## Integration Guide
+
+Here's a simple Colab notebook that demonstrates Phidata with Portkey integration
+
+[](https://dub.sh/Phidata-docs)
+
+## Make your agents Production-ready with Portkey
+
+Portkey makes your Phidata agents reliable, robust, and production-grade with its observability suite and AI Gateway. Seamlessly integrate 200+ LLMs with your Phidata agents using Portkey. Implement fallbacks, gain granular insights into agent performance and costs, and continuously optimize your AI operationsโall with just 2 lines of code.
+
+Let's dive deep! Let's go through each of the use cases!
+
+### 1\. [Interoperability](/product/ai-gateway/universal-api)
+
+Easily switch between 200+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `provider ` and `API key` in the `ChatOpenAI` object.
+
+
+
+### 5\. [Continuous Improvement](/product/observability/feedback)
+
+Improve your Agent runs by capturing qualitative & quantitative user feedback on your requests. Portkey's Feedback APIs provide a simple way to get weighted feedback from customers on any request you served, at any stage in your app. You can capture this feedback on a request or conversation level and analyze it by adding meta data to the relevant request.
+
+### 6\. [Caching](/product/ai-gateway/cache-simple-and-semantic)
+
+Agent runs are time-consuming and expensive due to their complex pipelines. Caching can significantly reduce these costs by storing frequently used data and responses. Portkey offers a built-in caching system that stores past responses, reducing the need for agent calls saving both time and money.
+
+```py
+{
+ "cache": {
+ "mode": "semantic" // Choose between "simple" or "semantic"
+ }
+}
+```
+
+### 7\. [Security & Compliance](/product/enterprise-offering/security-portkey)
+
+Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
+
+## [Portkey Config](/product/ai-gateway/configs)
+
+Many of these features are driven by Portkey's Config architecture. The Portkey app simplifies creating, managing, and versioning your Configs.
+
+For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai).
diff --git a/virtual_key_old/integrations/agents/pydantic-ai.mdx b/virtual_key_old/integrations/agents/pydantic-ai.mdx
new file mode 100644
index 00000000..a1aeb76d
--- /dev/null
+++ b/virtual_key_old/integrations/agents/pydantic-ai.mdx
@@ -0,0 +1,1190 @@
+---
+title: "Pydantic AI"
+description: "Use Portkey with PydanticAI to take your AI Agents to production"
+---
+
+## Introduction
+
+PydanticAI is a Python agent framework designed to make it less painful to build production-grade applications with Generative AI. It brings the same ergonomic design and developer experience to GenAI that FastAPI brought to web development.
+
+Portkey enhances PydanticAI with production-readiness features, turning your experimental agents into robust systems by providing:
+
+- **Complete observability** of every agent step, tool use, and interaction
+- **Built-in reliability** with fallbacks, retries, and load balancing
+- **Cost tracking and optimization** to manage your AI spend
+- **Access to 200+ LLMs** through a single integration
+- **Guardrails** to keep agent behavior safe and compliant
+- **Version-controlled prompts** for consistent agent performance
+
+
+
+
+Traces provide a hierarchical view of your agent's execution, showing the sequence of LLM calls, tool invocations, and state transitions.
+
+```python
+# Add trace_id to enable hierarchical tracing in Portkey
+portkey_client = AsyncPortkey(
+ api_key="YOUR_PORTKEY_API_KEY",
+ virtual_key="YOUR_LLM_PROVIDER_VIRTUAL_KEY",
+ trace_id="unique-session-id", # Add unique trace ID
+ metadata={"request_type": "user_query"}
+)
+```
+
+
+
+Portkey logs every interaction with LLMs, including:
+
+- Complete request and response payloads
+- Latency and token usage metrics
+- Cost calculations
+- Tool calls and function executions
+
+All logs can be filtered by metadata, trace IDs, models, and more, making it easy to debug specific agent runs.
+
+
+
+Portkey provides built-in dashboards that help you:
+
+- Track cost and token usage across all agent runs
+- Analyze performance metrics like latency and success rates
+- Identify bottlenecks in your agent workflows
+- Compare different agent configurations and LLMs
+
+You can filter and segment all metrics by custom metadata to analyze specific agent types, user groups, or use cases.
+
+
+
+Add custom metadata to your PydanticAI agent calls to enable powerful filtering and segmentation:
+
+```python
+portkey_client = AsyncPortkey(
+ api_key="YOUR_PORTKEY_API_KEY",
+ virtual_key="YOUR_LLM_PROVIDER_VIRTUAL_KEY",
+ metadata={
+ "agent_type": "weather_agent",
+ "environment": "production",
+ "_user": "user_123", # Special _user field for user analytics
+ "request_source": "mobile_app"
+ }
+)
+```
+
+This metadata can be used to filter logs, traces, and metrics on the Portkey dashboard, allowing you to analyze specific agent runs, users, or environments.
+
+
+
+This enables:
+- Per-user cost tracking and budgeting
+- Personalized user analytics
+- Team or organization-level metrics
+- Environment-specific monitoring (staging vs. production)
+
+
+
+
+
+
+
+
+
+
+ Official PydanticAI documentation
+Official Portkey documentation
+Get personalized guidance on implementing this integration
+
+
+
+Traces provide a hierarchical view of your agent's execution, showing the sequence of LLM calls, tool invocations, and state transitions.
+
+```python {10}
+from strands import Agent
+from strands.models.openai import OpenAIModel
+from strands_tools import calculator
+from portkey_ai import PORTKEY_GATEWAY_URL,createHeaders
+
+model = OpenAIModel(
+ client_args={
+ "api_key": "YOUR_PORTKEY_API_KEY",
+ "base_url": PORTKEY_GATEWAY_URL,
+ "default_headers":createHeaders(trace_id="strands")
+ },
+ model_id="gpt-4o",
+ params={
+ "max_tokens": 1000,
+ "temperature": 0.7,
+ }
+)
+
+agent = Agent(model=model, tools=[calculator])
+response = agent("What is 2+2")
+print(response)
+```
+
+
+
+Portkey logs every interaction with LLMs, including:
+
+- Complete request and response payloads
+- Latency and token usage metrics
+- Cost calculations
+- Tool calls and function executions
+
+All logs can be filtered by metadata, trace IDs, models, and more, making it easy to debug specific agent runs.
+
+
+
+Portkey provides built-in dashboards that help you:
+
+- Track cost and token usage across all agent runs
+- Analyze performance metrics like latency and success rates
+- Identify bottlenecks in your agent workflows
+- Compare different agent configurations and LLMs
+
+You can filter and segment all metrics by custom metadata to analyze specific agent types, user groups, or use cases.
+
+
+
+Add custom metadata to your Strands calls to enable powerful filtering and segmentation:
+
+```python
+from strands import Agent
+from strands.models.openai import OpenAIModel
+from strands_tools import calculator
+from portkey_ai import PORTKEY_GATEWAY_URL,createHeaders
+
+model = OpenAIModel(
+ client_args={
+ "api_key": "YOUR_PORTKEY_API_KEY",
+ "base_url": PORTKEY_GATEWAY_URL,
+ "default_headers":createHeaders(
+ trace_id="strands",
+ metadata={
+ "agent_type": "strands_agent",
+ "environment": "production",
+ "_user": "user_123",
+ "request_source": "mobile_app"
+ }
+ )
+ },
+ model_id="gpt-4o",
+ params={
+ "max_tokens": 1000,
+ "temperature": 0.7,
+ }
+)
+
+agent = Agent(model=model, tools=[calculator])
+response = agent("What is 2+2")
+print(response)
+```
+
+This metadata can be used to filter logs, traces, and metrics on the Portkey dashboard, allowing you to analyze specific agent runs, users, or environments.
+ When your requests exceed quota, catch RateLimitError:
Set timeouts and catch TimeoutError:
Verify your API key and header settings:
+```python +from portkey_ai import exceptions + +try: + response = agent("โฆ") +except exceptions.AuthenticationError: + print("Invalid API key or permissions") +``` +Use a simple exponential backoff:
+```python +import time +from portkey_ai import exceptions + +for attempt in range(3): + try: + result = agent("โฆ") + break + except exceptions.RateLimitError: + time.sleep(2 ** attempt) +``` +Official Portkey documentation
+Get personalized guidance on implementing this integration
+
+
+
+```python {11}
+from strands import Agent
+from strands.models.openai import OpenAIModel
+from strands_tools import calculator
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
+
+model = OpenAIModel(
+ client_args={
+ "api_key": "YOUR_PORTKEY_API_KEY",
+ "base_url": PORTKEY_GATEWAY_URL,
+ # Add trace ID to group related requests
+ "default_headers": createHeaders(trace_id="user_session_123")
+ },
+ model_id="gpt-4o",
+ params={"temperature": 0.7}
+)
+
+agent = Agent(model=model, tools=[calculator])
+response = agent("What's 15% of 2,847?")
+```
+
+All requests from this agent will be grouped under the same trace, making it easy to analyze the complete interaction flow.
+ Get personalized guidance on implementing this integration
+
+
+
+
+### 2\. Add Aporia's Guardrail Check
+
+* Now, navigate to the `Guardrails` page
+* Search for `Validate - Project` Guardrail Check and click on `Add`
+* Input your corresponding Aporia Project ID where you are defining the policies
+* Save the check, set any actions you want on the check, and create the Guardrail!
+
+| Check Name | Description | Parameters | Supported Hooks |
+| :------------------- | :----------------------------------------------------------------------------------- | :------------------ | :------------------------------------ |
+| Validate - Projects | Runs a project containing policies set in Aporia and returns a PASS or FAIL verdict | Project ID: string | beforeRequestHooks afterRequestHooks |
+
+
+ 
+
+Your Aporia Guardrail is now ready to be added to any Portkey request you'd like!
+
+### 3\. Add Guardrail ID to a Config and Make Your Request
+
+* When you save a Guardrail, you'll get an associated Guardrail ID - add this ID to the `before_request_hooks` or `after_request_hooks` params in your Portkey Config
+* Save this Config and pass it along with any Portkey request you're making!
+
+Your requests are now guarded by your Aporia policies and you can see the Verdict and any action you take directly on Portkey logs! More detailed logs for your requests will also be available on your Aporia dashboard.
+
+---
+
+## Get Support
+
+If you face any issues with the Aporia integration, just ping the @Aporia team on the [community forum](https://discord.gg/portkey-llms-in-prod-1143393887742861333).
diff --git a/virtual_key_old/integrations/guardrails/azure-guardrails.mdx b/virtual_key_old/integrations/guardrails/azure-guardrails.mdx
new file mode 100644
index 00000000..8ca7ebac
--- /dev/null
+++ b/virtual_key_old/integrations/guardrails/azure-guardrails.mdx
@@ -0,0 +1,221 @@
+---
+title: "Azure Guardrails"
+description: "Integrate Microsoft Azure's powerful content moderation services & PII guardrails with Portkey"
+---
+
+Microsoft Azure offers robust content moderation and PII readaction services that can now be seamlessly integrated with Portkey's guardrails ecosystem. This integration supports two powerful Azure services:
+
+
+
+
+
+## Using Azure Guardrails - Scenarios
+
+After setting up your guardrails, there are different ways to use them depending on your security requirements:
+
+### Detect and Monitor Only
+
+To simply detect but not block content:
+- Configure your guardrail actions without enabling "Deny"
+- Monitor the guardrail results in your Portkey logs
+- If any issues are detected, the response will include a `hook_results` object with details
+
+### Redact PII Automatically
+
+To automatically remove sensitive information:
+- Enable the `Redact` option for Azure PII Detection
+- When PII is detected, it will be automatically redacted and replaced with standardized identifiers
+- The response will include a `transformed` flag set to `true` in the results
+
+### Block Harmful Content
+
+To completely block requests that violate your policies:
+- Enable the `Deny` option in the guardrails action tab
+- If harmful content is detected, the request will fail with an appropriate status code
+- You can customize denial messages to provide guidance to users
+
+
+---
+
+## Need Support?
+
+If you encounter any issues with Azure Guardrails, please reach out to our support team through the [Portkey community forum](https://discord.gg/portkey-llms-in-prod-1143393887742861333).
diff --git a/virtual_key_old/integrations/guardrails/bedrock-guardrails.mdx b/virtual_key_old/integrations/guardrails/bedrock-guardrails.mdx
new file mode 100644
index 00000000..9e847fdf
--- /dev/null
+++ b/virtual_key_old/integrations/guardrails/bedrock-guardrails.mdx
@@ -0,0 +1,250 @@
+---
+title: "AWS Bedrock Guardrails"
+description: "Secure your AI applications with AWS Bedrock's guardrail capabilities through Portkey."
+---
+
+[AWS Bedrock Guardrails](https://aws.amazon.com/bedrock/) provides a comprehensive solution for securing your LLM applications, including content filtering, PII detection and redaction, and more.
+
+To get started with AWS Bedrock Guardrails, visit their documentation:
+
+
+
+
+In the Guardrail configuration UI, you'll need to provide:
+
+| Field | Description | Type |
+| :-- | :-- | :-- |
+| **Webhook URL** | Your webhook's endpoint URL | `string` |
+| **Headers** | Headers to include with webhook requests | `JSON` |
+| **Timeout** | Maximum wait time for webhook response | `number` (ms) |
+
+#### Webhook URL
+
+This should be a publicly accessible URL where your webhook is hosted.
+
+
+
+
+### 2\. Add Patronus' Guardrail Checks & Actions
+
+Navigate to the `Guardrails` page and you will see the Guardrail Checks offered by Patronus there. Add the ones you want, set actions, and create the Guardrail!
+
+
+
+
+#### List of Patronus Guardrail Checks
+
+| Check Name | Description | Parameters | Supported Hooks |
+| :-------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------ | :----------------- |
+| Retrieval Answer Relevance | Checks whether the answer is on-topic to the input question. Does not measure correctness. | **ON** or **OFF** | afterRequestHooks |
+| Custom Evaluator | Checks against custom criteria, based on Patronus evaluator profile name. | **string**(evaluator's profile name) | afterRequestHooks |
+| Is Concise | Check that the output is clear and concise. | **ON** or **OFF** | afterRequestHooks |
+| Is Helpful | Check that the output is helpful in its tone of voice. | **ON** or **OFF** | afterRequestHooks |
+| Is Polite | Check that the output is polite in conversation. | **ON** or **OFF** | afterRequestHooks |
+| No Apologies | Check that the output does not contain apologies. | **ON** or **OFF** | afterRequestHooks |
+| No Gender Bias | Check whether the output contains gender stereotypes. Useful to mitigate PR risk from sexist or gendered model outputs. | **ON** or **OFF** | afterRequestHooks |
+| No Racias Bias | Check whether the output contains any racial stereotypes or not. | **ON** or **OFF** | afterRequestHooks |
+| Detect Toxicity | Checks output for abusive and hateful messages. | **ON** or **OFF** | afterRequestHooks |
+| Detect PII | Checks for personally identifiable information (PII) - this is information that, in conjunction with other data, can identify an individual. | **ON** or **OFF** | afterRequestHooks |
+| Detect PHI | Checks for protected health information (PHI), defined broadly as any information about an individual's health status or provision of healthcare. | **ON** or **OFF** | afterRequestHooks |
+
+Your Patronus Guardrail is now ready to be added to any Portkey request you'd like!
+
+### 3\. Add Guardrail ID to a Config and Make Your Request
+
+
+
+Your Pillar Guardrail is now ready to be added to any Portkey request you'd like!
+
+### 3\. Add Guardrail ID to a Config and Make Your Request
+
+* When you save a Guardrail, you'll get an associated Guardrail ID - add this ID to the `before_request_hooks` or `after_request_hooks` params in your Portkey Config
+* Save this Config and pass it along with any Portkey request you're making!
+
+Your requests are now guarded by your Pillar checks and you can see the Verdict and any action you take directly on Portkey logs! More detailed logs for your requests will also be available on your Pillar dashboard.
+
+---
+
+## Get Support
+
+If you face any issues with the Pillar integration, just ping the @Pillar team on the [community forum](https://discord.gg/portkey-llms-in-prod-1143393887742861333).
diff --git a/virtual_key_old/integrations/guardrails/prompt-security.mdx b/virtual_key_old/integrations/guardrails/prompt-security.mdx
new file mode 100644
index 00000000..065a4903
--- /dev/null
+++ b/virtual_key_old/integrations/guardrails/prompt-security.mdx
@@ -0,0 +1,120 @@
+---
+title: "Prompt Security"
+description: "Prompt Security detects and protects against prompt injection, sensitive data exposure, and other AI security threats."
+---
+
+[Prompt Security](https://www.prompt.security/solutions/employees) provides advanced protection for your AI applications against various security threats including prompt injections and sensitive data exposure, helping ensure safe interactions with LLMs.
+
+To get started with Prompt Security, visit their website:
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made by Anthropic Computer Use. These logs include:
+- Complete request and response tracking
+- Code context and generation metrics
+- Developer attribution
+- Cost breakdown per coding session
+
+
+
+
+
+### 3. Unified Access to 250+ LLMs
+
+Easily switch between 250+ LLMs for different coding tasks. Use GPT-4 for complex architecture decisions, Claude for detailed code reviews, or specialized models for specific languages - all through a single interface.
+
+### 4. Advanced Metadata Tracking
+Track coding patterns and productivity metrics with custom metadata:
+- Language and framework usage
+- Code generation vs completion tasks
+- Time-of-day productivity patterns
+- Project-specific metrics
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
+- Complete request and response tracking
+- Metadata tags for filtering
+- Cost attribution and much more...
+
+
+
+
+
+
+
+
+### 3. Unified Access to 1600+ LLMs
+
+You can easily switch between 1600+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
+
+
+
+
+Find more information about Autogen here: [https://microsoft.github.io/autogen/docs/Getting-Started](https://microsoft.github.io/autogen/docs/Getting-Started)
+
+## Quick Start Integration
+
+Autogen supports a concept of [config\_list](https://microsoft.github.io/autogen/docs/llm%5Fconfiguration) which allows definitions of the LLM provider and model to be used. Portkey seamlessly integrates into the Autogen framework through a custom config we create.
+
+### Example using minimal configuration
+
+```py
+from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
+
+# Import the portkey library to fetch helper functions
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
+
+config_list = [
+ {
+ "api_key": 'Your OpenAI Key',
+ "model": "gpt-3.5-turbo",
+ "base_url": PORTKEY_GATEWAY_URL,
+ "api_type": "openai",
+ "default_headers": createHeaders(
+ api_key = "Your Portkey API Key",
+ provider = "openai",
+ )
+ }
+]
+
+assistant = AssistantAgent("assistant", llm_config={"config_list": config_list})
+user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False}) # IMPORTANT: set to True to run code in docker, recommended
+user_proxy.initiate_chat(assistant, message="Say this is also a test - part 2.")
+# This initiates an automated chat between the two agents to solve the task
+```
+
+Notice that we updated the `base_url` to Portkey's AI Gateway and then added `default_headers` to enable Portkey specific features.
+
+When we execute this script, it would yield the same results as without Portkey, but every request can now be inspected in the Portkey Analytics & Logs UI - including token, cost, accuracy calculations.
+
+
+ 
+
+All the config parameters supported in Portkey are available for use as part of the headers. Let's look at some examples:
+
+## Using 100+ models in Autogen through Portkey
+
+Since Portkey [seamlessly connects to 150+ models across providers](/integrations/llms), you can easily connect any of these to now run with Autogen.
+
+Let's see an example using **Mistral-7B on Anyscale** running with Autogen seamlessly:
+
+```py
+from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
+
+# Import the portkey library to fetch helper functions
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
+
+config_list = [
+ {
+ "api_key": 'Your Anyscale API Key',
+ "model": "mistralai/Mistral-7B-Instruct-v0.1",
+ "base_url": PORTKEY_GATEWAY_URL,
+ "api_type": "openai", # Portkey conforms to the openai api_type
+ "default_headers": createHeaders(
+ api_key = "Your Portkey API Key",
+ provider = "anyscale",
+ )
+ }
+]
+
+assistant = AssistantAgent("assistant", llm_config={"config_list": config_list})
+user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False}) # IMPORTANT: set to True to run code in docker, recommended
+user_proxy.initiate_chat(assistant, message="Say this is also a test - part 2.")
+# This initiates an automated chat between the two agents to solve the task
+```
+
+## Using a Virtual Key
+
+[Virtual keys](/product/ai-gateway/virtual-keys) in Portkey allow you to easily switch between providers without manually having to store and change their API keys. Let's use the same Mistral example above, but this time using a Virtual Key.
+
+
+
+```py
+from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
+
+# Import the portkey library to fetch helper functions
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
+
+config_list = [
+ {
+ # Set a dummy value, since we'll pick the API key from the virtual key
+ "api_key": 'X',
+
+ # Pick the model from the provider of your choice
+ "model": "mistralai/Mistral-7B-Instruct-v0.1",
+ "base_url": PORTKEY_GATEWAY_URL,
+ "api_type": "openai", # Portkey conforms to the openai api_type
+
+ "default_headers": createHeaders(
+ api_key = "Your Portkey API Key",
+ # Add your virtual key here
+ virtual_key = "Your Anyscale Virtual Key",
+ )
+ }
+]
+
+assistant = AssistantAgent("assistant", llm_config={"config_list": config_list})
+user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False}) # IMPORTANT: set to True to run code in docker, recommended
+user_proxy.initiate_chat(assistant, message="Say this is also a test - part 2.")
+# This initiates an automated chat between the two agents to solve the task
+```
+
+## Using Configs
+
+[Configs](/product/ai-gateway/configs) in Portkey unlock advanced management and routing functionality including [load balancing](/product/ai-gateway/load-balancing), [fallbacks](/product/ai-gateway/fallbacks), [canary testing](/product/ai-gateway/canary-testing), [switching models](/product/ai-gateway/universal-api) and more.
+
+You can use Portkey configs in Autogen like this:
+
+```py
+from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
+
+# Import the portkey library to fetch helper functions
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
+
+config_list = [
+ {
+ # Set a dummy value, since we'll pick the API key from the virtual key
+ "api_key": 'X',
+
+ # Pick the model from the provider of your choice
+ "model": "mistralai/Mistral-7B-Instruct-v0.1",
+ "base_url": PORTKEY_GATEWAY_URL,
+ "api_type": "openai", # Portkey conforms to the openai api_type
+
+ "default_headers": createHeaders(
+ api_key = "Your Portkey API Key",
+ # Add your Portkey config id
+ config = "Your Config ID",
+ )
+ }
+]
+
+assistant = AssistantAgent("assistant", llm_config={"config_list": config_list})
+user_proxy = UserProxyAgent("user_proxy", code_execution_config={"work_dir": "coding", "use_docker": False}) # IMPORTANT: set to True to run code in docker, recommended
+user_proxy.initiate_chat(assistant, message="Say this is also a test - part 2.")
+# This initiates an automated chat between the two agents to solve the task
+```
diff --git a/virtual_key_old/integrations/libraries/claude-code.mdx b/virtual_key_old/integrations/libraries/claude-code.mdx
new file mode 100644
index 00000000..ab1ed1f5
--- /dev/null
+++ b/virtual_key_old/integrations/libraries/claude-code.mdx
@@ -0,0 +1,342 @@
+---
+title: 'Claude Code'
+description: 'Integrate Portkey with Claude Code for enterprise-grade AI coding assistance with observability and governance'
+---
+
+Claude Code is Anthropic's agentic coding tool that lives in your terminal, understands your codebase, and helps you code faster through natural language commands.
+
+With Portkey integration, you can enhance Claude Code with enterprise features:
+
+- **Unified AI Gateway** - Route Claude Code through multiple providers (Anthropic, Bedrock, Vertex AI)
+- **Centralized Logs** - Track all Claude Code interactions
+- **Security & Governance** - Set rate limits, implement access controls, and audit all AI usage
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made by Cline. These logs include:
+- Complete request and response tracking
+- Code context and generation metrics
+- Developer attribution
+- Cost breakdown per coding session
+
+
+
+
+
+### 3. Unified Access to 250+ LLMs
+
+Easily switch between 250+ LLMs for different coding tasks. Use GPT-4 for complex architecture decisions, Claude for detailed code reviews, or specialized models for specific languages - all through a single interface.
+
+### 4. Advanced Metadata Tracking
+Track coding patterns and productivity metrics with custom metadata:
+- Language and framework usage
+- Code generation vs completion tasks
+- Time-of-day productivity patterns
+- Project-specific metrics
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
+- Complete request and response tracking
+- Metadata tags for filtering
+- Cost attribution and much more...
+
+
+
+
+
+
+
+
+### 3. Unified Access to 250+ LLMs
+
+You can easily switch between 250+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual_key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
+
+
+
+
+```python
+import dspy
+
+# Set up the Portkey-enabled client (as shown in the Getting Started section)
+
+class QA(dspy.Signature):
+ """Given the question, generate the answer"""
+ question = dspy.InputField(desc="User's question")
+ answer = dspy.OutputField(desc="often between 1 and 3 words")
+
+dspy.settings.configure(lm=turbo)
+
+predict = dspy.Predict(QA)
+
+# Make a prediction
+prediction = predict(question="Who won the Golden Boot in the 2022 FIFA World Cup?")
+print(prediction.answer)
+```
+
+When you make a request using Portkey with DSPy, you can view detailed information about the request in the Portkey dashboard. Here's what you'll see:
+
+
+ 
+
+
+* `Request Details`: Information about the specific request, including the model used, input, and output.
+* `Metrics`: Performance metrics such as latency, token usage, and cost.
+* `Logs`: Detailed logs of the request, including any errors or warnings.
+* `Traces`: A visual representation of the request flow, especially useful for complex DSPy modules.
+
+## Portkey Features with DSPy
+
+### 1. Interoperability
+
+Portkey's Unified API enables you to easily switch between **250**+ language models. This includes the LLMs that are not natively integrated with DSPy. Here's how you can modify your DSPy setup to use Claude from Gpt-4 model:
+
+
+
+
+### 3. Metrics
+
+Portkey's Observability suite helps you track key metrics like **cost** and **token** usage, which is crucial for managing the high cost of DSPy. The observability dashboard helps you track 40+ key metrics, giving you detailed insights into your DSPy run.
+
+
+ 
+
+
+### 4. Caching
+
+Caching can significantly reduce these costs by storing frequently used data and responses. While DSPy has built-in simple caching, Portkey also offers advanced semantic caching to help you save more time and money.
+
+Just modify your Portkey config as shown below and pass it with the `config` key in the `default_headers` param:
+
+```python
+config={ "cache": { "mode": "semantic" } }
+
+turbo = dspy.OpenAI(
+ api_base=PORTKEY_GATEWAY_URL + "/",
+ model='gpt-4o',
+ api_key="YOUR_OPENAI_API_KEY", # Enter your OpenAI API key
+ model_type="chat",
+ default_headers=createHeaders(
+ api_key="YOUR_PORTKEY_API_KEY",
+ metadata={'_user': "dspy"},
+ provider="openai",
+ config=config
+ )
+)
+dspy.settings.configure(lm=turbo)
+```
+
+### 5. Reliability
+
+Portkey offers built-in **fallbacks** between different LLMs or providers, **load-balancing** across multiple instances or API keys, and implementing automatic **retries** and request **timeouts**. This makes your DSPy more reliable and resilient.
+
+Similar to caching example above, just define your Config and pass it with the `Config` key in the `default_headers` param.
+
+```json
+{
+ "retry": {
+ "attempts": 5
+ },
+ "strategy": {
+ "mode": "loadbalance" // Choose between "loadbalance" or "fallback"
+ },
+ "targets": [
+ {
+ "provider": "openai",
+ "api_key": "OpenAI_API_Key"
+ },
+ {
+ "provider": "anthropic",
+ "api_key": "Anthropic_API_Key"
+ }
+ ]
+}
+```
+
+### 6. Virtual Keys
+
+Securely store your LLM API keys in Portkey vault and get a disposable virtual key with custom budget limits.
+
+Add your API key in Portkey UI [here](https://app.portkey.ai/) to get a virtual key, and pass it in your request like this:
+
+```python
+turbo = dspy.OpenAI(
+ api_base=PORTKEY_GATEWAY_URL + "/",
+ model='gpt-4o',
+ api_key="xx",
+ model_type="chat",
+ default_headers=createHeaders(
+ api_key="YOUR_PORTKEY_API_KEY",
+ virtual_key="MY_OPENAI_VIRTUAL_KEY"
+ )
+)
+dspy.settings.configure(lm=turbo)
+```
+
+## Advanced Examples
+
+### Retrieval-Augmented Generation (RAG) system
+
+Make your RAG prompts better with Portkey x DSPy
+
+
+
+
+
+## Troubleshoot - Missing LLM Calls in Traces
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made by Goose. These logs include:
+- Complete request and response tracking for debugging
+- Metadata tags for filtering by team or project
+- Cost attribution per task
+- Complete conversation history with the AI agent
+
+
+
+
+
+
+
+
+### 3. Unified Access to 1600+ LLMs
+
+You can easily switch between 1600+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across engineering teams and projects.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
+- Complete request and response tracking
+- Metadata tags for filtering
+- Cost attribution and much more...
+
+
+
+
+
+
+
+
+### 3. Unified Access to 1600+ LLMs
+
+You can easily switch between 1600+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
+
+
+
+
+## Using Virtual Keys for Multiple Models
+
+Portkey supports [Virtual Keys](/product/ai-gateway/virtual-keys) which are an easy way to store and manage API keys in a secure vault. Lets try using a Virtual Key to make LLM calls.
+
+#### 1\. Create a Virtual Key in your Portkey account and the id
+
+Let's try creating a new Virtual Key for Mistral like this
+
+
+ 
+
+
+#### 2\. Use Virtual Keys in the Portkey Headers
+
+The `virtualKey` parameter sets the authentication and provider for the AI provider being used. In our case we're using the Mistral Virtual key.
+
+
+
+
+This is extremely powerful since we gain control and visibility over the agent flows so we can identify problems and make updates as needed.
diff --git a/virtual_key_old/integrations/libraries/langchain-python.mdx b/virtual_key_old/integrations/libraries/langchain-python.mdx
new file mode 100644
index 00000000..1e5e8e85
--- /dev/null
+++ b/virtual_key_old/integrations/libraries/langchain-python.mdx
@@ -0,0 +1,420 @@
+---
+title: "Langchain (Python)"
+description: "Supercharge Langchain apps with Portkey: Multi-LLM, observability, caching, reliability, and prompt management."
+---
+
+
+
+
+This setup enables Portkey's advanced features for Langchain.
+
+## Key Portkey Features for Langchain
+
+Routing Langchain requests via Portkey's `ChatOpenAI` interface unlocks powerful capabilities:
+
+Use `ChatOpenAI` for OpenAI, Anthropic, Gemini, Mistral, and more. Switch providers easily with Virtual Keys or Configs.
+Reduce latency and costs with Portkey's Simple, Semantic, or Hybrid caching, enabled via Configs.
+Build robust apps with retries, timeouts, fallbacks, and load balancing, configured in Portkey.
+Get deep insights: LLM usage, costs, latency, and errors are automatically logged in Portkey.
+Manage, version, and use prompts from Portkey's Prompt Library within Langchain.
+Securely manage LLM provider API keys using Portkey Virtual Keys in your Langchain setup.
+
+
+
+---
+
+## 5. Prompt Management
+
+Portkey's Prompt Library helps manage prompts effectively:
+
+* **Version Control:** Store and track prompt changes.
+* **Parameterized Prompts:** Use variables with [mustache templating](/product/prompt-library/prompt-templates#templating-engine).
+* **Sandbox:** Test prompts with different LLMs in Portkey.
+
+### Using Portkey Prompts in Langchain
+
+1. Create prompt in Portkey, get `Prompt ID`.
+2. Use Portkey SDK to render prompt with variables.
+3. Transform rendered prompt to Langchain message format.
+4. Pass messages to Portkey-configured `ChatOpenAI`.
+
+```python
+import os
+from langchain_openai import ChatOpenAI
+from langchain_core.messages import SystemMessage, HumanMessage
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders, Portkey
+
+PORTKEY_API_KEY = os.environ.get("PORTKEY_API_KEY")
+client = Portkey(api_key=PORTKEY_API_KEY)
+
+PROMPT_ID = "pp-story-generator" # Your Portkey Prompt ID
+
+rendered_prompt = client.prompts.render(
+ prompt_id=PROMPT_ID,
+ variables={"character": "brave knight", "object": "magic sword"}
+).data
+
+langchain_messages = []
+if rendered_prompt and rendered_prompt.prompt:
+ for msg in rendered_prompt.prompt:
+ if msg.get("role") == "user": langchain_messages.append(HumanMessage(content=msg.get("content")))
+ elif msg.get("role") == "system": langchain_messages.append(SystemMessage(content=msg.get("content")))
+
+OPENAI_VIRTUAL_KEY = os.environ.get("OPENAI_VIRTUAL_KEY")
+portkey_headers = createHeaders(api_key=PORTKEY_API_KEY, virtual_key=OPENAI_VIRTUAL_KEY)
+
+llm_portkey_prompt = ChatOpenAI(
+ api_key="placeholder_key",
+ base_url=PORTKEY_GATEWAY_URL,
+ default_headers=portkey_headers,
+ model=rendered_prompt.model if rendered_prompt and rendered_prompt.model else "gpt-4o"
+)
+
+# if langchain_messages: response = llm_portkey_prompt.invoke(langchain_messages)
+```
+
+Manage prompts centrally in Portkey for versioning and collaboration.
+
+---
+
+## 6. Secure Virtual Keys
+
+Portkey's [Virtual Keys](/product/ai-gateway/virtual-keys) are vital for secure, flexible LLM ops with Langchain.
+
+**Benefits:**
+* **Secure Credentials:** Store provider API keys in Portkey's vault. Code uses Virtual Key IDs.
+* **Easy Configuration:** Switch providers/keys by changing `virtual_key` in `createHeaders`.
+* **Access Control:** Manage Virtual Key permissions in Portkey.
+* **Auditability:** Track usage via Portkey logs.
+
+Using Virtual Keys boosts security and simplifies config management.
+
+---
+
+## Langchain Embeddings
+
+Create embeddings with `OpenAIEmbeddings` via Portkey.
+
+```python
+from langchain_openai import OpenAIEmbeddings
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
+import os
+
+PORTKEY_API_KEY = os.environ.get("PORTKEY_API_KEY")
+OPENAI_EMBEDDINGS_VIRTUAL_KEY = os.environ.get("OPENAI_EMBEDDINGS_VIRTUAL_KEY")
+
+portkey_headers = createHeaders(api_key=PORTKEY_API_KEY, virtual_key=OPENAI_EMBEDDINGS_VIRTUAL_KEY)
+
+embeddings_model = OpenAIEmbeddings(
+ api_key="placeholder_key",
+ base_url=PORTKEY_GATEWAY_URL,
+ default_headers=portkey_headers,
+ model="text-embedding-3-small"
+)
+
+# embeddings = embeddings_model.embed_documents(["Hello world!", "Test."])
+```
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
+- Complete request and response tracking
+- Metadata tags for filtering
+- Cost attribution and much more...
+
+
+
+
+
+
+
+
+### 3. Unified Access to 1600+ LLMs
+
+You can easily switch between 1600+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
+
+Call various LLMs like Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, and AWS Bedrock with minimal code changes.
+Speed up your requests and save money on LLM calls by storing past responses in the Portkey cache. Choose between Simple and Semantic cache modes.
+Set up fallbacks between different LLMs or providers, load balance your requests across multiple instances or API keys, set automatic retries, and request timeouts.
+Portkey automatically logs all the key details about your requests, including cost, tokens used, response time, request and response bodies, and more. Send custom metadata and trace IDs for better analytics and debugging.
+Use Portkey as a centralized hub to store, version, and experiment with prompts across multiple LLMs, and seamlessly retrieve them in your LlamaIndex app for easy integration.
+Improve your LlamaIndex app by capturing qualitative & quantitative user feedback on your requests.
+Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
+
+
+## Overriding a Saved Config
+
+If you want to use a saved Config from the Portkey app in your LlamaIndex code but need to modify certain parts of it before making a request, you can easily achieve this using Portkey's Configs API. This approach allows you to leverage the convenience of saved Configs while still having the flexibility to adapt them to your specific needs.
+
+#### Here's an example of how you can fetch a saved Config using the Configs API and override the `model` parameter:
+
+```py Overriding Model in a Saved Config
+from llama_index.llms.openai import OpenAI
+from llama_index.core.llms import ChatMessage
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
+import requests
+import os
+
+def create_config(config_slug,model):
+ url = f'https://api.portkey.ai/v1/configs/{config_slug}'
+ headers = {
+ 'x-portkey-api-key': os.environ.get("PORTKEY_API_KEY"),
+ 'content-type': 'application/json'
+ }
+ response = requests.get(url, headers=headers).json()
+ config = json.loads(response['config'])
+ config['override_params']['model']=model
+ return config
+
+config=create_config("pc-llamaindex-xx","gpt-4-turbo")
+
+portkey = OpenAI(
+ api_base=PORTKEY_GATEWAY_URL,
+ api_key="xx" # Placeholder, no need to set
+ default_headers=createHeaders(
+ api_key=os.environ.get("PORTKEY_API_KEY"),
+ config=config
+ )
+)
+
+messages = [ChatMessage(role="user", content="1729")]
+
+resp = portkey.chat(messages)
+print(resp)
+```
+
+In this example:
+
+1. We define a helper function `get_customized_config` that takes a `config_slug` and a `model` as parameters.
+2. Inside the function, we make a GET request to the Portkey Configs API endpoint to fetch the saved Config using the provided `config_slug`.
+3. We extract the `config` object from the API response.
+4. We update the `model` parameter in the `override_params` section of the Config with the provided `custom_model`.
+5. Finally, we return the customized Config.
+
+We can then use this customized Config when initializing the OpenAI client from LlamaIndex, ensuring that our specific `model` override is applied to the saved Config.
+
+For more details on working with Configs in Portkey, refer to the [**Config documentation**.](/product/ai-gateway/configs)
+
+---
+
+## 1\. Interoperability - Calling Anthropic, Gemini, Mistral, and more
+
+Now that we have the OpenAI code up and running, let's see how you can use Portkey to send the request across multiple LLMs - we'll show **Anthropic**, **Gemini**, and **Mistral**. For the full list of providers & LLMs supported, check out [**this doc**](/guides/integrations).
+
+Switching providers just requires **changing 3 lines of code:**
+
+1. Change the `provider name`
+2. Change the `API key`, and
+3. Change the `model name`
+
+
+
+
+
+[**Check out Observability docs here.**](/product/observability)
+
+
+## 5\. Prompt Management
+
+Portkey features an advanced Prompts platform tailor-made for better prompt engineering. With Portkey, you can:
+
+* **Store Prompts with Access Control and Version Control:** Keep all your prompts organized in a centralized location, easily track changes over time, and manage edit/view permissions for your team.
+* **Parameterize Prompts**: Define variables and [mustache-approved tags](/product/prompt-library/prompt-templates#templating-engine) within your prompts, allowing for dynamic value insertion when calling LLMs. This enables greater flexibility and reusability of your prompts.
+* **Experiment in a Sandbox Environment**: Quickly iterate on different LLMs and parameters to find the optimal combination for your use case, without modifying your LlamaIndex code.
+
+#### Here's how you can leverage Portkey's Prompt Management in your LlamaIndex application:
+
+1. Create your prompt template on the Portkey app, and save it to get an associated `Prompt ID`
+2. Before making a Llamaindex request, render the prompt template using the Portkey SDK
+3. Transform the retrieved prompt to be compatible with LlamaIndex and send the request!
+
+#### Example: Using a Portkey Prompt Template in LlamaIndex
+
+```py Portkey Prompts in LlamaIndex
+import json
+import os
+from llama_index.llms.openai import OpenAI
+from llama_index.core.llms import ChatMessage
+from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders, Portkey
+
+### Initialize Portkey client with API key
+
+client = Portkey(api_key=os.environ.get("PORTKEY_API_KEY"))
+
+### Render the prompt template with your prompt ID and variables
+
+prompt_template = client.prompts.render(
+ prompt_id="pp-prompt-id",
+ variables={ "movie":"Dune 2" }
+).data.dict()
+
+config = {
+ "virtual_key":"GROQ_VIRTUAL_KEY", # You need to send the virtual key separately
+ "override_params":{
+ "model":prompt_template["model"], # Set the model name based on the value in the prompt template
+ "temperature":prompt_template["temperature"] # Similarly, you can also set other model params
+ }
+}
+
+portkey = OpenAI(
+ api_base=PORTKEY_GATEWAY_URL,
+ api_key="xx" # Placeholder, no need to set
+ default_headers=createHeaders(
+ api_key=os.environ.get("PORTKEY_API_KEY"),
+ config=config
+ )
+)
+
+### Transform the rendered prompt into LlamaIndex-compatible format
+
+messages = [ChatMessage(content=msg["content"], role=msg["role"]) for msg in prompt_template["messages"]]
+
+resp = portkey.chat(messages)
+
+print(resp)
+```
+
+[**Explore Prompt Management docs here**](/product/prompt-library).
+
+---
+
+## 6\. Continuous Improvement
+
+Now that you know how to trace & log your Llamaindex requests to Portkey, you can also start capturing user feedback to improve your app!
+
+You can append qualitative as well as quantitative feedback to any `trace ID` with the `portkey.feedback.create` method:
+
+```py Adding Feedback
+from portkey_ai import Portkey
+
+portkey = Portkey(
+ api_key="PORTKEY_API_KEY"
+)
+
+feedback = portkey.feedback.create(
+ trace_id="YOUR_LLAMAINDEX_TRACE_ID",
+ value=5, # Integer between -10 and 10
+ weight=1, # Optional
+ metadata={
+ # Pass any additional context here like comments, _user and more
+ }
+)
+
+print(feedback)
+```
+
+[**Check out the Feedback documentation for a deeper dive**](/product/observability/feedback).
+
+
+## 7\. Security & Compliance
+
+When you onboard more team members to help out on your Llamaindex app - permissioning, budgeting, and access management can become a mess! Using Portkey, you can set **budget limits** on provide API keys and implement **fine-grained user roles** and **permissions** to:
+
+* **Control access**: Restrict team members' access to specific features, Configs, or API endpoints based on their roles and responsibilities.
+* **Manage costs**: Set budget limits on API keys to prevent unexpected expenses and ensure that your LLM usage stays within your allocated budget.
+* **Ensure compliance**: Implement strict security policies and audit trails to maintain compliance with industry regulations and protect sensitive data.
+* **Simplify onboarding**: Streamline the onboarding process for new team members by assigning them appropriate roles and permissions, eliminating the need to share sensitive API keys or secrets.
+* **Monitor usage**: Gain visibility into your team's LLM usage, track costs, and identify potential security risks or anomalies through comprehensive monitoring and reporting.
+
+
+ 
+
+
+[**Read more about Portkey's Security & Enterprise offerings here**](/product/enterprise-offering).
+
+
+## Join Portkey Community
+
+Join the Portkey Discord to connect with other practitioners, discuss your LlamaIndex projects, and get help troubleshooting your queries.
+
+[**Link to Discord**](https://portkey.ai/community)
+
+For more detailed information on each feature and how to use them, please refer to the [Portkey Documentation](https://portkey.ai/docs).
diff --git a/virtual_key_old/integrations/libraries/microsoft-semantic-kernel.mdx b/virtual_key_old/integrations/libraries/microsoft-semantic-kernel.mdx
new file mode 100644
index 00000000..96eb25f0
--- /dev/null
+++ b/virtual_key_old/integrations/libraries/microsoft-semantic-kernel.mdx
@@ -0,0 +1,4 @@
+---
+title: Microsoft Semantic Kernel
+url: /api-reference/inference-api/sdks/c-sharp#using-portkey-with-semantic-kernel
+---
diff --git a/virtual_key_old/integrations/libraries/mindsdb.mdx b/virtual_key_old/integrations/libraries/mindsdb.mdx
new file mode 100644
index 00000000..6ed0a397
--- /dev/null
+++ b/virtual_key_old/integrations/libraries/mindsdb.mdx
@@ -0,0 +1,80 @@
+---
+title: "MindsDb"
+description: "Integrate MindsDB with Portkey to build enterprise-grade AI use-cases"
+---
+
+MindsDB connects to various data sources and LLMs, bringing data and AI together for easy AI automation.
+
+With Portkey, you can run MindsDB AI systems with 250+ LLMs and implement enterprise-grade features like [LLM observability](/product/observability), [caching](/product/ai-gateway/cache-simple-and-semantic), [advanced routing](/product/ai-gateway), and more to build production-grade MindsDB AI apps.
+
+## Prerequisites
+
+Before proceeding, ensure the following prerequisites are met:
+
+1. Install MindsDB locally via [Docker](https://docs.mindsdb.com/setup/self-hosted/docker) or [Docker Desktop](https://docs.mindsdb.com/setup/self-hosted/docker-desktop).
+2. To use Portkey within MindsDB, install the required dependencies following [this instruction](https://docs.mindsdb.com/setup/self-hosted/docker#install-dependencies).
+3. Obtain the [Portkey API key](https://app.portkey.ai) required to deploy and use Portkey within MindsDB.
+
+## Setup
+
+
+
+
+
+
+
+
+
+
+
+
+3. In your workflow, configure the OpenAI node to use your preferred model
+ - The model parameter in your config will override the default model in your n8n workflow
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
+- Complete request and response tracking
+- Metadata tags for filtering
+- Cost attribution and much more...
+
+
+
+
+
+
+
+
+### 3. Unified Access to 1600+ LLMs
+
+You can easily switch between 1600+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual_key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
+- Complete request and response tracking
+- Metadata tags for filtering
+- Cost attribution and much more...
+
+
+
+
+
+
+
+
+### 3. Unified Access to 250+ LLMs
+
+You can easily switch between 250+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
+
+
+
+
+## 2. Setting Up Portkey Pipeline
+
+To use OpenWebUI with Portkey, you'll need to configure three key components:
+
+**Portkey API Key**:
+Get your Portkey API key from [here](https://app.portkey.ai/api-keys). You'll need this for authentication with Portkey's services.
+
+**Virtual Keys**:
+Virtual Keys are Portkey's secure way to manage your LLM provider API keys. They provide essential controls like:
+- Budget limits for API usage
+- Rate limiting capabilities
+- Secure API key storage
+
+
+Craeate a [Virtual Key](https://app.portkey.ai/virtual-keys) in your Portkey dashboard and save it for future use.
+
+For detailed information on budget limits, [refer to this documentation](/product/ai-gateway/virtual-keys/budget-limits)
+
+**Using Configs (Optional)**:
+Configs in Portkey enhance your implementation with features like advanced routing, fallbacks, and retries. Here's a simple config example that implements 5 retry attempts on server errors:
+
+```json
+{
+ "retry": {
+ "attempts": 5
+ },
+ "virtual_key": "virtual-key-xxx"
+}
+```
+You can create and store these configs in Portkey's config library. This can later be accessed on using the Config Slug in Open WebUI.
+
+Configs are highly flexible and can be customized for various use cases. Learn more in our [Configs documentation](https://docs.portkey.ai/configs).
+
+
+## 3. Configure Pipeline Variables
+
+The pipeline setup involves configuring both credentials and model access in OpenWebUI.
+
+**Credentials Setup**:
+1. In OpenWebUI, navigate to `Workspace` โ `Functions`
+2. Click the `Valves` button to open the configuration interface
+3. Add the following credentials:
+ - Your Portkey API Key
+ - Config slug (if using Configs)
+ - Base URL (only needed for Open Source Gateway users)
+
+
+
+
+
+**Model Configuration**
+1. In the Functions section, click the `...` button and select `Edit`
+2. Find the virtual keys JSON object in the Portkey function code
+3. Update it with your virtual keys:
+ ```json
+ "virtual_keys": {
+ "openai": "YOUR_OPENAI_VIRTUAL_KEY",
+ "anthropic": "YOUR_ANTHROPIC_VIRTUAL_KEY"
+ }
+ ```
+4. Configure model names in the pipe function in this format:
+ ```json
+ {
+ "id": "provider_slug_from_portkey/model_id_from_provider", // syntax for ID
+ "name": "provider_slug_from_portkey/model_id_from_provider", // for easier navigation
+ }
+
+ ```
+ Example:
+```json
+ {
+ "id": "openai/gpt-4o",
+ "name": "openai/gpt-4o",
+ }
+ ```
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
+- Complete request and response tracking
+- Metadata tags for filtering
+- Cost attribution and much more...
+
+
+
+
+
+
+
+
+### 3. Unified Access to 1600+ LLMs
+
+You can easily switch between 1600+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
+
+
+
+
+---
+
+## 4\. Avoid Promptfoo Rate Limits & Leverage Cache
+
+Since promptfoo can make a lot of calls very quickly, you can use a loadbalanced config in Portkey with cache enabled. You can pass the config header similar to virtual keys in promptfoo.
+
+Here's a sample Config that you can save in the Portkey UI and get a respective config slug:
+
+```json
+{
+ "cache": { "mode": "simple" },
+ "strategy": { "mode": "loadbalance" },
+ "targets": [
+ { "virtual_key": "ACCOUNT_ONE" },
+ { "virtual_key": "ACCOUNT_TWO" },
+ { "virtual_key": "ACCOUNT_THREE" }
+ ]
+}
+```
+
+And then we can just add the saved Config's slug in the YAML:
+
+```yaml
+providers:
+ id: portkey:claude-3-opus20240229
+ config:
+ portkeyVirtualKey: ANTHROPIC_VIRTUAL_KEY
+ portkeyConfig: PORTKEY_CONFIG_SLUG
+```
+
+---
+
+##
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made by Roo. These logs include:
+- Complete request and response tracking
+- Code context and generation metrics
+- Developer attribution
+- Cost breakdown per coding session
+
+
+
+
+
+### 3. Unified Access to 250+ LLMs
+
+Easily switch between 250+ LLMs for different coding tasks. Use GPT-4 for complex architecture decisions, Claude for detailed code reviews, or specialized models for specific languages - all through a single interface.
+
+### 4. Advanced Metadata Tracking
+Track coding patterns and productivity metrics with custom metadata:
+- Language and framework usage
+- Code generation vs completion tasks
+- Time-of-day productivity patterns
+- Project-specific metrics
+
+
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+ 
+
+
+
+
+### Reliability
+
+Portkey enhances the robustness of your AI applications with built-in features such as [Caching](/product/ai-gateway/cache-simple-and-semantic), [Fallback](/product/ai-gateway/fallbacks) mechanisms, [Load balancing](/product/ai-gateway/load-balancing), [Conditional routing](/product/ai-gateway/conditional-routing), [Request timeouts](/product/ai-gateway/request-timeouts), etc.
+
+Here is how you can modify your config to include the following Portkey features-
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+### 2. Advanced Logs
+Portkey's logging dashboard provides detailed logs for every request made to your LLMs. These logs include:
+- Complete request and response tracking
+- Metadata tags for filtering
+- Cost attribution and much more...
+
+
+
+
+
+
+
+
+### 3. Unified Access to 1600+ LLMs
+
+You can easily switch between 1600+ LLMs. Call various LLMs such as Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, AWS Bedrock, and many more by simply changing the `virtual key` in your default `config` object.
+
+
+### 4. Advanced Metadata Tracking
+Using Portkey, you can add custom metadata to your LLM requests for detailed tracking and analytics. Use metadata tags to filter logs, track usage, and attribute costs across departments and teams.
+
+
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+ 
+
+
+
+
+
+In the individual log for any request, you can also see the exact status of your request and verify if it was cached, or delivered from cache with two `usage` parameters:
+
+* `cache_creation_input_tokens`: Number of tokens written to the cache when creating a new entry.
+* `cache_read_input_tokens`: Number of tokens retrieved from the cache for this request.
+
+
+ 
+
diff --git a/virtual_key_old/integrations/llms/anyscale-llama2-mistral-zephyr.mdx b/virtual_key_old/integrations/llms/anyscale-llama2-mistral-zephyr.mdx
new file mode 100644
index 00000000..743bec60
--- /dev/null
+++ b/virtual_key_old/integrations/llms/anyscale-llama2-mistral-zephyr.mdx
@@ -0,0 +1,356 @@
+---
+title: "Anyscale"
+description: "Integrate Anyscale endpoints with Portkey seamlessly and make your OSS models production-ready"
+---
+
+Portkey's suite of features - AI gateway, observability, prompt management, and continuous fine-tuning are all enabled for the OSS models (Llama2, Mistral, Zephyr, and more) available on Anyscale endpoints.
+
+
+ 
+
+
+ 
+
+
+ 
+
+
+
+**Here's a step-by-step guide:**
+
+1. Request access to Azure OpenAI [here](https://aka.ms/oai/access).
+2. Create a resource in the Azure portal [here](https://portal.azure.com/?microsoft%5Fazure%5Fmarketplace%5FItemHideKey=microsoft%5Fopenai%5Ftip#create/Microsoft.CognitiveServicesOpenAI). (This will be your **Resource Name**)
+3. Deploy a model in Azure OpenAI Studio [here](https://oai.azure.com/). (This will be your **Deployment Name)**
+4. Select your `Foundation Model` from the dropdowon on the modal.
+5. Now, on Azure OpenAI studio, go to any playground (chat or completions), click on a UI element called "View code". Note down the API version & API key from here. (This will be your **Azure API Version** & **Azure API Key**)
+
+When you input these details, the foundation model will be auto populated. More details in [this guide](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal).
+
+
+
+
+Log view for an image generation request on Azure OpenAI
+
+More information on image generation is available in the [API Reference](https://portkey.ai/docs/api-reference/completions-1#create-image).
+
+---
+
+## Making Requests Without Virtual Keys
+
+Here's how you can pass your Azure OpenAI details & secrets directly without using the Virtual Keys feature.
+
+### Key Mapping
+
+In a typical Azure OpenAI request,
+
+
+
+```sh
+curl https://{YOUR_RESOURCE_NAME}.openai.azure.com/openai/deployments/{YOUR_DEPLOYMENT_NAME}/chat/completions?api-version={API_VERSION} \
+ -H "Content-Type: application/json" \
+ -H "api-key: {YOUR_API_KEY}" \
+ -d '{
+ "model": "gpt-4o",
+ "messages": [
+ {
+ "role": "system",
+ "content": "You are a helpful assistant"
+ },
+ {
+ "role": "user",
+ "content": "what is a portkey?"
+ }
+ ]
+}'
+```
+
+| Parameter | Node SDK | Python SDK | REST Headers |
+| --------------------- | ------------------------------------ | ------------------------------------- | ----------------------------- |
+| AZURE RESOURCE NAME | azureResourceName | azure_resource_name | x-portkey-azure-resource-name |
+| AZURE DEPLOYMENT NAME | azureDeploymentId | azure_deployment_id | x-portkey-azure-deployment-id |
+| API VERSION | azureApiVersion | azure_api_version | x-portkey-azure-api-version |
+| AZURE API KEY | Authorization: "Bearer + {API_KEY}" | Authorization = "Bearer + {API_KEY}" | Authorization |
+| AZURE MODEL NAME | azureModelName | azure_model_name | x-portkey-azure-model-name |
+
+### Example
+
+
+
+
+* On the same [page](https://us-east-1.console.aws.amazon.com/iam/home#/security%5Fcredentials) under the '**Access keys'** section, where you created your Secret Access key, you will also find your **Access Key ID.**
+
+ 
+
+
+* And lastly, get Your `AWS Region` from the Home Page of[ AWS Bedrock](https://us-east-1.console.aws.amazon.com/bedrock/home?region=us-east-1#/overview) as shown in the image below.
+
+ 
+
+
+---
+
+## Next Steps
+
+The complete list of features supported in the SDK are available on the link below.
+
+
+
+1. Click **"Add Key"** and enable the **"Local/Privately hosted provider"** toggle
+2. Configure your deployment:
+ - Select the matching provider API specification (typically `OpenAI`)
+ - Enter your model's base URL in the `Custom Host` field
+ - Add required authentication headers and their values
+3. Click **"Create"** to generate your virtual key
+
+#### Step 2: Use Your Virtual Key in Requests
+
+After creating your virtual key, you can use it in your applications:
+
+
+
+
+
+## Troubleshooting
+
+| Issue | Possible Causes | Solutions |
+|-------|-----------------|-----------|
+| Connection Errors | Incorrect URL, network issues, firewall rules | Verify URL format, check network connectivity, confirm firewall allows traffic |
+| Authentication Failures | Invalid credentials, incorrect header format | Check credentials, ensure headers are correctly formatted and forwarded |
+| Timeout Errors | LLM server overloaded, request too complex | Adjust timeout settings, implement load balancing, simplify requests |
+| Inconsistent Responses | Different model versions, configuration differences | Standardize model versions, document expected behavior differences |
+
+## FAQs
+
+
+
+In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.
+
+
+
+
+
+1. Navigate to [Virtual Keys](https://app.portkey.ai/virtual-keys) in your Portkey dashboard
+2. Click **"Add Key"** and enable the **"Local/Privately hosted provider"** toggle
+3. Configure your deployment:
+ - Select the matching provider API specification (typically `OpenAI`)
+ - Enter your model's base URL in the `Custom Host` field
+ - Add required authentication headers and their values
+4. Click **"Create"** to generate your virtual key
+
+You can now use this virtual key in your requests:
+
+
+
+In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.
+
+
+
+
+In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.
+
+
+
+
+
+1. Navigate to [Virtual Keys](https://app.portkey.ai/virtual-keys) in your Portkey dashboard
+2. Click **"Add Key"** and enable the **"Local/Privately hosted provider"** toggle
+3. Configure your deployment:
+ - Select the matching provider API specification (typically `OpenAI`)
+ - Enter your model's base URL in the `Custom Host` field
+ - Add required authentication headers and their values
+4. Click **"Create"** to generate your virtual key
+
+You can now use this virtual key in your requests:
+
+
+
+
+## Codestral v/s Mistral API Endpoint
+
+Here's a handy guide for when you might want to make your requests to the Codestral endpoint v/s the original Mistral API endpoint:
+
+ 
+
+
+[For more, check out Mistral's Code Generation guide here](https://docs.mistral.ai/capabilities/code%5Fgeneration/#operation/listModels).
+
+---
+
+## Managing Mistral AI Prompts
+
+You can manage all prompts to Mistral AI in the [Prompt Library](/product/prompt-library). All the current models of Mistral AI are supported and you can easily start testing different prompts.
+
+Once you're ready with your prompt, you can use the `portkey.prompts.completions.create` interface to use the prompt in your application.
+
+
+### Mistral Tool Calling
+Tool calling feature lets models trigger external tools based on conversation context. You define available functions, the model chooses when to use them, and your application executes them and returns results.
+
+Portkey supports Mistral Tool Calling and makes it interoperable across multiple providers. With Portkey Prompts, you can templatize various your prompts & tool schemas as well.
+
+
+
+
+
+1. Navigate to [Virtual Keys](https://app.portkey.ai/virtual-keys) in your Portkey dashboard
+2. Click **"Add Key"** and enable the **"Local/Privately hosted provider"** toggle
+3. Configure your deployment:
+ - Select the matching provider API specification (typically `OpenAI`)
+ - Enter your model's base URL in the `Custom Host` field
+ - Add required authentication headers and their values
+4. Click **"Create"** to generate your virtual key
+
+You can now use this virtual key in your requests:
+
+
+
+
+In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.
+
+
+
+
+
+2. Use this prompt in your codebase using the Portkey SDK.
+
+
+
+
+More information on image generation is available in the [API Reference](/provider-endpoints/images/create-image#create-image).
+
+### Audio - Transcription, Translation, and Text-to-Speech
+
+Portkey's multimodal Gateway also supports the `audio` methods on OpenAI API. Check out the below guides for more info:
+
+Check out the below guides for more info:
+
+
+
+All requests, including those with fewer than 1024 tokens, will display a `cached_tokens` field of the `usage.prompt_tokens_details` [chat completions object](https://platform.openai.com/docs/api-reference/chat/object) indicating how many of the prompt tokens were a cache hit.
+
+For requests under 1024 tokens, `cached_tokens` will be zero.
+
+
+ 
+ cached_tokens field of the usage.prompt_tokens_details
+
+In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.
+
+
+
+
+Your OpenAI personal or service account API keys can be saved to Portkey. Additionally, your **[OpenAI Admin API Keys](https://platform.openai.com/settings/organization/admin-keys)** can also be saved to Portkey so that you can route to OpenAI Admin routes through Portkey API.
+
+
+
+
+## Local Setup
+If you do not want to use Portkey's hosted API, you can also run Portkey locally:
+
+
+
+Portkey runs on our popular [open source Gateway](https://git.new/portkey). You can spin it up locally to make requests without sending them to the Portkey API.
+
+
+
+---
+
+### OpenAI Structured Outputs
+
+Use structured outputs for more consistent and parseable responses:
+
+
+
+
+**Uploading Base64 encoded images**
+
+If you have an image or set of images locally, you can pass those to the model in base 64 encoded format. [Check out this example from OpenAI on how to do this](https://platform.openai.com/docs/guides/vision#uploading-base64-encoded-images).
+
+
+
+
+
+
+In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.
+
+
+
+
+
+
+In addition to the `user` parameter, Portkey allows you to send arbitrary custom metadata with your requests. This powerful feature enables you to associate additional context or information with each request, which can be useful for analysis, debugging, or other custom use cases.
+
+
+
+
+
+1. Navigate to [Virtual Keys](https://app.portkey.ai/virtual-keys) in your Portkey dashboard
+2. Click **"Add Key"** and enable the **"Local/Privately hosted provider"** toggle
+3. Configure your deployment:
+ - Select the matching provider API specification (typically `OpenAI`)
+ - Enter your model's base URL in the `Custom Host` field
+ - Add required authentication headers and their values
+4. Click **"Create"** to generate your virtual key
+
+You can now use this virtual key in your requests:
+
+
+ 
+
+
+### 3. Add Guardrail ID to a Config
+
+* When you save a guardrail, you'll get an associated Guardrail ID
+* Add this ID to the `input_guardrails` parameter in your Portkey Config.
+* Create these Configs in Portkey UI, save them, and get an associated Config ID to attach to your requests. [More here](/product/ai-gateway/configs).
+
+```json
+{
+ "input_guardrails": [
+ "your_exa_guardrail_id"
+ ]
+}
+```
+
+* Save this config and note its Config ID for use in your requests
+
+
+
+
+
+## Frequently Asked Questions
+
+
+
+
+
+
+## Migration Guide
+
+If you're already using Langfuse with OpenAI, migrating to use Portkey is simple:
+
+
+
+
+Based on your access level, you might see the relevant permissions on the API key modal - tick the ones you'd like, name your API key, and save it.
+
+## 2\. Integrate Portkey
+
+Portkey offers a variety of integration options, including SDKs, REST APIs, and native connections with platforms like OpenAI, Langchain, and LlamaIndex, among others.
+
+### Through the OpenAI SDK
+
+If you're using the **OpenAI SDK**, import the Portkey SDK and configure it within your OpenAI client object:
+
+
+
+ 
+
+
+## Understanding API Key Types
+
+**Note**: Service API keys provide system-level access and are distinct from user API keys which grant individual user access.
+
+- **Service API Keys**: Used for automated processes, integrations, and system-level operations. These keys typically have broader permissions and are not tied to individual users.
+- **User API Keys**: Associated with specific users and provide individualized access to Portkey resources. These keys are generally more limited in scope and tied to the permissions of the specific user.
+
+
+## Related Features
+
+
+
+
+
+## Logs vs. Logs Metadata
+- **Logs**: Complete log entries including request and response payloads
+- **Logs Metadata**: Information such as timestamps, model used, tokens consumed, and other metrics without the actual content
+
+## Related Features
+
+
+
+
+## Understanding Virtual Keys
+
+Virtual keys in Portkey securely store provider credentials and enable:
+
+- Centralized management of AI provider keys
+- Abstraction of actual provider keys from end users
+- Definition of routing rules, fallbacks, and other advanced features
+- Application of usage limits and tracking across providers
+
+By controlling who can view and manage these virtual keys, organizations can maintain security while enabling appropriate access for different team roles.
+
+
+## Related Features
+
+
+
+
+### Alert Thresholds
+
+You can configure alert thresholds to receive notifications before reaching your full budget:
+
+1. Enter a value in the **Alert Threshold** field
+2. When usage reaches this threshold, notifications will be sent to configured recipients
+3. The API key continues to function until the full budget limit is reached
+
+### Periodic Reset Options
+
+Budget limits can be set to automatically reset at regular intervals:
+
+
+ 
+
+
+- **No Periodic Reset**: The budget limit applies until exhausted
+- **Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC
+- **Reset Monthly**: Budget limits reset on the 1st of each month at 12 AM UTC
+
+## Rate Limits
+
+Rate limits control how frequently an API key can be used, helping you maintain application performance and prevent unexpected usage spikes.
+
+### Setting Up Rate Limits
+
+When creating a new API key or editing an existing one:
+
+1. Toggle on **Add Rate Limit**
+2. Choose your limit type:
+ - **Requests**: Limit based on number of API calls
+ - **Tokens**: Limit based on token consumption
+3. Specify the limit value and time interval
+
+
+ 
+
+
+### Time Intervals
+
+Rate limits can be applied using three different time intervals:
+
+- **Per Minute**: For granular control of high-frequency applications
+- **Per Hour**: For balanced control of moderate usage
+- **Per Day**: For broader usage management
+
+When a rate limit is reached, subsequent requests are rejected until the time interval resets.
+
+## Email Notifications
+
+Email notifications keep relevant stakeholders informed about API key usage and when limits are approached or reached.
+
+### Configuring Notifications
+
+To set up email notifications for an API key with budget limits:
+
+1. Toggle on **Email Notifications** when creating/editing an API key
+2. Add recipient email addresses:
+ - Type an email address and click **New** or press Enter
+ - Add multiple recipients as needed
+
+
+ 
+
+
+### Default Recipients
+
+When limits are reached or thresholds are crossed, Portkey automatically sends notifications to:
+
+- Organization administrators
+- Organization owners
+- The API key creator/owner
+
+You can add additional recipients such as finance team members, department heads, or project managers who need visibility into AI usage.
+
+## Availability
+
+These features are available to Portkey Enterprise customers and select Pro users. To enable these features for your account, please contact [support@portkey.ai](mailto:support@portkey.ai) or join the [Portkey Discord](https://portkey.ai/community) community.
+
+To learn more about the Portkey Enterprise plan, [schedule a consultation](https://portkey.sh/demo-16).
diff --git a/virtual_key_old/product/administration/enforce-default-config.mdx b/virtual_key_old/product/administration/enforce-default-config.mdx
new file mode 100644
index 00000000..f3632bef
--- /dev/null
+++ b/virtual_key_old/product/administration/enforce-default-config.mdx
@@ -0,0 +1,199 @@
+---
+title: "Enforcing Default Configs on API Keys"
+description: "Learn how to attach default configs to API keys for enforcing governance controls across your organization"
+---
+
+## Overview
+
+Portkey allows you to attach default configs to API keys, enabling you to enforce specific routing rules, security controls, and other governance measures across all API calls made with those keys. This feature provides a powerful way to implement organization-wide policies without requiring changes to individual application code.
+
+
+
+
+
+
+
+When a workspace reaches its rate limit, all subsequent requests from that workspace will be rejected until the time interval resets, regardless of which API key is used.
+
+## Setting Budget Limits
+
+Budget limits allow you to set maximum spending or token usage thresholds for an entire workspace, automatically preventing further usage when limits are reached.
+
+To set up budget limits for a workspace:
+
+1. Toggle on **Add Budget**
+2. Choose your limit type:
+ - **Cost**: Set a maximum spend in USD
+ - **Tokens**: Set a maximum token usage
+3. Enter the budget amount in the **Budget Limit ($)** field
+4. Optionally, set an **Alert Threshold ($)** to receive notifications before reaching the full budget
+5. Select a **Periodic Reset** option to determine when the budget refreshes
+
+
+ 
+
+
+### Alert Thresholds
+
+Alert thresholds trigger notifications when a percentage of the workspace budget has been consumed:
+
+1. Enter a value in the **Alert Threshold ($)** field
+2. When usage reaches this threshold, notifications will be sent to configured recipients
+3. The workspace continues to function until the full budget limit is reached
+
+### Periodic Reset Options
+
+Workspace budgets can be set to automatically reset at regular intervals:
+
+- **No Periodic Reset**: The budget limit applies until exhausted
+- **Reset Weekly**: Budget limits reset every Sunday at 12 AM UTC
+- **Reset Monthly**: Budget limits reset on the 1st of each month at 12 AM UTC
+
+
+## Notification System
+
+When workspace limits are approached or reached, Portkey automatically sends notifications to:
+
+- Organization administrators
+- Organization owners
+- Workspace administrators
+
+These notifications help teams stay informed about their resource usage and take appropriate action before limits are reached.
+
+## Use Cases
+
+Workspace budget limits are particularly useful for:
+
+- **Departmental Allocations**: Assign specific AI budgets to different departments (Marketing, Customer Support, R&D)
+- **Project Management**: Allocate resources based on project priority and requirements
+- **Cost Center Tracking**: Monitor and control spending across different cost centers
+- **Phased Rollouts**: Gradually increase limits as teams demonstrate value and mature their AI use cases
+
+
+
+## Availability
+
+Workspace Budget Limits are available to Portkey Enterprise customers and select Pro users. To enable these features for your account, please contact [support@portkey.ai](mailto:support@portkey.ai) or join the [Portkey Discord](https://portkey.ai/community) community.
diff --git a/virtual_key_old/product/administration/enforce-workspace-level-guardials.mdx b/virtual_key_old/product/administration/enforce-workspace-level-guardials.mdx
new file mode 100644
index 00000000..e5a8cf5d
--- /dev/null
+++ b/virtual_key_old/product/administration/enforce-workspace-level-guardials.mdx
@@ -0,0 +1,38 @@
+---
+title: "Enforcing Workspace Level Guardrails"
+---
+
+## Overview
+Portkey allows workspace owners to enforce request guardrails at the workspace level. This feature ensures that all API requests made within a workspace comply with predefined policies, enhancing security, compliance, and governance at a more granular level.
+
+## How It Works
+Workspace owners can define input and output guardrails in the workspace settings. These guardrails are enforced on all API requests made within the workspace, ensuring uniform policy enforcement across all users and applications in that workspace.
+
+- **Input Guardrails**: Define checks and constraints for incoming LLM requests within the workspace.
+- **Output Guardrails**: Ensure LLM responses align with the defined guardrail policies.
+
+The guardrails available here are the ones created within the user's workspace. Users can select any of these as the default input/output guardrails for their workspace. Multiple providers are supported for setting up guardrails. For a detailed list of supported providers and configurations:
+
+
+
+
+
+You can also [self-host](https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md) the gateway and then connect it to Portkey. Please reach out on hello@portkey.ai and we'll help you set this up!
diff --git a/virtual_key_old/product/ai-gateway/automatic-retries.mdx b/virtual_key_old/product/ai-gateway/automatic-retries.mdx
new file mode 100644
index 00000000..d16d29da
--- /dev/null
+++ b/virtual_key_old/product/ai-gateway/automatic-retries.mdx
@@ -0,0 +1,160 @@
+---
+title: "Automatic Retries"
+description: "LLM APIs often have inexplicable failures. With Portkey, you can rescue a substantial number of your requests with our in-built automatic retries feature. "
+---
+
+
+
+For each request we also calculate and show the cache response time and how much money you saved with each hit.
+
+---
+
+## How Cache works with Configs
+
+You can set cache at two levels:
+
+* **Top-level** that works across all the targets.
+* **Target-level** that works when that specific target is triggered.
+
+
+
+
+
+
+## Enabling Fallback on LLMs
+
+To enable fallbacks, you can modify the [config object](/api-reference/config-object) to include the `fallback` mode.
+
+Here's a quick example of a config to **fallback** to Anthropic's `claude-3.5-sonnet` if OpenAI's `gpt-4o` fails.
+
+```JSON
+{
+ "strategy": {
+ "mode": "fallback"
+ },
+ "targets": [
+ {
+ "virtual_key": "openai-virtual-key",
+ "override_params": {
+ "model": "gpt-4o"
+ }
+ },
+ {
+ "virtual_key": "anthropic-virtual-key",
+ "override_params": {
+ "model": "claude-3.5-sonnet-20240620"
+ }
+ }
+ ]
+}
+```
+
+In this scenario, if the OpenAI model encounters an error or fails to respond, Portkey will automatically retry the request with Anthropic.
+
+[Using Configs in your Requests](/product/ai-gateway/configs#using-configs)
+
+## Triggering fallback on specific error codes
+
+By default, fallback is triggered on any request that returns a **non-2xx** status code.
+
+You can change this behaviour by setting the optional `on_status_codes` param in your fallback config and manually inputting the status codes on which fallback will be triggered.
+
+
+```sh
+{
+ "strategy": {
+ "mode": "fallback",
+ "on_status_codes": [ 429 ]
+ },
+ "targets": [
+ {
+ "virtual_key": "openai-virtual-key"
+ },
+ {
+ "virtual_key": "azure-openai-virtual-key"
+ }
+ ]
+}
+```
+
+Here, fallback from OpenAI to Azure OpenAI will only be triggered when there is a `429` error code from the OpenAI request (i.e. rate limiting error)
+
+## Tracing Fallback Requests on Portkey
+
+Portkey logs all the requests that are sent as a part of your fallback config. This allows you to easily trace and see which targets failed and see which ones were eventually successful.
+
+To see your fallback trace,
+
+1. On the Logs page, first filter the logs with the specific `Config ID` where you've setup the fallback - this will show all the requests that have been sent with that config.
+2. Now, trace an individual request and all the failed + successful logs for it by filtering further on `Trace ID` \- this will show all the logs originating from a single request.
+
+
+ 
+
+
+## Caveats and Considerations
+
+While the Fallback on LLMs feature greatly enhances the reliability and resilience of your application, there are a few things to consider:
+
+1. Ensure the LLMs in your fallback list are compatible with your use case. Not all LLMs offer the same capabilities.
+2. Keep an eye on your usage with each LLM. Depending on your fallback list, a single request could result in multiple LLM invocations.
+3. Understand that each LLM has its own latency and pricing. Falling back to a different LLM could have implications on the cost and response time.
diff --git a/virtual_key_old/product/ai-gateway/files.mdx b/virtual_key_old/product/ai-gateway/files.mdx
new file mode 100644
index 00000000..dfa90f30
--- /dev/null
+++ b/virtual_key_old/product/ai-gateway/files.mdx
@@ -0,0 +1,55 @@
+---
+title: "Files"
+description: "Upload files to Portkey and reuse the content in your requests"
+---
+
+Portkey supports managing files in two ways:
+
+1. Uploading and managing files to any provider using the unified signature
+2. [Enterprise Only] Uploading files to Portkey and using them for batching/fine-tuning requests with any provider
+
+## 1. Uploading and managing files to any provider using the unified signature
+
+Please refer to the [Provider Specific Files](/integrations/llms/openai/files) documentation for more details.
+
+1. [OpenAI](/integrations/llms/openai/files)
+2. [Bedrock](/integrations/llms/bedrock/files)
+3. [Azure OpenAI](/integrations/llms/azure-openai/files)
+4. [Fireworks](/integrations/llms/fireworks/files)
+5. [Vertex](/integrations/llms/vertex-ai/files)
+
+## 2. [Enterprise Only] Uploading files to Portkey and using them for batching/fine-tuning requests with any provider
+
+With Portkey, you can upload files to Portkey and reuse them for [batching inference](/product/ai-gateway/batches) with any provider and [fine-tuning](/product/ai-gateway/fine-tuning) with supported providers.
+
+In this way, you can test your data with different foundation models, perform A/B testing with different foundation models, and perform batch inference with different foundation models.
+
+### Uploading Files
+
+```sh
+ curl --location --request POST 'https://api.portkey.ai/v1/files' \
+--header 'x-portkey-api-key: 
+
+
+#### Explore the AI Gateway's Multimodal capabilities below:
+
+
+
+## Managing Functions and Tools in Prompts
+
+Portkey's Prompt Library supports creating prompt templates with function/tool definitions, as well as letting you set the `tool choice` param. Portkey will also validate your tool definition on the fly, eliminating syntax errors.
+
+
+ 
+
+
+## Supported Providers and Models
+
+The following providers are supported for function calling with more providers getting added soon. Please raise a [request](/integrations/llms/suggest-a-new-integration) or a [PR](https://github.com/Portkey-AI/gateway/pulls) to add model or provider to the AI gateway.
+
+| Provider | Models |
+| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------- |
+| [OpenAI](/integrations/llms/openai) | gpt-4 series of modelsgpt-3.5-turbo series of models |
+| [Azure OpenAI](/integrations/llms/azure-openai) | gpt-4 series of modelsgpt-3.5-turbo series of models |
+| [Anyscale](/integrations/llms/anyscale-llama2-mistral-zephyr) | mistralai/Mistral-7B-Instruct-v0.1 mistralai/Mixtral-8x7B-Instruct-v0.1 |
+| [Together AI](/integrations/llms/together-ai) | mistralai/Mixtral-8x7B-Instruct-v0.1 mistralai/Mistral-7B-Instruct-v0.1 togethercomputer/CodeLlama-34b-Instruct |
+| [Fireworks AI](/integrations/llms/fireworks) | firefunction-v1fw-function-call-34b-v0 |
+| [Google Gemini](/integrations/llms/gemini) / [Vertex AI](/integrations/llms/vertex-ai) | gemini-1.0-progemini-1.0-pro-001gemini-1.5-pro-latest |
+
+## Cookbook
+
+[**Here's a detailed cookbook on function calling using Portkey.**](/guides/getting-started/function-calling)
diff --git a/virtual_key_old/product/ai-gateway/multimodal-capabilities/image-generation.mdx b/virtual_key_old/product/ai-gateway/multimodal-capabilities/image-generation.mdx
new file mode 100644
index 00000000..c6ea2286
--- /dev/null
+++ b/virtual_key_old/product/ai-gateway/multimodal-capabilities/image-generation.mdx
@@ -0,0 +1,142 @@
+---
+title: "Image Generation"
+description: Portkey's AI gateway supports image generation capabilities that many foundational model providers offer.
+---
+
+The most common use case is that of **text-to-image** where the user sends a prompt which the image model processes and returns an image.
+
+
+
+
+## Cookbook
+
+[**Here's a detailed cookbook on image generation using Portkey**](https://github.com/Portkey-AI/portkey-cookbook/blob/main/examples/image-generation.ipynb) which demonstrates the use of multiple providers and routing between them through Configs.
diff --git a/virtual_key_old/product/ai-gateway/multimodal-capabilities/speech-to-text.mdx b/virtual_key_old/product/ai-gateway/multimodal-capabilities/speech-to-text.mdx
new file mode 100644
index 00000000..cdc86c11
--- /dev/null
+++ b/virtual_key_old/product/ai-gateway/multimodal-capabilities/speech-to-text.mdx
@@ -0,0 +1,147 @@
+---
+title: "Speech-to-Text"
+description: "Portkey's AI gateway supports STT models like Whisper by OpenAI."
+---
+
+## Transcription & Translation Usage
+
+Portkey supports both `Transcription` and `Translation` methods for STT models and follows the OpenAI signature where you can send the file (in `flac`, `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `ogg`, `wav`, or `webm` formats) as part of the API request.
+
+Here's an example:
+
+OpenAI NodeJSOpenAI PythonREST
+
+
+
+## Creating prompt templates for vision models
+
+Portkey's prompt library supports creating templates with image inputs. If the same image will be used in all prompt calls, you can save it as part of the template's image URL itself. Or, if the image will be sent via the API as a variable, add a variable to the image link.
+
+
+ 
+
+## Supported Providers and Models
+
+Portkey supports all vision models from its integrated providers as they become available. The table below shows some examples of supported vision models. Please raise a [request](/integrations/llms/suggest-a-new-integration) or a [PR](https://github.com/Portkey-AI/gateway/pulls) to add a provider to the AI gateway.
+
+| Provider | Models | Functions |
+| ---------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------------------- |
+| [OpenAI](/integrations/llms/openai) | `gpt-4-vision-preview`, `gpt-4o`, `gpt-4o-mini ` | Create Chat Completion |
+| [Azure OpenAI](/integrations/llms/azure-openai) | `gpt-4-vision-preview`, `gpt-4o`, `gpt-4o-mini ` | Create Chat Completion |
+| [Gemini](/integrations/llms/gemini) | `gemini-1.0-pro-vision `, `gemini-1.5-flash`, `gemini-1.5-flash-8b`, `gemini-1.5-pro` | Create Chat Completion |
+| [Anthropic](/integrations/llms/anthropic) | `claude-3-sonnet`, `claude-3-haiku`, `claude-3-opus`, `claude-3.5-sonnet`, `claude-3.5-haiku` | Create Chat Completion |
+| [AWS Bedrock](/integrations/llms/aws-bedrock) | `anthropic.claude-3-5-sonnet anthropic.claude-3-5-haiku anthropic.claude-3-5-sonnet-20240620-v1:0` | Create Chat Completion |
+
+For a complete list of all supported provider (including non-vision LLMs), check out our [providers documentation](/integrations/llms).
diff --git a/virtual_key_old/product/ai-gateway/realtime-api.mdx b/virtual_key_old/product/ai-gateway/realtime-api.mdx
new file mode 100644
index 00000000..9889dff1
--- /dev/null
+++ b/virtual_key_old/product/ai-gateway/realtime-api.mdx
@@ -0,0 +1,187 @@
+---
+title: "Realtime API"
+description: "Use OpenAI's Realtime API with logs, cost tracking, and more!"
+---
+
+
+
+
+## Next Steps
+
+- [For more info on realtime API, refer here](https://platform.openai.com/docs/guides/realtime)
+- [Portkeys OpenAI Integration](/integrations/llms/openai)
+- [Logs](/product/observability/logs)
+- [Traces](/product/observability/traces)
+- [Guardrails](/product/ai-gateway/guardrails)
diff --git a/virtual_key_old/product/ai-gateway/remote-mcp.mdx b/virtual_key_old/product/ai-gateway/remote-mcp.mdx
new file mode 100644
index 00000000..928691ac
--- /dev/null
+++ b/virtual_key_old/product/ai-gateway/remote-mcp.mdx
@@ -0,0 +1,298 @@
+---
+title: "Remote MCP"
+description: Portkey's AI gateway has MCP server support that many foundational model providers offer.
+---
+
+[Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is an open protocol that standardizes how applications provide tools and context to LLMs. The MCP tool in the Responses API allows developers to give the model access to tools hosted on **Remote MCP servers**. These are MCP servers maintained by developers and organizations across the internet that expose these tools to MCP clients, like the Responses API.
+
+
+Portkey Supports using MCP server via the Response API. Calling a remote MCP server with the Responses API is straightforward. For example, here's how you can use the [DeepWiki](https://deepwiki.com/) MCP server to ask questions about nearly any public GitHub repository.
+
+
+
+Example MCP request
+--------------
+
+
+A Responses API request to OpenAI with MCP tools enabled.
+
+
+
+
+
+MCP Server Authentication
+--------------
+
+Unlike the DeepWiki MCP server, most other MCP servers require authentication. The MCP tool in the Responses API gives you the ability to flexibly specify headers that should be included in any request made to a remote MCP server. These headers can be used to share API keys, oAuth access tokens, or any other authentication scheme the remote MCP server implements.
+
+The most common header used by remote MCP servers is the `Authorization` header. This is what passing this header looks like:
+
+Use Stripe MCP tool
+
+
+
+
+To use the required deployment, simply pass the `alias` of the deployment as the `model` in LLM request body. In case the models is left empty or the specified alias does not exist, the default deployment is used.
+
+
+## How are the provider API keys stored?
+
+Your API keys are encrypted and stored in secure vaults, accessible only at the moment of a request. Decryption is performed exclusively in isolated workers and only when necessary, ensuring the highest level of data security.
+
+## How are the provider keys linked to the virtual key?
+
+We randomly generate virtual keys and link them separately to the securely stored keys. This means, your raw API keys can not be reverse engineered from the virtual keys.
+
+## Using Virtual Keys
+
+### Using the Portkey SDK
+
+Add the virtual key directly to the initialization configuration for Portkey.
+
+
+
+For more details, see [Bring Your Own LLM](/product/ai-gateway/byollm).
+
+## Setting Budget Limits
+
+Portkey provides a simple way to set budget limits for any of your virtual keys and helps you manage your spending on AI providers (and LLMs) - giving you confidence and control over your application's costs.
+
+[Budget Limits](/product/ai-gateway/virtual-keys/budget-limits)
+
+## Prompt Templates
+
+Choose your Virtual Key within Portkeyโs prompt templates, and it will be automatically retrieved and ready for use.
+
+## Langchain / LlamaIndex
+
+Set the virtual key when utilizing Portkey's custom LLM as shown below:
+
+```py
+# Example in Langchain
+llm = PortkeyLLM(api_key="PORTKEY_API_KEY",virtual_key="VIRTUAL_KEY")
+```
diff --git a/virtual_key_old/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx b/virtual_key_old/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx
new file mode 100644
index 00000000..b9f19803
--- /dev/null
+++ b/virtual_key_old/product/ai-gateway/virtual-keys/bedrock-amazon-assumed-role.mdx
@@ -0,0 +1,126 @@
+---
+title: "Connect Bedrock with Amazon Assumed Role"
+description: "How to create a virtual key for Bedrock using Amazon Assumed Role Authentication"
+---
+
+
+
+
+## Create an AWS Role for Portkey to Assume
+This role you create will be used by Porktey to execute InvokeModel commands on Bedrock models in your AWS account. The setup process will establish a minimal-permission ("least privilege") role and set it up to allow Porktey to assume this role.
+
+### Create a permission policy in your AWS account using the following JSON
+
+```json
+{
+ "Version": "2012-10-17",
+ "Statement": [
+ {
+ "Sid": "BedrockConsole",
+ "Effect": "Allow",
+ "Action": [
+ "bedrock:InvokeModel",
+ "bedrock:InvokeModelWithResponseStream"
+ ],
+ "Resource": "*"
+ }
+ ]
+}
+```
+
+
+ 
+
+
+### Create a new IAM role
+Choose *AWS account* as the trusted entity type. If you set an external ID be sure to copy it, we will need it later.
+
+
+ 
+
+
+### Add the above policy to the role
+Search for the policy you created above and add it to the role.
+
+
+ 
+
+
+### Configure Trust Relationship for the role
+Once the role is created, open the role and navigate to the *Trust relationships* tab and click *Edit trust policy*.
+This is where you will add the Portkey AWS account as a trusted entity.
+
+```sh Portkey Account ARN
+arn:aws:iam::299329113195:role/portkey-app
+```
+
+
+
+
+ You're all set! You can now use the virtual key to invoke Bedrock models.
diff --git a/virtual_key_old/product/ai-gateway/virtual-keys/budget-limits.mdx b/virtual_key_old/product/ai-gateway/virtual-keys/budget-limits.mdx
new file mode 100644
index 00000000..459913e7
--- /dev/null
+++ b/virtual_key_old/product/ai-gateway/virtual-keys/budget-limits.mdx
@@ -0,0 +1,88 @@
+---
+title: "Budget Limits"
+description: "Budget Limits lets you set cost limits on virtual keys"
+---
+
+Budget Limits lets you set cost or token limits on virtual keys
+
+
+
+
+> #### Key Considerations
+>
+> * Budget limits can be set as either cost-based (USD) or token-based
+> * The minimum cost limit you can set is **$1**
+> * The minimum token limit you can set is **100 tokens**
+> * Budget limits apply until exhausted or reset
+> * Budget limits are applied only to requests made after the limit is set; they do not apply retroactively
+> * Once set, budget limits **cannot be edited** by any organization member
+> * Budget limits work for **all providers** available on Portkey and apply to **all organization members** who use the virtual key
+
+## Alert Thresholds
+
+You can now set alert thresholds to receive notifications before your budget limit is reached:
+
+* For cost-based budgets, set thresholds in USD
+* For token-based budgets, set thresholds in tokens
+* Receive email notifications when usage reaches the threshold
+* Continue using the key until the full budget limit is reached
+
+## Periodic Reset Options
+
+You can configure budget limits to automatically reset at regular intervals:
+
+
+
+
+
+### Reset Period Options
+- **No Periodic Reset**: The budget limit applies until exhausted with no automatic renewal
+- **Reset Weekly**: Budget limits automatically reset every week
+- **Reset Monthly**: Budget limits automatically reset every month
+
+### Reset Timing
+- Weekly resets occur at the beginning of each week (Sunday at 12 AM UTC)
+- Monthly resets occur on the **1st** calendar day of the month, at **12 AM UTC**, irrespective of when the budget limit was set prior
+
+## Editing Budget Limits
+
+If you need to change or update a budget limit, you can **duplicate** the existing virtual key and create a new one with the desired limit.
+
+## Monitoring Your Spending and Usage
+
+You can track your spending and token usage for any specific virtual key by navigating to the Analytics tab and filtering by the **desired key** and **timeframe**.
+
+## Pricing Support and Limitations
+
+Budget limits currently apply to all providers and models for which Portkey has pricing support. If a specific request log shows `0 cents` in the COST column, it means that Portkey does not currently track pricing for that model, and it will not count towards the virtual key's budget limit.
+
+For token-based budgets, Portkey tracks both input and output tokens across all supported models.
+
+It's important to note that budget limits cannot be applied retrospectively. The spend counter starts from zero only after you've set a budget limit for a key.
+
+## Availability
+
+Budget Limits is currently available **exclusively to Portkey Enterprise** customers and select Pro users. If you're interested in enabling this feature for your account, please reach out to us at [support@portkey.ai](mailto:support@portkey.ai) or join the [Portkey Discord](https://portkey.ai/community) community.
+
+## Enterprise Plan
+
+To discuss Portkey Enterprise plan details and pricing, [you can schedule a quick call here](https://portkey.sh/demo-16).
diff --git a/virtual_key_old/product/ai-gateway/virtual-keys/rate-limits.mdx b/virtual_key_old/product/ai-gateway/virtual-keys/rate-limits.mdx
new file mode 100644
index 00000000..1e5b6083
--- /dev/null
+++ b/virtual_key_old/product/ai-gateway/virtual-keys/rate-limits.mdx
@@ -0,0 +1,77 @@
+---
+title: "Rate Limits"
+description: "Set Rate Limts to your virtual keys"
+---
+
+Rate Limits lets you set request or token consumption limits on virtual keys
+
+
+
+
+> #### Key Considerations
+>
+> * Rate limits can be set as either request-based or token-based
+> * Time intervals can be configured as per minute, per hour, or per day
+> * Setting the limit to 0 disables the virtual key
+> * Rate limits apply immediately after being set
+> * Once set, budget limits **cannot be edited** by any organization member
+> * Rate limits work for **all providers** available on Portkey and apply to **all organization members** who use the virtual key
+> * After a rate limit is reached, requests will be rejected until the time period resets
+
+## Rate Limit Intervals
+
+You can choose from three different time intervals for your rate limits:
+
+* **Per Minute**: Limits reset every minute, ideal for fine-grained control
+* **Per Hour**: Limits reset hourly, providing balanced usage control
+* **Per Day**: Limits reset daily, suitable for broader usage patterns
+
+## Exceeding Rate Limits
+
+When a rate limit is reached:
+* Subsequent requests are rejected with a specific error code
+* Error messages clearly indicate that the rate limit has been exceeded
+* The limit automatically resets after the specified time period has elapsed
+
+## Editing Rate Limits
+
+If you need to change or update a rate limit, you can **duplicate** the existing virtual key and create a new one with the desired limit.
+
+## Monitoring Your Usage
+
+You can track your request and token usage for any specific virtual key by navigating to the Analytics tab and filtering by the **desired key** and **timeframe**.
+
+## Use Cases for Rate Limits
+
+* **Cost Control**: Prevent unexpected usage spikes that could lead to high costs
+* **Performance Management**: Ensure your application maintains consistent performance
+* **Fairness**: Distribute API access fairly across teams or users
+* **Security**: Mitigate potential abuse or DoS attacks
+* **Provider Compliance**: Stay within the rate limits imposed by underlying AI providers
+
+## Availability
+
+Rate Limits is currently available **exclusively to Portkey Enterprise** customers and select Pro users. If you're interested in enabling this feature for your account, please reach out to us at [support@portkey.ai](mailto:support@portkey.ai) or join the [Portkey Discord](https://portkey.ai/community) community.
+
+## Enterprise Plan
+
+To discuss Portkey Enterprise plan details and pricing, [you can schedule a quick call here](https://portkey.sh/demo-16).
diff --git a/virtual_key_old/product/autonomous-fine-tuning.mdx b/virtual_key_old/product/autonomous-fine-tuning.mdx
new file mode 100644
index 00000000..29275245
--- /dev/null
+++ b/virtual_key_old/product/autonomous-fine-tuning.mdx
@@ -0,0 +1,98 @@
+---
+title: 'Autonomous Fine-tuning'
+description: 'Automatically create, manage, and execute fine-tuning jobs for Large Language Models (LLMs) across multiple providers.'
+---
+
+
+
+
+With the ability to create and manage multiple organizations, you can tailor access control to match your company's structure and project requirements. Users can be assigned to specific organizations, and they can seamlessly switch between them using Portkey's intuitive user interface.
+
+
+ 
+
+
+## 2\. Fine-Grained User Roles and Permissions
+
+Portkey offers a comprehensive Role-Based Access Control (RBAC) system that allows you to define and assign user roles with granular permissions. By default, Portkey provides three roles: `Owner`, `Admin`, and `Member`, each with a predefined set of permissions across various features.
+
+* `Owners` have complete control over the organization, including user management, billing, and all platform features.
+* `Admins` have elevated privileges, allowing them to manage users, prompts, configs, guardrails, virtual keys, and API keys.
+* `Members` have access to essential features like logs, analytics, prompts, configs, and virtual keys, with limited permissions.
+
+| Feature | Owner Role | Admin Role | Member Role |
+| ------------------ | ------------------------------------------- | ------------------------------------------- | -------------------------- |
+| Logs and Analytics | View, Filter, Group | View, Filter, Group | View, Filter, Group |
+| Prompts | List, View, Create, Update, Delete, Publish | List, View, Create, Update, Delete, Publish | List, View, Create, Update |
+| Configs | List, View, Create, Update, Delete | List, View, Create, Update, Delete | List, View, Create |
+| Guardrails | List, View, Create, Update, Delete | List, View, Create, Update, Delete | List, View, Create, Update |
+| Virtual Keys | List, Create, Edit, Duplicate, Delete, Copy | List, Create, Edit, Duplicate, Delete, Copy | List, Copy |
+| Team | Add users, assign roles | Add users, assign roles | \- |
+| Organisation | Update | Update | \- |
+| API Keys | Create, Edit, Delete, Update, Rotate | Create, Edit, Delete, Update, Rotate | \- |
+| Billing | Manage | \- | \- |
+
+You can easily add team members to your organization and assign them appropriate roles based on their responsibilities. Portkey's user-friendly interface simplifies the process of inviting users and managing their roles, ensuring that the right people have access to the right resources.
+
+
+ 
+
+
+## 3\. Secure and Customizable API Key Management
+
+Portkey provides a secure and flexible API key management system that allows you to create and manage multiple API keys with fine-grained permissions. Each API key can be customized to grant specific access levels to different features, such as metrics, completions, prompts, configs, guardrails, virtual keys, team management, and API key management.
+
+| Feature | Permissions | Default |
+| --------------------------- | ----------------------------- | -------- |
+| Metrics | Disabled, Enabled | Disabled |
+| Completions (all LLM calls) | Disabled, Enabled | Enabled |
+| Prompts | Disabled, Read, Write, Delete | Read |
+| Configs | Disabled, Read, Write, Delete | Disabled |
+| Guardrails | Disabled, Read, Write, Delete | Disabled |
+| Virtual Keys | Disabled, Read, Write, Delete | Disabled |
+| Users (Team Management) | Disabled, Read, Write, Delete | Disabled |
+
+By default, a new organization is provisioned with a master API key that has all permissions enabled. Owners and admins can edit and manage these keys, as well as create new API keys with tailored permissions. This granular control enables you to enforce the principle of least privilege, ensuring that each API key has access only to the necessary resources.
+
+Portkey's API key management system provides a secure and auditable way to control access to your organization's data and resources, reducing the risk of unauthorized access and data breaches.
+
+## Audit Logs
+
+Portkey maintains detailed audit logs that capture all administrative activities across the platform. These logs provide visibility into actions related to prompts, configs, guardrails, virtual keys, team management, organization updates, and API key modifications.
+
+Each log entry includes information about the user, the action performed, the affected resource, and a timestamp. This ensures traceability and accountability, helping teams monitor changes and investigate any unauthorized activity.
+
+Audit logs can be filtered by user, action type, resource, and time range, making it easy to track specific events. Organizations can use this data to enforce security policies, ensure compliance, and maintain operational integrity.
+
+
+
+
+
+Portkeyโs audit logging system provides a clear and structured way to review platform activity, ensuring security and compliance across all operations.
diff --git a/virtual_key_old/product/enterprise-offering/analytics-logs-export.mdx b/virtual_key_old/product/enterprise-offering/analytics-logs-export.mdx
new file mode 100644
index 00000000..b3ed07b3
--- /dev/null
+++ b/virtual_key_old/product/enterprise-offering/analytics-logs-export.mdx
@@ -0,0 +1,50 @@
+---
+title: OTel Integration (Analytics Data)
+---
+
+ Portkey supports sending your Analytics data to OpenTelemetry (OTel) compatible collectors, allowing you to integrate Portkey's analytics with your existing observability infrastructure.
+
+ ## Overview
+
+ While Portkey leverages Clickhouse as the primary Analytics Store for the Control Panel by default, enterprise customers can integrate Portkey's analytics data with their existing data infrastructure through OpenTelemetry.
+
+ ## Configuration
+
+ Portkey supports pushing your analytics data to an OTEL compatible endpoint. The following environment variables are needed for pushing to OTEL:
+
+ ```yaml
+ OTEL_PUSH_ENABLED: true
+ OTEL_ENDPOINT: http://localhost:4318
+ ```
+
+ Additionally, you can configure arbitrary resource attributes of the OTEL logs by setting a comma-separated value for `OTEL_RESOURCE_ATTRIBUTES`:
+
+ ```
+ OTEL_RESOURCE_ATTRIBUTES: ApplicationShortName=gateway,AssetId=12323,deployment.service=production
+ ```
+
+ ## Integration Options
+
+ Enterprise customers commonly use these analytics exports with:
+
+ - **Datadog**: Monitor and analyze your AI operations alongside other application metrics
+ - **AWS S3**: Store analytics data for long-term retention and analysis
+ - **Other OTEL-compatible systems**: Any system that can ingest OpenTelemetry data can be used with this feature
+
+ ## Use Cases
+
+ This feature enables:
+
+ - Centralized observability across your entire tech stack
+ - Long-term storage of analytics data
+ - Custom analytics dashboards in your preferred tools
+ - Integration with existing alerting systems
+
+ ## Getting Support
+
+ For additional assistance with setting up analytics data export:
+
+ - Join our [Discord community](https://portkey.sh/reddit-discord)
+ - Email us at support@portkey.ai
+
+ Our team can help you with best practices for configuring your OTEL collectors and integrating with your existing systems.
diff --git a/virtual_key_old/product/enterprise-offering/audit-logs.mdx b/virtual_key_old/product/enterprise-offering/audit-logs.mdx
new file mode 100644
index 00000000..7da163d5
--- /dev/null
+++ b/virtual_key_old/product/enterprise-offering/audit-logs.mdx
@@ -0,0 +1,140 @@
+---
+title: "Audit Logs"
+description: "Track and monitor all administrative activities across your Portkey organization with comprehensive audit logging."
+---
+
+
+
+
+## Key Benefits
+
+- **Enhanced Security**: Track all changes to your organization's resources and configurations
+- **Compliance Support**: Maintain detailed records to help meet regulatory requirements
+- **Operational Visibility**: Understand who is making changes and when
+- **Troubleshooting**: Investigate issues by reviewing recent configuration changes
+- **Accountability**: Ensure users are responsible for their actions within the platform
+
+## Logged Information
+
+Each audit log entry contains detailed information about administrative activities:
+
+| Field | Description |
+|-------|-------------|
+| Timestamp | Date and time when the action occurred |
+| User | The individual who performed the action |
+| Workspace | The workspace context in which the action was performed (if applicable) |
+| Action | The type of operation performed (create, update, delete, etc.) |
+| Resource | The specific resource or entity that was affected |
+| Response Status | HTTP status code indicating the result of the action |
+| Client IP | IP address from which the request originated |
+| Country | Geographic location associated with the request |
+
+
+
+
+
+### Available Filters
+
+- **Method**: Filter by HTTP method (PUT, POST, DELETE)
+- **Request ID**: Search for a specific request by its unique identifier
+- **Resource Type**: Filter by type of resource affected:
+ - Workspaces
+ - API Keys
+ - Virtual Keys
+ - Configs
+ - Prompts
+ - Guardrails
+ - Integrations
+ - Collections
+ - Organization
+ - Labels
+ - Custom Resource Types
+- **Action**: Filter by the type of action performed:
+ - Create
+ - Update
+ - Delete
+ - Publish
+ - Export
+ - Rotate
+ - Manage
+ - Duplicate
+- **Response Status**: Filter by HTTP response status codes
+- **Workspace**: Filter by specific workspace
+- **User**: Filter by the user who performed the action
+- **Client IP**: Filter by originating IP address
+- **Country**: Filter by geographic location of requests
+- **Time Range**: Filter logs within a specific time period
+
+## Enterprise Features
+
+Portkey's Audit Logs include enterprise-grade capabilities:
+
+### 1. Complete Visibility
+- Full user attribution for every action
+- Detailed timestamps and change history
+- Cross-workspace tracking
+- Searchable audit trail
+
+### 2. Compliance & Security
+- SOC 2, ISO 27001, GDPR, and HIPAA compliant
+- PII data protection
+- Indefinite log retention
+
+### 3. Enterprise-Grade Features
+- Role-based access control
+- Cross-organization visibility
+- Custom retention policies
+
+> As a Director of AI Infrastructure at a Fortune 100 Healthcare company explained: *"Having a detailed audit trail isn't just about compliance. It's about being able to debug production issues quickly, understand usage patterns, and make data-driven decisions about our AI infrastructure."*
+
+
+## Related Features
+
+
+
+
+## Components and Sizing Recommendations
+
+| Component | Options | Sizing Recommendations |
+| --------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
+| AI Gateway | Deploy as a Docker container in your Kubernetes cluster using Helm Charts | AWS NodeGroup t4g.medium instance, with at least 4GiB of memory and two vCPUs For high reliability, deploy across multiple Availability Zones. |
+| Logs store | AWS S3 | Each log document is \~10kb in size (uncompressed) |
+| Cache (Prompts, Configs & Virtual Keys) | Elasticache or self-hosted Redis | Deploy in the same VPC as the Portkey Gateway. |
+
+## Helm Chart
+
+This deployment uses the Portkey AI hybrid Helm chart to deploy the Portkey AI Gateway. You can find more information about the Helm chart in the [Portkey AI Helm Chart GitHub Repository](https://github.com/Portkey-AI/helm/blob/main/charts/portkey-gateway/README.md).
+
+## Prerequisites
+
+1. Create a Portkey account on [Portkey AI](https://app.portkey.ai)
+2. Portkey team will share the credentials for the private Docker registry.
+
+## Markeplace Listing
+
+### Visit Portkey AI AWS Marketplace Listing
+
+You can find the Portkey AI AWS Marketplace listing [here](https://aws.amazon.com/marketplace/pp/prodview-o2leb4xcrkdqa).
+
+
+
+
+
+### Subscribe to Portkey AI Enterprise Edition
+
+Subscribe to the Portkey AI Enterprise Edition to gain access to the Portkey AI Gateway.
+
+
+
+
+
+### Quick Launch
+
+Upon subscribing to the Portkey AI Enterprise Edition, you will be able to select Quick Launch from within your AWS Console Subscriptions.
+
+
+
+
+
+### Launch the Cloud Formation Template
+
+Select the Portkey AI Enterprise Edition and click on Quick Launch.
+
+
+
+
+### Run the Cloud Formation Template
+
+Fill the required parameters and click on Next and run the Cloud Formation Template.
+
+
+
+
+
+## Cloud Formation Steps
+
+- Creates a new EKS cluster and NodeGroup in your selected VPC and Subnets
+- Sets up IAM Roles needed for S3 bucket access using STS and Lambda execution
+- Uses AWS Lambda to:
+ - Install the Portkey AI Helm chart to your EKS cluster
+ - Upload the values.yaml file to the S3 bucket
+- Allows for changes to the values file or helm chart deployment by updating and re-running the same Lambda function in your AWS account
+
+### Cloudformation Template
+
+> Our cloudformation template has passed the AWS Marketplace validation and security review.
+
+```yaml portkey-hybrid-eks-cloudformation.template.yaml [expandable]
+AWSTemplateFormatVersion: "2010-09-09"
+Description: Portkey deployment template for AWS Marketplace
+
+Metadata:
+ AWS::CloudFormation::Interface:
+ ParameterGroups:
+ - Label:
+ default: "Required Parameters"
+ Parameters:
+ - VPCID
+ - Subnet1ID
+ - Subnet2ID
+ - ClusterName
+ - NodeGroupName
+ - NodeGroupInstanceType
+ - SecurityGroupID
+ - CreateNewCluster
+ - HelmChartVersion
+ - PortkeyDockerUsername:
+ NoEcho: true
+ - PortkeyDockerPassword:
+ NoEcho: true
+ - PortkeyClientAuth:
+ NoEcho: true
+ - Label:
+ default: "Optional Parameters"
+ Parameters:
+ - PortkeyOrgId
+ - PortkeyGatewayIngressEnabled
+ - PortkeyGatewayIngressSubdomain
+ - PortkeyFineTuningEnabled
+
+Parameters:
+ # Required Parameters
+ VPCID:
+ Type: AWS::EC2::VPC::Id
+ Description: VPC where the EKS cluster will be created
+ Default: Select a VPC
+
+ Subnet1ID:
+ Type: AWS::EC2::Subnet::Id
+ Description: First subnet ID for EKS cluster
+ Default: Select your subnet
+
+ Subnet2ID:
+ Type: AWS::EC2::Subnet::Id
+ Description: Second subnet ID for EKS cluster
+ Default: Select your subnet
+
+ # Optional Parameters with defaults
+ ClusterName:
+ Type: String
+ Description: Name of the EKS cluster (if not provided, a new EKS cluster will be created)
+ Default: portkey-eks-cluster
+
+ NodeGroupName:
+ Type: String
+ Description: Name of the EKS node group (if not provided, a new EKS node group will be created)
+ Default: portkey-eks-cluster-node-group
+
+ NodeGroupInstanceType:
+ Type: String
+ Description: EC2 instance type for the node group (if not provided, t3.medium will be used)
+ Default: t3.medium
+ AllowedValues:
+ - t3.medium
+ - t3.large
+ - t3.xlarge
+
+ PortkeyDockerUsername:
+ Type: String
+ Description: Docker username for Portkey (provided by the Portkey team)
+ Default: portkeyenterprise
+
+ PortkeyDockerPassword:
+ Type: String
+ Description: Docker password for Portkey (provided by the Portkey team)
+ Default: ""
+ NoEcho: true
+
+ PortkeyClientAuth:
+ Type: String
+ Description: Portkey Client ID (provided by the Portkey team)
+ Default: ""
+ NoEcho: true
+
+ PortkeyOrgId:
+ Type: String
+ Description: Portkey Organisation ID (provided by the Portkey team)
+ Default: ""
+
+ HelmChartVersion:
+ Type: String
+ Description: Version of the Helm chart to deploy
+ Default: "latest"
+ AllowedValues:
+ - latest
+
+ SecurityGroupID:
+ Type: String
+ Description: Optional security group ID for the EKS cluster (if not provided, a new security group will be created)
+ Default: ""
+
+ CreateNewCluster:
+ Type: String
+ AllowedValues: [true, false]
+ Default: true
+ Description: Whether to create a new EKS cluster or use an existing one
+
+ PortkeyGatewayIngressEnabled:
+ Type: String
+ AllowedValues: [true, false]
+ Default: false
+ Description: Whether to enable the Portkey Gateway ingress
+
+ PortkeyGatewayIngressSubdomain:
+ Type: String
+ Description: Subdomain for the Portkey Gateway ingress
+ Default: ""
+
+ PortkeyFineTuningEnabled:
+ Type: String
+ AllowedValues: [true, false]
+ Default: false
+ Description: Whether to enable the Portkey Fine Tuning
+
+Conditions:
+ CreateSecurityGroup: !Equals [!Ref SecurityGroupID, ""]
+ ShouldCreateCluster: !Equals [!Ref CreateNewCluster, true]
+
+Resources:
+ PortkeyAM:
+ Type: AWS::IAM::Role
+ DeletionPolicy: Delete
+ Properties:
+ RoleName: PortkeyAM
+ AssumeRolePolicyDocument:
+ Version: "2012-10-17"
+ Statement:
+ - Effect: Allow
+ Principal:
+ AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
+ Action: sts:AssumeRole
+ Policies:
+ - PolicyName: PortkeyEKSAccess
+ PolicyDocument:
+ Version: "2012-10-17"
+ Statement:
+ - Effect: Allow
+ Action:
+ - "eks:DescribeCluster"
+ - "eks:ListClusters"
+ - "eks:ListNodegroups"
+ - "eks:ListFargateProfiles"
+ - "eks:ListNodegroups"
+ - "eks:CreateCluster"
+ - "eks:CreateNodegroup"
+ - "eks:DeleteCluster"
+ - "eks:DeleteNodegroup"
+ - "eks:UpdateClusterConfig"
+ - "eks:UpdateKubeconfig"
+ Resource: !Sub "arn:aws:eks:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}"
+ - Effect: Allow
+ Action:
+ - "sts:AssumeRole"
+ Resource: !Sub "arn:aws:iam::${AWS::AccountId}:role/PortkeyAM"
+ - Effect: Allow
+ Action:
+ - "sts:GetCallerIdentity"
+ Resource: "*"
+ - Effect: Allow
+ Action:
+ - "iam:ListRoles"
+ - "iam:GetRole"
+ Resource: "*"
+ - Effect: Allow
+ Action:
+ - "bedrock:InvokeModel"
+ - "bedrock:InvokeModelWithResponseStream"
+ Resource: "*"
+ - Effect: Allow
+ Action:
+ - "s3:GetObject"
+ - "s3:PutObject"
+ Resource:
+ - !Sub "arn:aws:s3:::${AWS::AccountId}-${AWS::Region}-portkey-logs/*"
+
+ PortkeyLogsBucket:
+ Type: AWS::S3::Bucket
+ DeletionPolicy: Delete
+ Properties:
+ BucketName: !Sub "${AWS::AccountId}-${AWS::Region}-portkey-logs"
+ VersioningConfiguration:
+ Status: Enabled
+ PublicAccessBlockConfiguration:
+ BlockPublicAcls: true
+ BlockPublicPolicy: true
+ IgnorePublicAcls: true
+ RestrictPublicBuckets: true
+ BucketEncryption:
+ ServerSideEncryptionConfiguration:
+ - ServerSideEncryptionByDefault:
+ SSEAlgorithm: AES256
+
+ # EKS Cluster Role
+ EksClusterRole:
+ Type: AWS::IAM::Role
+ DeletionPolicy: Delete
+ Properties:
+ RoleName: EksClusterRole-Portkey
+ AssumeRolePolicyDocument:
+ Version: "2012-10-17"
+ Statement:
+ - Effect: Allow
+ Principal:
+ Service: eks.amazonaws.com
+ Action: sts:AssumeRole
+ ManagedPolicyArns:
+ - arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
+
+ # EKS Cluster Security Group (if not provided)
+ EksSecurityGroup:
+ Type: AWS::EC2::SecurityGroup
+ Condition: CreateSecurityGroup
+ DeletionPolicy: Delete
+ Properties:
+ GroupDescription: Security group for Portkey EKS cluster
+ VpcId: !Ref VPCID
+ SecurityGroupIngress:
+ - IpProtocol: tcp
+ FromPort: 8787
+ ToPort: 8787
+ CidrIp: PORTKEY_IP
+ SecurityGroupEgress:
+ - IpProtocol: tcp
+ FromPort: 443
+ ToPort: 443
+ CidrIp: 0.0.0.0/0
+
+ # EKS Cluster
+ EksCluster:
+ Type: AWS::EKS::Cluster
+ Condition: ShouldCreateCluster
+ DeletionPolicy: Delete
+ DependsOn: EksClusterRole
+ Properties:
+ Name: !Ref ClusterName
+ Version: "1.32"
+ RoleArn: !GetAtt EksClusterRole.Arn
+ ResourcesVpcConfig:
+ SecurityGroupIds:
+ - !If
+ - CreateSecurityGroup
+ - !Ref EksSecurityGroup
+ - !Ref SecurityGroupID
+ SubnetIds:
+ - !Ref Subnet1ID
+ - !Ref Subnet2ID
+ AccessConfig:
+ AuthenticationMode: API_AND_CONFIG_MAP
+
+ LambdaExecutionRole:
+ Type: AWS::IAM::Role
+ DeletionPolicy: Delete
+ DependsOn: EksCluster
+ Properties:
+ RoleName: PortkeyLambdaRole
+ AssumeRolePolicyDocument:
+ Version: "2012-10-17"
+ Statement:
+ - Effect: Allow
+ Principal:
+ Service: lambda.amazonaws.com
+ Action: sts:AssumeRole
+ ManagedPolicyArns:
+ - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
+ Policies:
+ - PolicyName: EKSAccess
+ PolicyDocument:
+ Version: "2012-10-17"
+ Statement:
+ - Effect: Allow
+ Action:
+ - ec2:DescribeInstances
+ - ec2:DescribeRegions
+ Resource: "*"
+ - Effect: Allow
+ Action:
+ - "sts:AssumeRole"
+ Resource: !GetAtt PortkeyAM.Arn
+ - Effect: Allow
+ Action:
+ - "s3:GetObject"
+ - "s3:PutObject"
+ Resource:
+ - !Sub "arn:aws:s3:::${AWS::AccountId}-${AWS::Region}-portkey-logs/*"
+ - Effect: Allow
+ Action:
+ - "eks:DescribeCluster"
+ - "eks:ListClusters"
+ - "eks:ListNodegroups"
+ - "eks:ListFargateProfiles"
+ - "eks:ListNodegroups"
+ - "eks:CreateCluster"
+ - "eks:CreateNodegroup"
+ - "eks:DeleteCluster"
+ - "eks:DeleteNodegroup"
+ - "eks:CreateFargateProfile"
+ - "eks:DeleteFargateProfile"
+ - "eks:DescribeFargateProfile"
+ - "eks:UpdateClusterConfig"
+ - "eks:UpdateKubeconfig"
+ Resource: !Sub "arn:aws:eks:${AWS::Region}:${AWS::AccountId}:cluster/${ClusterName}"
+
+ LambdaClusterAdmin:
+ Type: AWS::EKS::AccessEntry
+ DependsOn: EksCluster
+ Properties:
+ ClusterName: !Ref ClusterName
+ PrincipalArn: !GetAtt LambdaExecutionRole.Arn
+ Type: STANDARD
+ KubernetesGroups:
+ - system:masters
+ AccessPolicies:
+ - PolicyArn: "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy"
+ AccessScope:
+ Type: "cluster"
+
+ # Node Group Role
+ NodeGroupRole:
+ Type: AWS::IAM::Role
+ DeletionPolicy: Delete
+ Properties:
+ RoleName: NodeGroupRole-Portkey
+ AssumeRolePolicyDocument:
+ Version: "2012-10-17"
+ Statement:
+ - Effect: Allow
+ Principal:
+ Service: ec2.amazonaws.com
+ Action: sts:AssumeRole
+ ManagedPolicyArns:
+ - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
+ - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
+ - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
+
+ # EKS Node Group
+ EksNodeGroup:
+ Type: AWS::EKS::Nodegroup
+ DependsOn: EksCluster
+ DeletionPolicy: Delete
+ Properties:
+ CapacityType: ON_DEMAND
+ ClusterName: !Ref ClusterName
+ NodegroupName: !Ref NodeGroupName
+ NodeRole: !GetAtt NodeGroupRole.Arn
+ InstanceTypes:
+ - !Ref NodeGroupInstanceType
+ ScalingConfig:
+ MinSize: 1
+ DesiredSize: 1
+ MaxSize: 1
+ Subnets:
+ - !Ref Subnet1ID
+ - !Ref Subnet2ID
+
+ PortkeyInstallerFunction:
+ Type: AWS::Lambda::Function
+ DependsOn: EksNodeGroup
+ DeletionPolicy: Delete
+ Properties:
+ FunctionName: portkey-eks-installer
+ Runtime: nodejs18.x
+ Handler: index.handler
+ MemorySize: 1024
+ EphemeralStorage:
+ Size: 1024
+ Code:
+ ZipFile: |
+ const fs = require('fs');
+ const zlib = require('zlib');
+ const { pipeline } = require('stream');
+ const path = require('path');
+ const https = require('https');
+ const { promisify } = require('util');
+ const { execSync } = require('child_process');
+ const { EKSClient, DescribeClusterCommand } = require('@aws-sdk/client-eks');
+
+ async function unzipAwsCli(zipPath, destPath) {
+ // ZIP file format: https://en.wikipedia.org/wiki/ZIP_(file_format)
+ const data = fs.readFileSync(zipPath);
+ let offset = 0;
+
+ // Find end of central directory record
+ const EOCD_SIGNATURE = 0x06054b50;
+ for (let i = data.length - 22; i >= 0; i--) {
+ if (data.readUInt32LE(i) === EOCD_SIGNATURE) {
+ offset = i;
+ break;
+ }
+ }
+
+ // Read central directory info
+ const numEntries = data.readUInt16LE(offset + 10);
+ let centralDirOffset = data.readUInt32LE(offset + 16);
+
+ // Process each file
+ for (let i = 0; i < numEntries; i++) {
+ // Read central directory header
+ const signature = data.readUInt32LE(centralDirOffset);
+ if (signature !== 0x02014b50) {
+ throw new Error('Invalid central directory header');
+ }
+
+ const fileNameLength = data.readUInt16LE(centralDirOffset + 28);
+ const extraFieldLength = data.readUInt16LE(centralDirOffset + 30);
+ const fileCommentLength = data.readUInt16LE(centralDirOffset + 32);
+ const localHeaderOffset = data.readUInt32LE(centralDirOffset + 42);
+
+ // Get filename
+ const fileName = data.slice(
+ centralDirOffset + 46,
+ centralDirOffset + 46 + fileNameLength
+ ).toString();
+
+ // Read local file header
+ const localSignature = data.readUInt32LE(localHeaderOffset);
+ if (localSignature !== 0x04034b50) {
+ throw new Error('Invalid local file header');
+ }
+
+ const localFileNameLength = data.readUInt16LE(localHeaderOffset + 26);
+ const localExtraFieldLength = data.readUInt16LE(localHeaderOffset + 28);
+
+ // Get file data
+ const fileDataOffset = localHeaderOffset + 30 + localFileNameLength + localExtraFieldLength;
+ const compressedSize = data.readUInt32LE(centralDirOffset + 20);
+ const uncompressedSize = data.readUInt32LE(centralDirOffset + 24);
+ const compressionMethod = data.readUInt16LE(centralDirOffset + 10);
+
+ // Create directory if needed
+ const fullPath = path.join(destPath, fileName);
+ const directory = path.dirname(fullPath);
+ if (!fs.existsSync(directory)) {
+ fs.mkdirSync(directory, { recursive: true });
+ }
+
+ // Extract file
+ if (!fileName.endsWith('/')) { // Skip directories
+ const fileData = data.slice(fileDataOffset, fileDataOffset + compressedSize);
+
+ if (compressionMethod === 0) { // Stored (no compression)
+ fs.writeFileSync(fullPath, fileData);
+ } else if (compressionMethod === 8) { // Deflate
+ const inflated = require('zlib').inflateRawSync(fileData);
+ fs.writeFileSync(fullPath, inflated);
+ } else {
+ throw new Error(`Unsupported compression method: ${compressionMethod}`);
+ }
+ }
+
+ // Move to next entry
+ centralDirOffset += 46 + fileNameLength + extraFieldLength + fileCommentLength;
+ }
+ }
+
+
+ async function extractTarGz(source, destination) {
+ // First, let's decompress the .gz file
+ const gunzip = promisify(zlib.gunzip);
+ console.log('Reading source file...');
+ const compressedData = fs.readFileSync(source);
+
+ console.log('Decompressing...');
+ const tarData = await gunzip(compressedData);
+
+ // Now we have the raw tar data
+ // Tar files are made up of 512-byte blocks
+ let position = 0;
+
+ while (position < tarData.length) {
+ // Read header block
+ const header = tarData.slice(position, position + 512);
+ position += 512;
+
+ // Get filename from header (first 100 bytes)
+ const filename = header.slice(0, 100)
+ .toString('utf8')
+ .replace(/\0/g, '')
+ .trim();
+
+ if (!filename) break; // End of tar
+
+ // Get file size from header (bytes 124-136)
+ const sizeStr = header.slice(124, 136)
+ .toString('utf8')
+ .replace(/\0/g, '')
+ .trim();
+ const size = parseInt(sizeStr, 8); // Size is in octal
+
+ console.log(`Found file: ${filename} (${size} bytes)`);
+
+ if (filename === 'linux-amd64/helm') {
+ console.log('Found helm binary, extracting...');
+ // Extract the file content
+ const content = tarData.slice(position, position + size);
+
+ // Write to destination
+ const outputPath = path.join(destination, 'helm');
+ fs.writeFileSync(outputPath, content);
+ console.log(`Helm binary extracted to: ${outputPath}`);
+ return; // We found what we needed
+ }
+
+ // Move to next file
+ position += size;
+ // Move to next 512-byte boundary
+ position += (512 - (size % 512)) % 512;
+ }
+
+ throw new Error('Helm binary not found in archive');
+ }
+
+ async function downloadFile(url, dest) {
+ return new Promise((resolve, reject) => {
+ const file = fs.createWriteStream(dest);
+ https.get(url, (response) => {
+ response.pipe(file);
+ file.on('finish', () => {
+ file.close();
+ resolve();
+ });
+ }).on('error', reject);
+ });
+ }
+
+ async function setupBinaries() {
+ const { STSClient, GetCallerIdentityCommand, AssumeRoleCommand } = require("@aws-sdk/client-sts");
+ const { SignatureV4 } = require("@aws-sdk/signature-v4");
+ const { defaultProvider } = require("@aws-sdk/credential-provider-node");
+ const crypto = require('crypto');
+
+ const tmpDir = '/tmp/bin';
+ if (!fs.existsSync(tmpDir)) {
+ fs.mkdirSync(tmpDir, { recursive: true });
+ }
+ console.log('Setting up AWS CLI...');
+ const awsCliUrl = 'https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip';
+ const awsZipPath = `${tmpDir}/awscliv2.zip`;
+ await unzipAwsCli(awsZipPath, tmpDir);
+ execSync(`chmod +x ${tmpDir}/aws/install ${tmpDir}/aws/dist/aws`);
+ execSync(`${tmpDir}/aws/install --update --install-dir /tmp/aws-cli --bin-dir /tmp/aws-bin`, { stdio: 'inherit' });
+
+ try {
+ await new Promise((resolve, reject) => {
+ const https = require('https');
+ const fs = require('fs');
+
+ const file = fs.createWriteStream('/tmp/kubectl');
+
+ const request = https.get('https://dl.k8s.io/release/v1.32.1/bin/linux/amd64/kubectl', response => {
+ if (response.statusCode === 302 || response.statusCode === 301) {
+ https.get(response.headers.location, redirectResponse => {
+ redirectResponse.pipe(file);
+ file.on('finish', () => {
+ file.close();
+ resolve();
+ });
+ }).on('error', err => {
+ fs.unlink('/tmp/kubectl', () => {});
+ reject(err);
+ });
+ return;
+ }
+
+ response.pipe(file);
+ file.on('finish', () => {
+ file.close();
+ resolve();
+ });
+ });
+
+ request.on('error', err => {
+ fs.unlink('/tmp/kubectl', () => {});
+ reject(err);
+ });
+ });
+
+ execSync('chmod +x /tmp/kubectl', {
+ stdio: 'inherit'
+ });
+ } catch (error) {
+ console.error('Error installing kubectl:', error);
+ throw error;
+ }
+
+ console.log('Setting up helm...');
+ const helmUrl = 'https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz';
+ const helmTarPath = `${tmpDir}/helm.tar.gz`;
+ await downloadFile(helmUrl, helmTarPath);
+
+ await extractTarGz(helmTarPath, tmpDir);
+ execSync(`chmod +x ${tmpDir}/helm`);
+ fs.unlinkSync(helmTarPath);
+
+ process.env.PATH = `${tmpDir}:${process.env.PATH}`;
+
+ execSync(`/tmp/aws-bin/aws --version`);
+ }
+
+ exports.handler = async (event, context) => {
+ try {
+
+ const { CLUSTER_NAME, NODE_GROUP_NAME, CLUSTER_ARN, CHART_VERSION,
+ PORTKEY_AWS_REGION, PORTKEY_AWS_ACCOUNT_ID, PORTKEYAM_ROLE_ARN,
+ PORTKEY_DOCKER_USERNAME, PORTKEY_DOCKER_PASSWORD,
+ PORTKEY_CLIENT_AUTH, ORGANISATIONS_TO_SYNC } = process.env;
+
+ console.log(process.env)
+
+ if (!CLUSTER_NAME || !PORTKEY_AWS_REGION || !CHART_VERSION ||
+ !PORTKEY_AWS_ACCOUNT_ID || !PORTKEYAM_ROLE_ARN) {
+ throw new Error('Missing one or more required environment variables.');
+ }
+
+ await setupBinaries();
+
+ const awsCredentialsDir = '/tmp/.aws';
+ if (!fs.existsSync(awsCredentialsDir)) {
+ fs.mkdirSync(awsCredentialsDir, { recursive: true });
+ }
+
+ // Write AWS credentials file
+ const credentialsContent = `[default]
+ aws_access_key_id = ${process.env.AWS_ACCESS_KEY_ID}
+ aws_secret_access_key = ${process.env.AWS_SECRET_ACCESS_KEY}
+ aws_session_token = ${process.env.AWS_SESSION_TOKEN}
+ region = ${process.env.PORTKEY_AWS_REGION}
+ `;
+
+ fs.writeFileSync(`${awsCredentialsDir}/credentials`, credentialsContent);
+
+ // Write AWS config file
+ const configContent = `[default]
+ region = ${process.env.PORTKEY_AWS_REGION}
+ output = json
+ `;
+ fs.writeFileSync(`${awsCredentialsDir}/config`, configContent);
+
+ // Set AWS config environment variables
+ process.env.AWS_CONFIG_FILE = `${awsCredentialsDir}/config`;
+ process.env.AWS_SHARED_CREDENTIALS_FILE = `${awsCredentialsDir}/credentials`;
+
+ // Define kubeconfig path
+ const kubeconfigDir = `/tmp/${CLUSTER_NAME.trim()}`;
+ const kubeconfigPath = path.join(kubeconfigDir, 'config');
+
+ // Create the directory if it doesn't exist
+ if (!fs.existsSync(kubeconfigDir)) {
+ fs.mkdirSync(kubeconfigDir, { recursive: true });
+ }
+
+ console.log(`Updating kubeconfig for cluster: ${CLUSTER_NAME}`);
+ execSync(`/tmp/aws-bin/aws eks update-kubeconfig --name ${process.env.CLUSTER_NAME} --region ${process.env.PORTKEY_AWS_REGION} --kubeconfig ${kubeconfigPath}`, {
+ stdio: 'inherit',
+ env: {
+ ...process.env,
+ HOME: '/tmp',
+ AWS_CONFIG_FILE: `${awsCredentialsDir}/config`,
+ AWS_SHARED_CREDENTIALS_FILE: `${awsCredentialsDir}/credentials`
+ }
+ });
+
+ // Set KUBECONFIG environment variable
+ process.env.KUBECONFIG = kubeconfigPath;
+
+ let kubeconfig = fs.readFileSync(kubeconfigPath, 'utf8');
+
+ // Replace the command line to use full path
+ kubeconfig = kubeconfig.replace(
+ 'command: aws',
+ 'command: /tmp/aws-bin/aws'
+ );
+
+ fs.writeFileSync(kubeconfigPath, kubeconfig);
+
+ // Setup Helm repository
+ console.log('Setting up Helm repository...');
+ await new Promise((resolve, reject) => {
+ try {
+ execSync(`helm repo add portkey-ai https://portkey-ai.github.io/helm`, {
+ stdio: 'inherit',
+ env: { ...process.env, HOME: '/tmp' }
+ });
+ resolve();
+ } catch (error) {
+ reject(error);
+ }
+ });
+
+ await new Promise((resolve, reject) => {
+ try {
+ execSync(`helm repo update`, {
+ stdio: 'inherit',
+ env: { ...process.env, HOME: '/tmp' }
+ });
+ resolve();
+ } catch (error) {
+ reject(error);
+ }
+ });
+
+ // Create values.yaml
+ const valuesYAML = `
+ replicaCount: 1
+
+ images:
+ gatewayImage:
+ repository: "docker.io/portkeyai/gateway_enterprise"
+ pullPolicy: IfNotPresent
+ tag: "1.9.0"
+ dataserviceImage:
+ repository: "docker.io/portkeyai/data-service"
+ pullPolicy: IfNotPresent
+ tag: "1.0.2"
+
+ imagePullSecrets: [portkeyenterpriseregistrycredentials]
+ nameOverride: ""
+ fullnameOverride: ""
+
+ imageCredentials:
+ - name: portkeyenterpriseregistrycredentials
+ create: true
+ registry: https://index.docker.io/v1/
+ username: ${PORTKEY_DOCKER_USERNAME}
+ password: ${PORTKEY_DOCKER_PASSWORD}
+
+ useVaultInjection: false
+
+ environment:
+ create: true
+ secret: true
+ data:
+ SERVICE_NAME: portkeyenterprise
+ PORT: "8787"
+ LOG_STORE: s3_assume
+ LOG_STORE_REGION: ${PORTKEY_AWS_REGION}
+ AWS_ROLE_ARN: ${PORTKEYAM_ROLE_ARN}
+ LOG_STORE_GENERATIONS_BUCKET: portkey-gateway
+ ANALYTICS_STORE: control_plane
+ CACHE_STORE: redis
+ REDIS_URL: redis://redis:6379
+ REDIS_TLS_ENABLED: "false"
+ PORTKEY_CLIENT_AUTH: ${PORTKEY_CLIENT_AUTH}
+ ORGANISATIONS_TO_SYNC: ${ORGANISATIONS_TO_SYNC}
+
+ serviceAccount:
+ create: true
+ automount: true
+ annotations: {}
+ name: ""
+
+ podAnnotations: {}
+ podLabels: {}
+
+ podSecurityContext: {}
+ securityContext: {}
+
+ service:
+ type: LoadBalancer
+ port: 8787
+ targetPort: 8787
+ protocol: TCP
+ additionalLabels: {}
+ annotations: {}
+
+ ingress:
+ enabled: ${PORTKEY_GATEWAY_INGRESS_ENABLED}
+ className: ""
+ annotations: {}
+ hosts:
+ - host: ${PORTKEY_GATEWAY_INGRESS_SUBDOMAIN}
+ paths:
+ - path: /
+ pathType: ImplementationSpecific
+ tls: []
+
+ resources: {}
+
+ livenessProbe:
+ httpGet:
+ path: /v1/health
+ port: 8787
+ initialDelaySeconds: 30
+ periodSeconds: 60
+ timeoutSeconds: 5
+ failureThreshold: 5
+ readinessProbe:
+ httpGet:
+ path: /v1/health
+ port: 8787
+ initialDelaySeconds: 30
+ periodSeconds: 60
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 5
+
+ autoscaling:
+ enabled: true
+ minReplicas: 1
+ maxReplicas: 10
+ targetCPUUtilizationPercentage: 80
+
+ volumes: []
+ volumeMounts: []
+ nodeSelector: {}
+ tolerations: []
+ affinity: {}
+ autoRestart: false
+
+ dataservice:
+ name: "dataservice"
+ enabled: ${PORTKEY_FINE_TUNING_ENABLED}
+ containerPort: 8081
+ finetuneBucket: ${PORTKEY_AWS_ACCOUNT_ID}-${PORTKEY_AWS_REGION}-portkey-logs
+ logexportsBucket: ${PORTKEY_AWS_ACCOUNT_ID}-${PORTKEY_AWS_REGION}-portkey-logs
+ deployment:
+ autoRestart: true
+ replicas: 1
+ labels: {}
+ annotations: {}
+ podSecurityContext: {}
+ securityContext: {}
+ resources: {}
+ startupProbe:
+ httpGet:
+ path: /health
+ port: 8081
+ initialDelaySeconds: 60
+ failureThreshold: 3
+ periodSeconds: 10
+ timeoutSeconds: 1
+ livenessProbe:
+ httpGet:
+ path: /health
+ port: 8081
+ failureThreshold: 3
+ periodSeconds: 10
+ timeoutSeconds: 1
+ readinessProbe:
+ httpGet:
+ path: /health
+ port: 8081
+ failureThreshold: 3
+ periodSeconds: 10
+ timeoutSeconds: 1
+ extraContainerConfig: {}
+ nodeSelector: {}
+ tolerations: []
+ affinity: {}
+ volumes: []
+ volumeMounts: []
+ service:
+ type: ClusterIP
+ port: 8081
+ labels: {}
+ annotations: {}
+ loadBalancerSourceRanges: []
+ loadBalancerIP: ""
+ serviceAccount:
+ create: true
+ name: ""
+ labels: {}
+ annotations: {}
+ autoscaling:
+ enabled: false
+ createHpa: false
+ minReplicas: 1
+ maxReplicas: 5
+ targetCPUUtilizationPercentage: 80`
+
+ // Write values.yaml
+ const valuesYamlPath = '/tmp/values.yaml';
+ fs.writeFileSync(valuesYamlPath, valuesYAML);
+
+ const { S3Client, PutObjectCommand, GetObjectCommand } = require("@aws-sdk/client-s3");
+ const s3Client = new S3Client({ region: process.env.PORTKEY_AWS_REGION });
+ try {
+ const response = await s3Client.send(new GetObjectCommand({
+ Bucket: `${process.env.PORTKEY_AWS_ACCOUNT_ID}-${process.env.PORTKEY_AWS_REGION}-portkey-logs`,
+ Key: 'values.yaml'
+ }));
+ const existingValuesYAML = await response.Body.transformToString();
+ console.log('Found existing values.yaml in S3, using it instead of default');
+ fs.writeFileSync(valuesYamlPath, existingValuesYAML);
+ } catch (error) {
+ if (error.name === 'NoSuchKey') {
+ // Upload the default values.yaml to S3
+ await s3Client.send(new PutObjectCommand({
+ Bucket: `${process.env.PORTKEY_AWS_ACCOUNT_ID}-${process.env.PORTKEY_AWS_REGION}-portkey-logs`,
+ Key: 'values.yaml',
+ Body: valuesYAML,
+ ContentType: 'text/yaml'
+ }));
+ console.log('Default values.yaml written to S3 bucket');
+ } else {
+ throw error;
+ }
+ }
+
+ // Install/upgrade Helm chart
+ console.log('Installing helm chart...');
+ await new Promise((resolve, reject) => {
+ try {
+ execSync(`helm upgrade --install portkey-ai portkey-ai/gateway -f ${valuesYamlPath} -n portkeyai --create-namespace --kube-context ${process.env.CLUSTER_ARN} --kubeconfig ${kubeconfigPath}`, {
+ stdio: 'inherit',
+ env: {
+ ...process.env,
+ HOME: '/tmp',
+ PATH: `/tmp/aws-bin:${process.env.PATH}`
+ }
+ });
+ resolve();
+ } catch (error) {
+ reject(error);
+ }
+ });
+
+ return {
+ statusCode: 200,
+ body: JSON.stringify({
+ message: 'EKS installation and helm chart deployment completed successfully',
+ event: event
+ })
+ };
+ } catch (error) {
+ console.error('Error:', error);
+ return {
+ statusCode: 500,
+ body: JSON.stringify({
+ message: 'Error during EKS installation and helm chart deployment',
+ error: error.message
+ })
+ };
+ }
+ };
+ Role: !GetAtt LambdaExecutionRole.Arn
+ Timeout: 900
+ Environment:
+ Variables:
+ CLUSTER_NAME: !Ref ClusterName
+ NODE_GROUP_NAME: !Ref NodeGroupName
+ CLUSTER_ARN: !GetAtt EksCluster.Arn
+ CHART_VERSION: !Ref HelmChartVersion
+ PORTKEY_AWS_REGION: !Ref "AWS::Region"
+ PORTKEY_AWS_ACCOUNT_ID: !Ref "AWS::AccountId"
+ PORTKEYAM_ROLE_ARN: !GetAtt PortkeyAM.Arn
+ PORTKEY_DOCKER_USERNAME: !Ref PortkeyDockerUsername
+ PORTKEY_DOCKER_PASSWORD: !Ref PortkeyDockerPassword
+ PORTKEY_CLIENT_AUTH: !Ref PortkeyClientAuth
+ ORGANISATIONS_TO_SYNC: !Ref PortkeyOrgId
+ PORTKEY_GATEWAY_INGRESS_ENABLED: !Ref PortkeyGatewayIngressEnabled
+ PORTKEY_GATEWAY_INGRESS_SUBDOMAIN: !Ref PortkeyGatewayIngressSubdomain
+ PORTKEY_FINE_TUNING_ENABLED: !Ref PortkeyFineTuningEnabled
+```
+
+### Lambda Function
+
+
+
+
+
+#### Steps
+
+1. **Sets up required binaries** - Downloads and configures AWS CLI, kubectl, and Helm binaries in the Lambda environment to enable interaction with AWS services and Kubernetes.
+2. **Configures AWS credentials** - Creates temporary AWS credential files in the Lambda environment to authenticate with AWS services.
+3. **Connects to EKS cluster** - Updates the kubeconfig file to establish a connection with the specified Amazon EKS cluster.
+4. **Manages Helm chart deployment** - Adds the Portkey AI Helm repository and deploys/upgrades the Portkey AI Gateway using Helm charts.
+5. **Handles configuration values** - Creates a values.yaml file with environment-specific configurations and stores it in an S3 bucket for future reference or updates.
+6. **Provides idempotent deployment** - Checks for existing configurations in S3 and uses them if available, allowing the function to be run multiple times for updates without losing custom configurations.
+
+```javascript portkey-hybrid-eks-cloudformation.lambda.js [expandable]
+ const fs = require('fs');
+ const zlib = require('zlib');
+ const { pipeline } = require('stream');
+ const path = require('path');
+ const https = require('https');
+ const { promisify } = require('util');
+ const { execSync } = require('child_process');
+ const { EKSClient, DescribeClusterCommand } = require('@aws-sdk/client-eks');
+
+ async function unzipAwsCli(zipPath, destPath) {
+ // ZIP file format: https://en.wikipedia.org/wiki/ZIP_(file_format)
+ const data = fs.readFileSync(zipPath);
+ let offset = 0;
+
+ // Find end of central directory record
+ const EOCD_SIGNATURE = 0x06054b50;
+ for (let i = data.length - 22; i >= 0; i--) {
+ if (data.readUInt32LE(i) === EOCD_SIGNATURE) {
+ offset = i;
+ break;
+ }
+ }
+
+ // Read central directory info
+ const numEntries = data.readUInt16LE(offset + 10);
+ let centralDirOffset = data.readUInt32LE(offset + 16);
+
+ // Process each file
+ for (let i = 0; i < numEntries; i++) {
+ // Read central directory header
+ const signature = data.readUInt32LE(centralDirOffset);
+ if (signature !== 0x02014b50) {
+ throw new Error('Invalid central directory header');
+ }
+
+ const fileNameLength = data.readUInt16LE(centralDirOffset + 28);
+ const extraFieldLength = data.readUInt16LE(centralDirOffset + 30);
+ const fileCommentLength = data.readUInt16LE(centralDirOffset + 32);
+ const localHeaderOffset = data.readUInt32LE(centralDirOffset + 42);
+
+ // Get filename
+ const fileName = data.slice(
+ centralDirOffset + 46,
+ centralDirOffset + 46 + fileNameLength
+ ).toString();
+
+ // Read local file header
+ const localSignature = data.readUInt32LE(localHeaderOffset);
+ if (localSignature !== 0x04034b50) {
+ throw new Error('Invalid local file header');
+ }
+
+ const localFileNameLength = data.readUInt16LE(localHeaderOffset + 26);
+ const localExtraFieldLength = data.readUInt16LE(localHeaderOffset + 28);
+
+ // Get file data
+ const fileDataOffset = localHeaderOffset + 30 + localFileNameLength + localExtraFieldLength;
+ const compressedSize = data.readUInt32LE(centralDirOffset + 20);
+ const uncompressedSize = data.readUInt32LE(centralDirOffset + 24);
+ const compressionMethod = data.readUInt16LE(centralDirOffset + 10);
+
+ // Create directory if needed
+ const fullPath = path.join(destPath, fileName);
+ const directory = path.dirname(fullPath);
+ if (!fs.existsSync(directory)) {
+ fs.mkdirSync(directory, { recursive: true });
+ }
+
+ // Extract file
+ if (!fileName.endsWith('/')) { // Skip directories
+ const fileData = data.slice(fileDataOffset, fileDataOffset + compressedSize);
+
+ if (compressionMethod === 0) { // Stored (no compression)
+ fs.writeFileSync(fullPath, fileData);
+ } else if (compressionMethod === 8) { // Deflate
+ const inflated = require('zlib').inflateRawSync(fileData);
+ fs.writeFileSync(fullPath, inflated);
+ } else {
+ throw new Error(`Unsupported compression method: ${compressionMethod}`);
+ }
+ }
+
+ // Move to next entry
+ centralDirOffset += 46 + fileNameLength + extraFieldLength + fileCommentLength;
+ }
+ }
+
+
+ async function extractTarGz(source, destination) {
+ // First, let's decompress the .gz file
+ const gunzip = promisify(zlib.gunzip);
+ console.log('Reading source file...');
+ const compressedData = fs.readFileSync(source);
+
+ console.log('Decompressing...');
+ const tarData = await gunzip(compressedData);
+
+ // Now we have the raw tar data
+ // Tar files are made up of 512-byte blocks
+ let position = 0;
+
+ while (position < tarData.length) {
+ // Read header block
+ const header = tarData.slice(position, position + 512);
+ position += 512;
+
+ // Get filename from header (first 100 bytes)
+ const filename = header.slice(0, 100)
+ .toString('utf8')
+ .replace(/\0/g, '')
+ .trim();
+
+ if (!filename) break; // End of tar
+
+ // Get file size from header (bytes 124-136)
+ const sizeStr = header.slice(124, 136)
+ .toString('utf8')
+ .replace(/\0/g, '')
+ .trim();
+ const size = parseInt(sizeStr, 8); // Size is in octal
+
+ console.log(`Found file: ${filename} (${size} bytes)`);
+
+ if (filename === 'linux-amd64/helm') {
+ console.log('Found helm binary, extracting...');
+ // Extract the file content
+ const content = tarData.slice(position, position + size);
+
+ // Write to destination
+ const outputPath = path.join(destination, 'helm');
+ fs.writeFileSync(outputPath, content);
+ console.log(`Helm binary extracted to: ${outputPath}`);
+ return; // We found what we needed
+ }
+
+ // Move to next file
+ position += size;
+ // Move to next 512-byte boundary
+ position += (512 - (size % 512)) % 512;
+ }
+
+ throw new Error('Helm binary not found in archive');
+ }
+
+ async function downloadFile(url, dest) {
+ return new Promise((resolve, reject) => {
+ const file = fs.createWriteStream(dest);
+ https.get(url, (response) => {
+ response.pipe(file);
+ file.on('finish', () => {
+ file.close();
+ resolve();
+ });
+ }).on('error', reject);
+ });
+ }
+
+ async function setupBinaries() {
+ const { STSClient, GetCallerIdentityCommand, AssumeRoleCommand } = require("@aws-sdk/client-sts");
+ const { SignatureV4 } = require("@aws-sdk/signature-v4");
+ const { defaultProvider } = require("@aws-sdk/credential-provider-node");
+ const crypto = require('crypto');
+
+ const tmpDir = '/tmp/bin';
+ if (!fs.existsSync(tmpDir)) {
+ fs.mkdirSync(tmpDir, { recursive: true });
+ }
+
+ // Download and setup AWS CLI
+ console.log('Setting up AWS CLI...');
+ const awsCliUrl = 'https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip';
+ const awsZipPath = `${tmpDir}/awscliv2.zip`;
+ await downloadFile(awsCliUrl, awsZipPath);
+ // Extract using our custom unzip function
+ await unzipAwsCli(awsZipPath, tmpDir);
+ execSync(`chmod +x ${tmpDir}/aws/install ${tmpDir}/aws/dist/aws`);
+ // Install AWS CLI
+ execSync(`${tmpDir}/aws/install --update --install-dir /tmp/aws-cli --bin-dir /tmp/aws-bin`, { stdio: 'inherit' });
+
+ // Download and setup kubectl
+ try {
+ // Download kubectl binary using Node.js https
+ await new Promise((resolve, reject) => {
+ const https = require('https');
+ const fs = require('fs');
+
+ const file = fs.createWriteStream('/tmp/kubectl');
+
+ const request = https.get('https://dl.k8s.io/release/v1.32.1/bin/linux/amd64/kubectl', response => {
+ if (response.statusCode === 302 || response.statusCode === 301) {
+ https.get(response.headers.location, redirectResponse => {
+ redirectResponse.pipe(file);
+ file.on('finish', () => {
+ file.close();
+ resolve();
+ });
+ }).on('error', err => {
+ fs.unlink('/tmp/kubectl', () => {});
+ reject(err);
+ });
+ return;
+ }
+
+ response.pipe(file);
+ file.on('finish', () => {
+ file.close();
+ resolve();
+ });
+ });
+
+ request.on('error', err => {
+ fs.unlink('/tmp/kubectl', () => {});
+ reject(err);
+ });
+ });
+
+ execSync('chmod +x /tmp/kubectl', {
+ stdio: 'inherit'
+ });
+ } catch (error) {
+ console.error('Error installing kubectl:', error);
+ throw error;
+ }
+
+ console.log('Setting up helm...');
+ const helmUrl = 'https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz';
+ const helmTarPath = `${tmpDir}/helm.tar.gz`;
+ await downloadFile(helmUrl, helmTarPath);
+
+ await extractTarGz(helmTarPath, tmpDir);
+ execSync(`chmod +x ${tmpDir}/helm`);
+ fs.unlinkSync(helmTarPath);
+
+ process.env.PATH = `${tmpDir}:${process.env.PATH}`;
+
+ execSync(`/tmp/aws-bin/aws --version`);
+ }
+
+ exports.handler = async (event, context) => {
+ try {
+
+ const { CLUSTER_NAME, NODE_GROUP_NAME, CLUSTER_ARN, CHART_VERSION,
+ PORTKEY_AWS_REGION, PORTKEY_AWS_ACCOUNT_ID, PORTKEYAM_ROLE_ARN,
+ PORTKEY_DOCKER_USERNAME, PORTKEY_DOCKER_PASSWORD,
+ PORTKEY_CLIENT_AUTH, ORGANISATIONS_TO_SYNC } = process.env;
+
+ console.log(process.env)
+
+ if (!CLUSTER_NAME || !PORTKEY_AWS_REGION || !CHART_VERSION ||
+ !PORTKEY_AWS_ACCOUNT_ID || !PORTKEYAM_ROLE_ARN) {
+ throw new Error('Missing one or more required environment variables.');
+ }
+
+ await setupBinaries();
+
+ const awsCredentialsDir = '/tmp/.aws';
+ if (!fs.existsSync(awsCredentialsDir)) {
+ fs.mkdirSync(awsCredentialsDir, { recursive: true });
+ }
+
+ // Write AWS credentials file
+ const credentialsContent = `[default]
+ aws_access_key_id = ${process.env.AWS_ACCESS_KEY_ID}
+ aws_secret_access_key = ${process.env.AWS_SECRET_ACCESS_KEY}
+ aws_session_token = ${process.env.AWS_SESSION_TOKEN}
+ region = ${process.env.PORTKEY_AWS_REGION}
+ `;
+
+ fs.writeFileSync(`${awsCredentialsDir}/credentials`, credentialsContent);
+
+ // Write AWS config file
+ const configContent = `[default]
+ region = ${process.env.PORTKEY_AWS_REGION}
+ output = json
+ `;
+ fs.writeFileSync(`${awsCredentialsDir}/config`, configContent);
+
+ // Set AWS config environment variables
+ process.env.AWS_CONFIG_FILE = `${awsCredentialsDir}/config`;
+ process.env.AWS_SHARED_CREDENTIALS_FILE = `${awsCredentialsDir}/credentials`;
+
+ // Define kubeconfig path
+ const kubeconfigDir = `/tmp/${CLUSTER_NAME.trim()}`;
+ const kubeconfigPath = path.join(kubeconfigDir, 'config');
+
+ // Create the directory if it doesn't exist
+ if (!fs.existsSync(kubeconfigDir)) {
+ fs.mkdirSync(kubeconfigDir, { recursive: true });
+ }
+
+ console.log(`Updating kubeconfig for cluster: ${CLUSTER_NAME}`);
+ execSync(`/tmp/aws-bin/aws eks update-kubeconfig --name ${process.env.CLUSTER_NAME} --region ${process.env.PORTKEY_AWS_REGION} --kubeconfig ${kubeconfigPath}`, {
+ stdio: 'inherit',
+ env: {
+ ...process.env,
+ HOME: '/tmp',
+ AWS_CONFIG_FILE: `${awsCredentialsDir}/config`,
+ AWS_SHARED_CREDENTIALS_FILE: `${awsCredentialsDir}/credentials`
+ }
+ });
+
+ // Set KUBECONFIG environment variable
+ process.env.KUBECONFIG = kubeconfigPath;
+
+ let kubeconfig = fs.readFileSync(kubeconfigPath, 'utf8');
+
+ // Replace the command line to use full path
+ kubeconfig = kubeconfig.replace(
+ 'command: aws',
+ 'command: /tmp/aws-bin/aws'
+ );
+
+ fs.writeFileSync(kubeconfigPath, kubeconfig);
+
+ // Setup Helm repository
+ console.log('Setting up Helm repository...');
+ await new Promise((resolve, reject) => {
+ try {
+ execSync(`helm repo add portkey-ai https://portkey-ai.github.io/helm`, {
+ stdio: 'inherit',
+ env: { ...process.env, HOME: '/tmp' }
+ });
+ resolve();
+ } catch (error) {
+ reject(error);
+ }
+ });
+
+ await new Promise((resolve, reject) => {
+ try {
+ execSync(`helm repo update`, {
+ stdio: 'inherit',
+ env: { ...process.env, HOME: '/tmp' }
+ });
+ resolve();
+ } catch (error) {
+ reject(error);
+ }
+ });
+
+ // Create values.yaml
+ const valuesYAML = `
+ replicaCount: 1
+
+ images:
+ gatewayImage:
+ repository: "docker.io/portkeyai/gateway_enterprise"
+ pullPolicy: IfNotPresent
+ tag: "1.9.0"
+ dataserviceImage:
+ repository: "docker.io/portkeyai/data-service"
+ pullPolicy: IfNotPresent
+ tag: "1.0.2"
+
+ imagePullSecrets: [portkeyenterpriseregistrycredentials]
+ nameOverride: ""
+ fullnameOverride: ""
+
+ imageCredentials:
+ - name: portkeyenterpriseregistrycredentials
+ create: true
+ registry: https://index.docker.io/v1/
+ username: ${PORTKEY_DOCKER_USERNAME}
+ password: ${PORTKEY_DOCKER_PASSWORD}
+
+ useVaultInjection: false
+
+ environment:
+ create: true
+ secret: true
+ data:
+ SERVICE_NAME: portkeyenterprise
+ PORT: "8787"
+ LOG_STORE: s3_assume
+ LOG_STORE_REGION: ${PORTKEY_AWS_REGION}
+ AWS_ROLE_ARN: ${PORTKEYAM_ROLE_ARN}
+ LOG_STORE_GENERATIONS_BUCKET: portkey-gateway
+ ANALYTICS_STORE: control_plane
+ CACHE_STORE: redis
+ REDIS_URL: redis://redis:6379
+ REDIS_TLS_ENABLED: "false"
+ PORTKEY_CLIENT_AUTH: ${PORTKEY_CLIENT_AUTH}
+ ORGANISATIONS_TO_SYNC: ${ORGANISATIONS_TO_SYNC}
+
+ serviceAccount:
+ create: true
+ automount: true
+ annotations: {}
+ name: ""
+
+ podAnnotations: {}
+ podLabels: {}
+
+ podSecurityContext: {}
+ securityContext: {}
+
+ service:
+ type: LoadBalancer
+ port: 8787
+ targetPort: 8787
+ protocol: TCP
+ additionalLabels: {}
+ annotations: {}
+
+ ingress:
+ enabled: ${PORTKEY_GATEWAY_INGRESS_ENABLED}
+ className: ""
+ annotations: {}
+ hosts:
+ - host: ${PORTKEY_GATEWAY_INGRESS_SUBDOMAIN}
+ paths:
+ - path: /
+ pathType: ImplementationSpecific
+ tls: []
+
+ resources: {}
+
+ livenessProbe:
+ httpGet:
+ path: /v1/health
+ port: 8787
+ initialDelaySeconds: 30
+ periodSeconds: 60
+ timeoutSeconds: 5
+ failureThreshold: 5
+ readinessProbe:
+ httpGet:
+ path: /v1/health
+ port: 8787
+ initialDelaySeconds: 30
+ periodSeconds: 60
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 5
+
+ autoscaling:
+ enabled: true
+ minReplicas: 1
+ maxReplicas: 10
+ targetCPUUtilizationPercentage: 80
+
+ volumes: []
+ volumeMounts: []
+ nodeSelector: {}
+ tolerations: []
+ affinity: {}
+ autoRestart: false
+
+ dataservice:
+ name: "dataservice"
+ enabled: ${PORTKEY_FINE_TUNING_ENABLED}
+ containerPort: 8081
+ finetuneBucket: ${PORTKEY_AWS_ACCOUNT_ID}-${PORTKEY_AWS_REGION}-portkey-logs
+ logexportsBucket: ${PORTKEY_AWS_ACCOUNT_ID}-${PORTKEY_AWS_REGION}-portkey-logs
+ deployment:
+ autoRestart: true
+ replicas: 1
+ labels: {}
+ annotations: {}
+ podSecurityContext: {}
+ securityContext: {}
+ resources: {}
+ startupProbe:
+ httpGet:
+ path: /health
+ port: 8081
+ initialDelaySeconds: 60
+ failureThreshold: 3
+ periodSeconds: 10
+ timeoutSeconds: 1
+ livenessProbe:
+ httpGet:
+ path: /health
+ port: 8081
+ failureThreshold: 3
+ periodSeconds: 10
+ timeoutSeconds: 1
+ readinessProbe:
+ httpGet:
+ path: /health
+ port: 8081
+ failureThreshold: 3
+ periodSeconds: 10
+ timeoutSeconds: 1
+ extraContainerConfig: {}
+ nodeSelector: {}
+ tolerations: []
+ affinity: {}
+ volumes: []
+ volumeMounts: []
+ service:
+ type: ClusterIP
+ port: 8081
+ labels: {}
+ annotations: {}
+ loadBalancerSourceRanges: []
+ loadBalancerIP: ""
+ serviceAccount:
+ create: true
+ name: ""
+ labels: {}
+ annotations: {}
+ autoscaling:
+ enabled: false
+ createHpa: false
+ minReplicas: 1
+ maxReplicas: 5
+ targetCPUUtilizationPercentage: 80`
+
+ // Write values.yaml
+ const valuesYamlPath = '/tmp/values.yaml';
+ fs.writeFileSync(valuesYamlPath, valuesYAML);
+
+ const { S3Client, PutObjectCommand, GetObjectCommand } = require("@aws-sdk/client-s3");
+ const s3Client = new S3Client({ region: process.env.PORTKEY_AWS_REGION });
+ try {
+ const response = await s3Client.send(new GetObjectCommand({
+ Bucket: `${process.env.PORTKEY_AWS_ACCOUNT_ID}-${process.env.PORTKEY_AWS_REGION}-portkey-logs`,
+ Key: 'values.yaml'
+ }));
+ const existingValuesYAML = await response.Body.transformToString();
+ console.log('Found existing values.yaml in S3, using it instead of default');
+ fs.writeFileSync(valuesYamlPath, existingValuesYAML);
+ } catch (error) {
+ if (error.name === 'NoSuchKey') {
+ // Upload the default values.yaml to S3
+ await s3Client.send(new PutObjectCommand({
+ Bucket: `${process.env.PORTKEY_AWS_ACCOUNT_ID}-${process.env.PORTKEY_AWS_REGION}-portkey-logs`,
+ Key: 'values.yaml',
+ Body: valuesYAML,
+ ContentType: 'text/yaml'
+ }));
+ console.log('Default values.yaml written to S3 bucket');
+ } else {
+ throw error;
+ }
+ }
+
+ // Install/upgrade Helm chart
+ console.log('Installing helm chart...');
+ await new Promise((resolve, reject) => {
+ try {
+ execSync(`helm upgrade --install portkey-ai portkey-ai/gateway -f ${valuesYamlPath} -n portkeyai --create-namespace --kube-context ${process.env.CLUSTER_ARN} --kubeconfig ${kubeconfigPath}`, {
+ stdio: 'inherit',
+ env: {
+ ...process.env,
+ HOME: '/tmp',
+ PATH: `/tmp/aws-bin:${process.env.PATH}`
+ }
+ });
+ resolve();
+ } catch (error) {
+ reject(error);
+ }
+ });
+
+ return {
+ statusCode: 200,
+ body: JSON.stringify({
+ message: 'EKS installation and helm chart deployment completed successfully',
+ event: event
+ })
+ };
+ } catch (error) {
+ console.error('Error:', error);
+ return {
+ statusCode: 500,
+ body: JSON.stringify({
+ message: 'Error during EKS installation and helm chart deployment',
+ error: error.message
+ })
+ };
+ }
+ };
+```
+
+### Post Deployment Verification
+
+#### Verify Portkey AI Deployment
+
+```bash
+kubectl get all -n portkeyai
+```
+
+
+
+
+
+
+#### Verify Portkey AI Gateway Endpoint
+
+```bash
+export POD_NAME=$(kubectl get pods -n portkeyai -l app.kubernetes.io/name=gateway -o jsonpath="{.items[0].metadata.name}")
+kubectl port-forward $POD_NAME 8787:8787 -n portkeyai
+```
+
+Visiting localhost:8787/v1/health will return `Server is healthy`
+
+
+
+
+
+Your Portkey AI Gateway is now ready to use!
diff --git a/virtual_key_old/product/enterprise-offering/components.mdx b/virtual_key_old/product/enterprise-offering/components.mdx
new file mode 100644
index 00000000..970f6671
--- /dev/null
+++ b/virtual_key_old/product/enterprise-offering/components.mdx
@@ -0,0 +1,55 @@
+---
+title: "Enterprise Components"
+---
+
+Portkey's Enterprise Components provide the core infrastructure needed for production deployments. Each component handles a specific function - analytics, logging, or caching - with multiple implementation options to match your requirements.
+
+---
+
+## Analytics Store
+
+Portkey leverages Clickhouse as the primary Analytics Store for the Control Panel, offering powerful capabilities for handling large-scale analytical workloads.
+
+
+
+
+This hierarchy allows for efficient management of resources and access control across your organization. At the top level, you have your Account, which can contain one or more Organizations. Each Organization can have multiple Workspaces, providing a way to separate teams, projects, or departments within your company.
+
+
+
+
+
+
+### JWKS Configuration
+
+To validate JWTs, you must configure one of the following:
+
+- **JWKS URL**: A URL from which the public keys will be dynamically fetched.
+- **JWKS JSON**: A static JSON containing public keys.
+
+## JWT Requirements
+
+### Supported Algorithm
+
+- JWTs must be signed using **RS256** (RSA Signature with SHA-256).
+
+### Required Claims
+
+Your JWT payload must contain the following claims:
+
+| **Claim Key** | **Description** |
+|------------------------------------|------------------------------------------|
+| `portkey_oid` / `organisation_id` | Unique identifier for the organization. |
+| `portkey_workspace` / `workspace_slug` | Identifier for the workspace. |
+| `scope` / `scopes` | Permissions granted by the token. |
+
+### User Identification
+
+Portkey identifies users in the following order of precedence for logging and metrics:
+
+1. `email_id`
+2. `sub`
+3. `uid`
+
+## Authentication Process
+
+1. The client sends an HTTP request with the JWT in the `x-portkey-api-key` header:
+
+ ```http
+ x-portkey-api-key:
+
+
+Resources that must be deleted before removing a workspace like - Prompts, Prompt partials, Virtual keys, Configs, Guardrails and more.
+
+Once all resources have been removed, enter the workspace name in the confirmation field to proceed with deletion.
+
+
+
+
+
+
+
+### Privacy Protections
+
+At Portkey AI, we prioritize user privacy. Our privacy protocols are designed to comply with international data protection regulations, ensuring that all data is handled responsibly.
+
+We engage in minimal data retention and deploy advanced anonymization technologies to protect personal information and sensitive data from being exposed or improperly used.
+
+Read our privacy policy here - [https://portkey.ai/privacy](https://portkey.ai/privacy)[\-policy](https://portkey.ai/privacy-policy)
+
+## System Integrity and Reliability
+
+### **Network and System Security**:
+
+We protect our systems with advanced firewall technologies and DDoS prevention mechanisms to thwart a wide range of online threats. Our security measures are designed to shield our infrastructure from malicious attacks and ensure continuous service availability.
+
+### **Reliability and Availability**:
+
+Portkey AI offers an industry-leading [99.995% uptime](https://status.portkey.ai), supported by a global network of 310 data centers.
+
+This extensive distribution allows for effective load balancing and edge deployments, minimizing latency and ensuring fast, reliable service delivery across geographical locations.
+
+Our failover mechanisms are sophisticated, designed to handle unexpected scenarios seamlessly and without service interruption.
+
+## Incident Management and Continuous Improvement
+
+### Incident Response
+
+Our proactive incident response team is equipped with the tools and procedures necessary to quickly address and resolve security incidents.
+
+This includes comprehensive risk assessments, immediate containment actions, and detailed investigations to prevent future occurrences.
+
+We maintain transparent communication with our clients throughout the incident management process. Please review our [status page](https://status.portkey.ai) for incident reports.
+
+### Updates and Continuous Improvement
+
+Security at Portkey AI is dynamic; we continually refine our security measures and systems to address emerging threats and incorporate best practices. Our ongoing commitment to improvement helps us stay ahead of the curve in cybersecurity and operational performance.
+
+## Contact Information
+
+For more detailed information or specific inquiries regarding our security measures, please reach out to our support team:
+
+* **Email**: support@portkeyai.com, dpo@portkey.ai
+
+## Useful Links
+
+[Privacy Policy](https://portkey.ai/privacy-policy)
+
+[Terms of Service](https://portkey.ai/terms)
+
+[Data Processing Agreement](https://portkey.ai/dpa)
+
+[Trust Portal](https://trust.portkey.ai)
diff --git a/virtual_key_old/product/guardrails.mdx b/virtual_key_old/product/guardrails.mdx
new file mode 100644
index 00000000..64e22d2a
--- /dev/null
+++ b/virtual_key_old/product/guardrails.mdx
@@ -0,0 +1,665 @@
+---
+title: "Guardrails"
+description: "Ship to production confidently with Portkey Guardrails on your requests & responses"
+---
+
+
+
+Let's see in detail below:
+
+
+
+
+Each Guardrail Check has a custom input field based on its usecase โ just add the relevant details to the form and save your check.
+
+
+
+
+### There are 6 Types of Guardrail Actions
+
+| Action | State | Description | Impact |
+|:-------------- |:-------------------------------------- |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Async** | **TRUE** This is the **default** state | Run the Guardrail checks **asynchronously** along with the LLM request. | Will add no latency to your requestUseful when you only want to log guardrail checks without affecting the request |
+| **Async** | **FALSE** | **On Request** Run the Guardrail check **BEFORE** sending the request to the **LLM** **On Response** Run the Guardrail check **BEFORE** sending the response to the **user** | Will add latency to the requestUseful when your Guardrail critical and you want more orchestration over your request based on the Guardrail result |
+| **Deny** | **TRUE** | **On Request & Response** If any of the Guardrail checks **FAIL**, the request will be killed with a **446** status code. If all of the Guardrail checks **SUCCEED**, the request/response will be sent further with a **200** status code. | This is useful when your Guardrails are critical and upon them failing, you can not run the requestWe would advice running this action on a subset of your requests to first see the impact |
+| **Deny** | **FALSE** This is the **default** state | **On Request & Response** If any of the Guardrail checks **FAIL**, the request will STILL be sent, but with a **246** status code. If all of the Guardrail checks **SUCCEED**, the request/response will be sent further with a **200** status code. | This is useful when you want to log the Guardrail result but do not want it to affect your result |
+| **On Success** | **Send Feedback** | If **all of the** Guardrail checks **PASS**, append your custom defined feedback to the request | We recommend setting up this actionThis will help you build an "Evals dataset" of Guardrail results on your requests over time |
+| **On Failure** | **Send Feedback** | If **any of the** Guardrail checks **FAIL**, append your custom feedback to the request | We recommend setting up this actionThis will help you build an "Evals dataset" of Guardrail results on your requests over time |
+
+Set the relevant actions you want with your checks, name your Guardrail and save it! When you save the Guardrail, you will get an associated `$Guardrail_ID` that you can then add to your request.
+
+---
+
+## 3\. "Enable" the Guardrails through Configs
+
+This is where Portkey's magic comes into play. The Guardrail you created above is yet not an `Active` guardrail because it is not attached to any request.
+
+Configs is one of Portkey's most powerful features and is used to define all kinds of request orchestration - everything from caching, retries, fallbacks, timeouts, to load balancing.
+
+
+
+
+
+---
+
+
+
+
+
+
+
+
+## Understanding Guardrail Response Structure
+
+When Guardrails are enabled and configured to run synchronously (`async=false`), Portkey adds a `hook_results` object to your API responses. This object provides detailed information about the guardrail checks that were performed and their outcomes.
+
+### Hook Results Structure
+
+The `hook_results` object contains two main sections:
+
+```json
+"hook_results": {
+ "before_request_hooks": [...], // Guardrails applied to the input
+ "after_request_hooks": [...] // Guardrails applied to the output
+}
+```
+
+Each section contains an array of guardrail execution results, structured as follows:
+
+
+
+
+## Get Support
+
+If you're implementing guardrails for embeddings and need assistance, reach out to the Portkey team on the [community forum](https://discord.gg/portkey-llms-in-prod-1143393887742861333).
+
+## Learn More
+
+- [Portkey Guardrails Overview](/product/guardrails)
+- [List of Guardrail Checks](/product/guardrails/list-of-guardrail-checks)
+- [Creating Raw Guardrails in JSON](/product/guardrails/creating-raw-guardrails-in-json)
diff --git a/virtual_key_old/product/guardrails/list-of-guardrail-checks.mdx b/virtual_key_old/product/guardrails/list-of-guardrail-checks.mdx
new file mode 100644
index 00000000..cf2baa3d
--- /dev/null
+++ b/virtual_key_old/product/guardrails/list-of-guardrail-checks.mdx
@@ -0,0 +1,226 @@
+---
+title: "List of Guardrail Checks"
+---
+
+Each Guardrail Check has a specific purpose, it's own parameters, supported hooks, and sources.
+
+## Partner Guardrails
+
+
+
+
+## Guardrails Support
+
+PII redaction is supported across 5 guardrail providers:
+
+
+
+
+##### **Special Case: Connecting to Self-Hosted LLMs**
+
+You can manage your privately hosted models alongside commercial providers.
+
+1. When creating a new integration, enable the **"Local/Privately hosted provider"** toggle.
+2. Select the API specification your LLM implements (e.g., OpenAI).
+3. Enter your model's base URL in the **"Custom Host"** field and add any required authentication headers.
+
+This allows you to use your self-hosted models with all Portkey features. For more details, see [Bring Your Own LLM](/product/ai-gateway/byollm).
+
+#### **Step 2: Provision to Workspaces**
+
+Here, you decide which teams get access to this provider and under what conditions.
+
+1. You will see a list of all workspaces within your organization.
+2. Use the toggle next to a workspace name to **enable or disable** access.
+3. For each enabled workspace, you can optionally click **Edit Budget & Rate Limits** to set specific spending caps or request limits that apply *only to that workspace* for this integration.
+4. **(Optional) For Provisioning to New Workspaces:** Toggle on **"Automatically provision this integration for new workspaces"** to ensure any future teams automatically get access with a default budget/rate limit you define.
+5. Click **Next**.
+
+#### **Step 3: Provision Specific Models**
+
+This is where you enforce model governance and control costs.
+
+1. You will see a list of all models available from the provider you're connecting.
+2. You can **Clear all** and then select only the models you wish to approve for use.
+3. **(Optional) For Dynamic Models:** If you're using a provider like Fireworks AI with many community models, you can toggle on **"Automatically enable new models"**. For providers like OpenAI or Azure, we recommend an explicit allow-list for better cost control.
+4. Click **Create Integration**.
+
+**That's it!** You have successfully created and provisioned a centrally managed integration. It will now appear in your **`Connected`** tab. The workspaces you provisioned will see this as an available "AI Provider" in their Model Catalog, with access only to the models you specified and constrained by the budgets you set.
+
+### **Setting Budgets and Rate Limits**
+
+When provisioning an integration to a workspace, you can enforce powerful governance rules by setting budget and rate limits. This gives you granular control over costs and usage patterns.
+
+
+
+
+## Features
+
+Portkey records all your multimodal requests and responses, making it easy to view, monitor, and debug interactions.
+Portkey supports request tracing to help you monitor your applications throughout the lifecycle of a request.
+A comprehensive view of 21+ key metrics. Use it to analyze data, spot trends, and make informed decisions.
+Streamline your data view with customizable filters. Zero in on data that matters most.
+Enrich your LLM APIs with custom metadata. Assign unique tags for swift grouping and troubleshooting.
+Add feedback values and weights to complete the loop.
+Set up budget limits for your provider API keys and gain confidence over your application's costs.
+
+
+
+## Charts
+
+The dashboard provides insights into your [users](/product/observability/analytics#users), [errors](/product/observability/analytics#errors), [cache](/product/observability/analytics#cache), [feedback](/product/observability/analytics#feedback) and also summarizes information by [metadata](/product/observability/analytics#metadata-summary).
+
+### Overview
+
+The overview tab is a 70,000ft view of your application's performance. This highlights the cost, tokens used, mean latency, requests and information on your users and top models.
+
+This is a good starting point to then dive deeper.
+
+### Users
+
+The users tab provides an overview of the user information associated with your Portkey requests. This data is derived from the `user` parameter in OpenAI SDK requests or the special `_user` key in the Portkey [metadata header](/product/observability/metadata).
+
+Portkey currently does not provide analytics on usage patterns for individual team members in your Portkey organization. The users tab is designed to track end-user behavior in your application, not internal team usage.
+
+
+ 
+
+### Errors
+
+Portkey captures errors automatically for API and Accuracy errors. The charts give you a quick sense of error rates allowing you to debug further when needed.
+
+The dashboard also shows you the number of requests rescued by Portkey through the various AI gateway strategies.
+
+
+ 
+
+
+
+### Cache
+
+When you enable cache through the AI gateway, you can view data on the latency improvements and cost savings due to cache.
+
+### Feedback
+
+Portkey allows you to collect feedback on LLM requests through the logs dashboard or via API. You can view analytics on this feedback collected on this dashboard.
+
+### Metadata Summary
+
+Group your request data by metadata parameters to unlock insights on usage. Select the metadata property to use in the dropdown and view the request data grouped by values of that metadata parameter.
+
+This lets you answer questions like:
+
+1. Which users are we spending the most on?
+2. Which organisations have the highest latency?
+
+
+ 
+
diff --git a/virtual_key_old/product/observability/auto-instrumentation.mdx b/virtual_key_old/product/observability/auto-instrumentation.mdx
new file mode 100644
index 00000000..123b6467
--- /dev/null
+++ b/virtual_key_old/product/observability/auto-instrumentation.mdx
@@ -0,0 +1,40 @@
+---
+title: "Auto-Instrumentation [BETA]"
+description: "Portkey's auto-instrumentation allows you to instrument tracing and logging for multiple LLM/Agent frameworks and view the logs, traces, and metrics in a single place."
+---
+
+
+
+## Saved Filters
+
+Quickly access your frequently used filter combinations with the `Saved Filters` feature. Save any set of filters directly from the search bar on the Logs or Analytics pages. Saved filters allow you to instantly apply complex filter rules without retyping them every time.
+
+ 
+
+
+Saved filters are accessible to all organization members, who can also edit, rename, or delete them as needed. Share saved filters with teammates to streamline collaboration and ensure everyone has access to the right data views.
diff --git a/virtual_key_old/product/observability/logs-export-deprecated.mdx b/virtual_key_old/product/observability/logs-export-deprecated.mdx
new file mode 100644
index 00000000..5d0bf6a6
--- /dev/null
+++ b/virtual_key_old/product/observability/logs-export-deprecated.mdx
@@ -0,0 +1,72 @@
+---
+title: "(DEPRECATED) Logs Export"
+description: "Easily access your Portkey logs data for further analysis and reporting"
+---
+
+
+
+
+4. After setting your parameters, click **Request Export**.
+
+
+## Available Export Fields
+
+When configuring your log export, you can select from the following fields:
+
+| Field Name | Description |
+| -------------------- | --------------------------------------------------------- |
+| ID | Unique identifier for the log entry |
+| Trace ID | Identifier for tracing related requests |
+| Created At | Timestamp of the request |
+| Request | Request JSON payload |
+| Response | Response JSON payload |
+| AI Provider | Name of the AI provider used |
+| AI Model | Name of the AI model used |
+| Request Tokens | Number of tokens in the request |
+| Response Tokens | Number of tokens in the response |
+| Total Tokens | Total number of tokens (request + response) |
+| Cost | Cost of the request in cents (USD) |
+| Cost Currency | Currency of the cost (USD) |
+| Response Time | Response time in milliseconds |
+| Status Code | HTTP response status code |
+| Config | Config ID used for the request |
+| Prompt Slug | Prompt ID used for the request |
+| Metadata | Custom metadata key-value pairs |
+
+
+5. Once your export is processed, you'll see it in the exports list with a status indicator:
+ - **Draft**: Export job created but not yet started
+ - **Success**: Export completed successfully
+ - **Failure**: Export job failed. Click on the `Start Again` button to retry the job.
+
+6. Clik on the **Start** button the dashboard to start the logs-export job
+
+7. For completed exports, click the **Download** button to get your logs data file. You can
+
+
+ 
+
+
+
+
+## Share Logs with Teammates
+
+Each log on Portkey has a unique URL. You can copy the link from the address bar and directly share it with anyone in your org.
+
+## Request Status Guide
+
+The Status column on the Logs page gives you a snapshot of the gateway activity for every request.
+
+Portkeyโs gateway featuresโ[Cache](/product/ai-gateway/cache-simple-and-semantic), [Retries](/product/ai-gateway/automatic-retries), [Fallback](/product/ai-gateway/fallbacks), [Loadbalance](/product/ai-gateway/load-balancing), [Conditional Routing](/product/ai-gateway/conditional-routing)โare all tracked here with their exact states (`disabled`, `triggered`, etc.), making it a breeze to monitor and optimize your usage.
+
+**Common Queries Answered:**
+
+* **Is the cache working?**: Enabled caching but unsure if it's active? The Status column will confirm it for you.
+* **How many retries happened?**: Curious about the retry count for a successful request? See it in a glance.
+* **Fallback and Loadbalance**: Want to know if load balance is active or which fallback option was triggered? See it in a glance.
+
+
+ 
+
+
+| Option | ๐ด Inactive State | ๐ข Possible Active States |
+| --------------- | --------------------- | ------------------------------------------------------- |
+| **Cache** | Cache Disabled | Cache Miss,Cache Refreshed,Cache Hit,Cache Semantic Hit |
+| **Retry** | Retry Not Triggered | Retry Success on {x} Tries,Retry Failed |
+| **Fallback** | Fallback Disabled | Fallback Active |
+| **Loadbalance** | Loadbalancer Disabled | Loadbalancer Active |
+
+## Manual Feedback
+
+As you're viewing logs, you can also add manual feedback on the logs to be analysed and filtered later. This data can be viewed on the [feedback analytics dashboards](/product/observability/analytics#feedback).
+
+
+ 
+
+
+## Configs & Prompt IDs in Logs
+
+If your request has an attached [Config](/product/ai-gateway/configs) or if it's originating from a [prompt template](/product/prompt-library), you can see the relevant Config or Prompt IDs separately in the log's details on Portkey. And to dig deeper, you can just click on the IDs and Portkey will take you to the respective Config or Prompt playground where you can view the full details.
+
+
+ 
+
+## Debug Requests with Log Replay
+
+You can rerun any buggy request with just one click, straight from the log details page. The `Replay` button opens your request in a fresh prompt playground where you can rerun the request and edit it right there until it works.
+
+
+ 
+
+
+
+
diff --git a/virtual_key_old/product/observability/metadata.mdx b/virtual_key_old/product/observability/metadata.mdx
new file mode 100644
index 00000000..8fa2b5fa
--- /dev/null
+++ b/virtual_key_old/product/observability/metadata.mdx
@@ -0,0 +1,208 @@
+---
+title: Metadata
+description: Add custom context to your AI requests for better observability and analytics
+---
+
+
+
+
+
+
+### Request Logs
+
+You can also apply any metadata filters to the logs or analytics and filter data by any metadata key you've used:
+
+
+
+
+
+## Enterprise Features
+
+For enterprise users, Portkey offers advanced metadata governance and lets you define metadata at multiple levels:
+
+1. **Request level** - Applied to a single request
+2. **API key level** - Applied to all requests using that key
+3. **Workspace level** - Applied to all requests in a workspace
+
+
+
+## How Tracing Works
+
+Portkey implements OpenTelemetry-compliant tracing. When you include a `trace ID` with your requests, all related LLM calls are grouped together in the Traces View, appearing as "spans" within that trace.
+
+> "Span" is another word for subgrouping of LLM calls. Based on how you instrument, it can refer to another group within your trace or to a single LLM call.
+
+## Trace Tree Structure
+
+Portkey uses a tree data structure for tracing, **similar to OTel.**
+
+Each node in the tree is a span with a unique `spanId` and optional `spanName`. Child spans link to a parent via the `parentSpanId`. Parentless spans become root nodes.
+
+```
+traceId
+โโ parentSpanId
+โ โโ spanId
+โ โโ spanName
+```
+
+| Key - Node | Key - Python | Expected Value | Required? |
+| ------------ | ---------------- | -------------- | --------- |
+| traceId | trace\_id | Unique string | YES |
+| spanId | span\_id | Unique string | NO |
+| spanName | span\_name | string | NO |
+| parentSpanId | parent\_span\_id | Unique string | NO |
+
+---
+
+## Enabling Tracing
+
+You can enable tracing by passing the `trace tree` values while making your request (or while instantiating your client).
+
+Based on these values, Portkey will instrument your requests, and will show the exact trace with its spans on the "Traces" view in Logs page.
+
+
+
+
+Insights from analyzing 2 trillion+ tokens, across 90+ regions and 650+ teams in production. The report contains:
+- Trends shaping AI adoption and LLM provider growth.
+- Benchmarks to optimize speed, cost and reliability.
+- Strategies to scale production-grade AI systems.
+
+
+
+
+---
+
+## Collaborations
+
+Portkey supports various open source projects with additional production capabilities through its custom integrations.
+
+
+
+
+
+
+The dashboard enables you to understand trends in your prompt usage over time and identify potential opportunities for optimization. For more details on using the analytics dashboard and available filters, refer to Portkey's [Analytics documentation](/product/observability/analytics).
+
+## Prompt Logs
+
+The Logs section on Portkey's dashboard provides detailed information about each individual prompt call, giving you visibility into exactly how your prompts are being used in real-time. You can easily filter your prompts using `prompt-id` in the logs view.
+
+
+
+
+
+Each log entry shows the timestamp, model used, request path, user, tokens consumed, cost, and status. This granular data helps you understand exactly how your prompts are performing in production and identify any issues that need attention.
+
+For information on filtering and searching logs, refer to Portkey's [Logs documentation](/product/observability/logs).
+
+
+
+
+This chronological view makes it easy to see how your template is being used and how it's performing over time. You can quickly access detailed information about each call directly from this history view.
+
+The template history is particularly useful when you're iterating on a prompt design, as it allows you to see the immediate impact of your changes. Combined with [Prompt Versioning](/product/prompt-engineering-studio/prompt-versioning), this gives you a complete view of your prompt's evolution and performance.
+
+
+## Next Steps
+
+Now that you understand how to monitor your prompts, explore these related features:
+
+- [Prompt Versioning](/product/prompt-engineering-studio/prompt-versioning) - Track changes to your prompts over time
+- [Prompt API](/product/prompt-engineering-studio/prompt-api) - Integrate optimized prompts into your applications
+- [Prompt Playground](/product/prompt-engineering-studio/prompt-playground) - Test and refine your prompts based on observability insights
+- [Prompt Partials](/product/prompt-engineering-studio/prompt-partial) - Create reusable components for your prompts
+- [Tool Library](/product/prompt-engineering-studio/tool-library) - Enhance your prompts with specialized tools
diff --git a/virtual_key_old/product/prompt-engineering-studio/prompt-partial.mdx b/virtual_key_old/product/prompt-engineering-studio/prompt-partial.mdx
new file mode 100644
index 00000000..aa9c8107
--- /dev/null
+++ b/virtual_key_old/product/prompt-engineering-studio/prompt-partial.mdx
@@ -0,0 +1,174 @@
+---
+title: "Prompt Partials"
+description: "With Prompt Partials, you can save your commonly used templates (which could be your instruction set, data structure explanation, examples etc.) separately from your prompts and flexibly incorporate them wherever required."
+---
+
+
+
+
+You can create a new Partial and use it for any purpose in any of your prompt templates. For example, here's a prompt partial where we are separately storing the instructions:
+
+
+ 
+
+
+Upon saving, each Partial generates a unique ID that you can use inside [prompt templates](/product/prompt-engineering-studio/prompt-playground#prompt-templates).
+
+### Template Engine
+
+Partials also follow the [Mustache template engine](https://mustache.github.io/) and let you easily handle data input at runtime by using tags.
+
+Portkey supports `{{variable}}`, `{{#block}} 
+
+
+When a partial is incorporated in a template, all the variables/blocks defined are also rendered on the Prompt variables section:
+
+
+ 
+
+
+When a new Partial version is **Published**, your partial that is in use in any of the prompt templates also gets automatically updated.
+
+### Using Different Versions of Partials
+
+Similar to prompt templates, you can reference specific versions of your prompt partials in the playground. By default, when you use a partial, Portkey uses the published version, but you can specify any version you want.
+
+To reference a specific version of a partial, use the following syntax:
+
+```
+{{>prompt-partial-id@version-number}}
+```
+
+For example:
+
+```
+{{>pp-instructions-123@5}}
+```
+
+This will use version 5 of the prompt partial with ID "pp-instructions-123".
+
+**Note:** Unlike prompt templates, prompt partials do not support `labels`, `@latest`, `@published` for versioning. You can only reference partials by their version number, `@latest`, or the published version.
+
+### Making a Prompt Completion Request
+
+All the variables/tags defined inside the partial can now be directly called at the time of making a `prompts.completions` request:
+
+
+
+
+### Enhancing Prompts with Tools
+
+Some models support function calling, allowing the AI to request specific information or take actions. The Playground makes it easy to experiment with these capabilities.
+
+Click "Add Tool" button to define functions the model can call. For example:
+
+
+
+
+
+```json
+{
+ "type": "function",
+ "function": {
+ "name": "get_weather",
+ "description": "Get current weather for a location",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "location": {
+ "type": "string",
+ "description": "City and state, e.g., San Francisco, CA"
+ }
+ },
+ "required": ["location"]
+ }
+ }
+}
+```
+
+You can add multiple tools from the [tool library](/product/prompt-engineering-studio/tool-library) for the specific prompt template. You can also choose the parameter "tool_choice" from the UI to control how the model uses the available tools.
+
+This tool definition teaches the model how to request weather information for a specific location.
+
+### Configuring Model Parameters
+
+Each model offers various parameters that affect its output. Access these by clicking the "Parameters" button:
+
+- **Temperature**: Controls randomness (lower = more deterministic)
+- **Top P**: Alternative to temperature for controlling diversity
+- **Max Tokens**: Limits response length
+- **Response Format**: An important setting that allows users to define how they want the model to output. Currently there are 3 options:
+ - Text (default free-form text)
+ - JSON object (structured JSON response)
+ - JSON schema (requires providing a schema in the menu to make the model conform to your exact structure)
+- **Thinking Mode**: Reasoning models think before they answer, producing a long internal chain of thought before responding to the user. You can access to the model's reasoning/thinking process sent by the provider. This feature:
+ - Is only available for select reasoning-capable models (like Claude 3.7 Sonnet)
+ - Can be activated by checking the "Thinking" checkbox in the Parameters panel
+ - Allows you to set a budget of tokens dedicated specifically to the thinking process (if the provider supports it)
+
+And more... Experiment with these settings to find the perfect balance for your use case.
+
+### Pretty Mode vs JSON Mode
+
+The Playground offers two interface modes for working with prompts:
+
+**Pretty Mode**
+
+The default user-friendly interface with formatted messages and simple controls. This is ideal for most prompt engineering tasks and provides an intuitive way to craft and test prompts.
+
+**JSON Mode**
+
+For advanced users who need granular control, you can toggle to JSON mode by clicking the "JSON" button. This reveals the raw JSON structure of your prompt, allowing for precise editing and advanced configurations.
+
+JSON mode is particularly useful when:
+- Working with multimodal inputs like images
+- Creating complex conditional logic
+- Defining precise message structures
+- Debugging API integration issues
+
+You can switch between modes at any time using the toggle in the interface.
+
+### Multimodality: Working with Images
+
+For multimodal models that support images, you can upload images directly in the Playground using the ๐งท icon on the message input box.
+
+Alternatively, you can use JSON mode to incorporate images using variables. Toggle from PRETTY to JSON mode using the button on the dashboard, then structure your prompt like this:
+
+```json
+[
+ {
+ "content": [
+ {
+ "type": "text",
+ "text": "You're a helpful assistant."
+ }
+ ],
+ "role": "system"
+ },
+ {
+ "role": "user",
+ "content": [
+ { "type": "text", "text": "what's in this image?" },
+ {
+ "type": "image_url",
+ "image_url": {
+ "url" : "{{your_image_url}}"
+ }
+ }
+ ]
+ }
+]
+```
+
+Now you can pass the image URL as a variable in your prompt template, and the model will be able to analyze the image content.
+
+# Prompt Templates
+
+**Portkey uses** [**Mustache**](https://mustache.github.io/mustache.5.html) **under the hood to power the prompt templates.**
+
+Mustache is a commonly used logic-less templating engine that follows a simple schema for defining variables and more.
+
+With Mustache, prompt templates become even more extensible by letting you incorporate various `{{tags}}` in your prompt template and easily pass your data.
+
+The most common usage of mustache templates is for `{{variables}}`, used to pass a value at runtime.
+
+### Using Variables in Prompt Templates
+
+Let's look at the following template:
+
+
+ 
+
+
+As you can see, `{{customer_data}}` and `{{chat_query}}` are defined as variables in the template and you can pass their value at runtime:
+
+
+
+
+**And the prompt template uses the partial like this:**
+
+ 
+
+
+**We can pass the data object inside the variables:**
+
+
+
+
+### Publishing Prompts
+
+Publishing a prompt version marks it as the default version that will be used when no specific version is requested. This is especially important for production environments.
+
+Updating the Prompt does not automatically update your prompt in production. While updating, you can tick `Publish prompt changes` which will also update your prompt deployment to the latest version.
+
+1. Create and test your new prompt version
+2. When ready for production, click "Update" and check "Publish prompt changes"
+3. Portkey will save the new version and mark it as the published version
+4. All default API calls will now use this version
+
+
+ 
+
+
+### Viewing Version History
+
+**All** of your prompt versions can be seen by clicking the `Version History` button on the playground:
+
+
+ 
+
+
+You can `Restore` or `Publish` any of the previous versions by clicking on the ellipsis menu.
+
+### Comparing Versions
+
+To compare different versions of your prompt:
+
+1. Select the versions you want to compare from the version history panel
+2. Click "Compare on playground" to see a side-by-side of different prompt versions
+
+This helps you understand how prompts have evolved and which changes might have impacted performance.
+
+## Using Different Prompt Versions
+
+By default, when you pass the `PROMPT_ID` in `prompts.completions.create` method, Portkey sends the request to the `Published` version of your prompt.
+
+You can also call any specific prompt version by appending version identifiers to your `PROMPT_ID`.
+
+### Version Number References
+
+**For example:**
+
+```js
+response = portkey.prompts.completions.create(
+ prompt_id="pp-classification-prompt@12",
+ variables={ }
+)
+```
+
+Here, the request is sent to **Version 12** of the prompt template.
+
+### Special Version References
+
+Portkey supports special version references:
+
+```js
+// Latest version (may not be published)
+response = portkey.prompts.completions.create(
+ prompt_id="pp-classification-prompt@latest",
+ variables={ }
+)
+
+// Published version (default when no suffix is provided)
+response = portkey.prompts.completions.create(
+ prompt_id="pp-classification-prompt",
+ variables={ }
+)
+```
+
+**Important Notes:**
+* `@latest` refers to the most recent version, regardless of publication status
+* When no suffix is provided, Portkey defaults to the `Published` version
+* Each version is immutable once created - to make changes, you must create a new version
+
+## Prompt Labels
+
+Labels provide a more flexible and meaningful way to reference prompt versions compared to version numbers. You can add version tags/labels like `platform-team`, `gpt-model-prompt` to any prompt version to track changes and call them directly:
+
+
+
+
+### Using Labels in Your Code
+
+