+
+
+That's it! You now get:
+- ✅ Full observability (costs, latency, logs)
+- ✅ Dynamic model selection per request
+- ✅ Automatic fallbacks and retries (via configs)
+- ✅ Budget controls per team/project
+
+## Why Add Portkey to Langchain?
+
+Langchain handles application orchestration. Portkey adds production features:
+
+
-
+Just change the model string—everything else stays the same:
-## Using Virtual Keys for Multiple Models
+```javascript
+// OpenAI
+const openaiModel = new ChatOpenAI({
+ model: "@openai-prod/gpt-4o",
+ configuration: { baseURL: "https://api.portkey.ai/v1" },
+ apiKey: "PORTKEY_API_KEY"
+});
-Portkey supports [Virtual Keys](/product/ai-gateway/virtual-keys) which are an easy way to store and manage API keys in a secure vault. Lets try using a Virtual Key to make LLM calls.
+// Anthropic
+const anthropicModel = new ChatOpenAI({
+ model: "@anthropic-prod/claude-sonnet-4",
+ configuration: { baseURL: "https://api.portkey.ai/v1" },
+ apiKey: "PORTKEY_API_KEY"
+});
-#### 1\. Create a Virtual Key in your Portkey account and the id
+// Google Gemini
+const geminiModel = new ChatOpenAI({
+ model: "@google-prod/gemini-2.0-flash",
+ configuration: { baseURL: "https://api.portkey.ai/v1" },
+ apiKey: "PORTKEY_API_KEY"
+});
+```
-Let's try creating a new Virtual Key for Mistral like this
+
-
+## Using with Langchain Agents
-#### 2\. Use Virtual Keys in the Portkey Headers
+Langchain agents are the primary use case. Portkey works seamlessly with agent workflows:
-The `virtualKey` parameter sets the authentication and provider for the AI provider being used. In our case we're using the Mistral Virtual key.
+```javascript
+import { ChatOpenAI } from "@langchain/openai";
+import { tool } from "@langchain/core/tools";
+import { createReactAgent } from "@langchain/langgraph/prebuilt";
+import { z } from "zod";
+
+// Define tools
+const searchTool = tool(
+ async ({ query }) => `Results for: ${query}`,
+ {
+ name: "search",
+ description: "Search for information",
+ schema: z.object({ query: z.string() })
+ }
+);
+
+const weatherTool = tool(
+ async ({ location }) => `Weather in ${location}: Sunny, 72°F`,
+ {
+ name: "get_weather",
+ description: "Get weather for a location",
+ schema: z.object({ location: z.string() })
+}
+);
-
+
-const portkeyConf = {
- baseURL: PORTKEY_GATEWAY_URL,
- defaultHeaders: createHeaders({apiKey: PORTKEY_API_KEY, virtualKey: MISTRAL_VK})
-}
+## Works With All Langchain Features
+
+✅ **Agents** - Full compatibility with LangGraph agents
+✅ **LCEL** - LangChain Expression Language
+✅ **Chains** - All chain types supported
+✅ **Streaming** - Token-by-token streaming
+✅ **Tool Calling** - Function/tool calling
+✅ **LangGraph** - Complex workflows
+
+### Streaming
-const chatModel = new ChatOpenAI({
- apiKey: "X",
- configuration: portkeyConf,
- model: "mistral-large-latest"
+```javascript
+const model = new ChatOpenAI({
+ model: "@openai-prod/gpt-4o",
+ configuration: { baseURL: "https://api.portkey.ai/v1" },
+ apiKey: "PORTKEY_API_KEY",
+ streaming: true
});
-const response = await chatModel.invoke("What is the meaning of life, universe and everything?");
-console.log("Response:", response);
+const stream = await model.stream("Write a short story");
+for await (const chunk of stream) {
+ process.stdout.write(chunk.content);
+}
```
-The Portkey AI gateway will authenticate the API request to Mistral and get the response back in the OpenAI format for you to consume.
-
-The AI gateway extends Langchain's `ChatOpenAI` class making it a single interface to call any provider and any model.
+### Chains & Prompts
-## Embeddings
+```javascript
+import { ChatOpenAI } from "@langchain/openai";
+import { ChatPromptTemplate } from "@langchain/core/prompts";
-Embeddings in Langchain through Portkey work the same way as the Chat Models using the `OpenAIEmbeddings` class. Let's try to create an embedding using OpenAI's embedding model
+const model = new ChatOpenAI({
+ model: "@openai-prod/gpt-4o",
+ configuration: { baseURL: "https://api.portkey.ai/v1" },
+ apiKey: "PORTKEY_API_KEY"
+});
-```js
-import { OpenAIEmbeddings } from "@langchain/openai";
-import { createHeaders, PORTKEY_GATEWAY_URL} from "portkey-ai"
+const prompt = ChatPromptTemplate.fromMessages([
+ ["human", "Tell me a short joke about {topic}"]
+]);
-const PORTKEY_API_KEY = "...";
-const OPENAI_VK = "..." // Add OpenAI's Virtual Key created in Portkey
+const chain = prompt.pipe(model);
+const response = await chain.invoke({ topic: "ice cream" });
+console.log(response.content);
+```
-const portkeyConf = {
- baseURL: PORTKEY_GATEWAY_URL,
- defaultHeaders: createHeaders({apiKey: PORTKEY_API_KEY, virtualKey: OPENAI_VK})
-}
+### Tool Calling
-/* Create instance */
-const embeddings = new OpenAIEmbeddings({
- apiKey: "X",
- configuration: portkeyConf,
+```javascript
+import { ChatOpenAI } from "@langchain/openai";
+import { z } from "zod";
+import { tool } from "@langchain/core/tools";
+
+const getWeather = tool(
+ async ({ location }) => `Weather in ${location}: Sunny, 72°F`,
+ {
+ name: "get_weather",
+ description: "Get current weather in a location",
+ schema: z.object({
+ location: z.string().describe("City and state, e.g. San Francisco, CA")
+ })
+ }
+);
+
+const model = new ChatOpenAI({
+ model: "@openai-prod/gpt-4o",
+ configuration: { baseURL: "https://api.portkey.ai/v1" },
+ apiKey: "PORTKEY_API_KEY"
});
-/* Embed queries */
-const res = await embeddings.embedQuery("Hello world");
-console.log("Response:", res);
+const modelWithTools = model.bindTools([getWeather]);
+const response = await modelWithTools.invoke("What's the weather in NYC?");
+console.log(response.tool_calls);
```
-## Chains & Prompts
+## Dynamic Model Selection
-[Chains](https://python.langchain.com/docs/modules/chains/) enable the integration of various Langchain concepts for simultaneous execution while Langchain supports [Prompt Templates](https://python.langchain.com/docs/modules/model%5Fio/prompts/) to construct inputs for language models. Lets see how this would work with Portkey
+For dynamic model routing based on query complexity or task type, use **Portkey Configs** with conditional routing:
-```js
+```javascript
import { ChatOpenAI } from "@langchain/openai";
-import { ChatPromptTemplate } from "@langchain/core/prompts";
-import { createHeaders, PORTKEY_GATEWAY_URL} from "portkey-ai"
+import { createHeaders } from "portkey-ai";
-const PORTKEY_API_KEY = "...";
-const OPENAI_VK = "..." // Add OpenAI's Virtual Key created in Portkey
-
-const portkeyConf = {
- baseURL: PORTKEY_GATEWAY_URL,
- defaultHeaders: createHeaders({apiKey: PORTKEY_API_KEY, virtualKey: OPENAI_VK})
-}
+// Define routing config (created in Portkey dashboard)
+const config = {
+ strategy: {
+ mode: "conditional",
+ conditions: [
+ {
+ query: { "metadata.complexity": { "$eq": "simple" } },
+ then: "cheap-model"
+ },
+ {
+ query: { "metadata.complexity": { "$eq": "complex" } },
+ then: "advanced-model"
+ }
+ ],
+ default: "cheap-model"
+ },
+ targets: [
+ {
+ name: "cheap-model",
+ override_params: { model: "@openai-prod/gpt-4o-mini" }
+ },
+ {
+ name: "advanced-model",
+ override_params: { model: "@openai-prod/o1" }
+ }
+ ]
+};
+
+const model = new ChatOpenAI({
+ model: "gpt-4o",
+ configuration: {
+ baseURL: "https://api.portkey.ai/v1",
+ defaultHeaders: createHeaders({ config })
+ },
+ apiKey: "PORTKEY_API_KEY"
+});
-// Initialise the chat model
-const chatModel = new ChatOpenAI({
- apiKey: "X",
- configuration: portkeyConf
+// Route to cheap model
+const response1 = await model.invoke("What is 2+2?", {
+ metadata: { complexity: "simple" }
});
-// Define the chat prompt template
-const prompt = ChatPromptTemplate.fromMessages([
- ["human", "Tell me a short joke about {topic}"],
-]);
+// Route to advanced model
+const response2 = await model.invoke("Solve this differential equation...", {
+ metadata: { complexity: "complex" }
+});
+```
-// Invoke the chain with the prompt and chat model
-const chain = prompt.pipe(chatModel);
-const res = await chain.invoke({ topic: "ice cream" });
+All routing decisions are tracked in Portkey with full observability—see which models were used, costs per model, and performance comparisons.
-console.log(res)
-```
+
-
+// After (add configuration, change model and apiKey)
+const model = new ChatOpenAI({
+ model: "@openai-prod/gpt-4o", // Add provider slug
+ configuration: {
+ baseURL: "https://api.portkey.ai/v1" // Add this
+ },
+ apiKey: "PORTKEY_API_KEY", // Change to Portkey key
+ temperature: 0.7 // Keep existing params
+});
+```
-This is extremely powerful since we gain control and visibility over the agent flows so we can identify problems and make updates as needed.
+**Benefits:**
+- Zero code changes to your existing Langchain logic
+- Instant observability for all requests
+- Production-grade reliability features
+- Cost controls and budgets
+
+## Next Steps
+
+
-This setup enables Portkey's advanced features for Langchain.
+That's it! You now get:
+- ✅ Full observability (costs, latency, logs)
+- ✅ Dynamic model selection per request
+- ✅ Automatic fallbacks and retries (via configs)
+- ✅ Budget controls per team/project
-## Key Portkey Features for Langchain
+## Why Add Portkey to Langchain?
-Routing Langchain requests via Portkey's `ChatOpenAI` interface unlocks powerful capabilities:
+Langchain handles application orchestration. Portkey adds production features:
Use `ChatOpenAI` for OpenAI, Anthropic, Gemini, Mistral, and more. Switch providers easily with Virtual Keys or Configs.
+Reduce latency and costs with Portkey's Simple, Semantic, or Hybrid caching, enabled via Configs.
+Build robust apps with retries, timeouts, fallbacks, and load balancing, configured in Portkey.
+Get deep insights: LLM usage, costs, latency, and errors are automatically logged in Portkey.
-Manage, version, and use prompts from Portkey's Prompt Library within Langchain.
-Securely manage LLM provider API keys using Portkey Virtual Keys in your Langchain setup.
+
+
-portkey_cached_headers = createHeaders(
- api_key=PORTKEY_API_KEY,
- provider="@openai"
-)
+## Works With All Langchain Features
-llm_cached = ChatOpenAI(
- api_key="placeholder_key", # Config can handle auth via virtual_key
- base_url=PORTKEY_GATEWAY_URL,
- default_headers=portkey_cached_headers
-)
+✅ **Agents** - Full compatibility with `create_agent`
+✅ **LCEL** - LangChain Expression Language
+✅ **Chains** - All chain types supported
+✅ **Streaming** - Token-by-token streaming
+✅ **Tool Calling** - Function/tool calling
+✅ **LangGraph** - Complex workflows
-# response1 = llm_cached.invoke("Capital of France?")
-# response2 = llm_cached.invoke("Tell me France's capital.")
-```
+### Streaming
-Similar requests can now hit the cache. Monitor cache performance in your Portkey dashboard.
+```python
+model = ChatOpenAI(
+ model="@openai-prod/gpt-4o",
+ base_url="https://api.portkey.ai/v1",
+ api_key="PORTKEY_API_KEY",
+ streaming=True
+)
----
+for chunk in model.stream("Write a short story"):
+ print(chunk.content, end="", flush=True)
+```
-## 3. Enhanced Reliability
+### Tool Calling
-Portkey improves Langchain app resilience via Configs:
+```python
+from pydantic import BaseModel, Field
-* **Retries:** Auto-retry failed LLM requests.
-* **Fallbacks:** Define backup LLMs if a primary fails.
-* **Load Balancing:** Distribute requests across keys or models.
-* **Timeouts:** Set max request durations.
+class GetWeather(BaseModel):
+ '''Get current weather in a location'''
+ location: str = Field(..., description="City and state, e.g. San Francisco, CA")
-### Example: Fallbacks with Retries
+model = ChatOpenAI(
+ model="@openai-prod/gpt-4o",
+ base_url="https://api.portkey.ai/v1",
+ api_key="PORTKEY_API_KEY"
+)
-1. **Define/Save Portkey Config:** Create a Config for retries and fallbacks (e.g., `gpt-4o` then `claude-3-sonnet`).
+model_with_tools = model.bind_tools([GetWeather])
+response = model_with_tools.invoke("What's the weather in NYC?")
+print(response.tool_calls)
+```
- ```json
- // Example Portkey Config for reliability
- {
- "strategy": { "mode": "fallback" },
- "targets": [
- { "override_params": {"model": "gpt-4o"}, "provider":"@vk_openai", "retry": {"count": 2} },
- { "override_params": {"model": "claude-3-sonnet-20240229"}, "provider":"@vk_anthropic" }
- ]
- }
- ```
- Assume saved Config ID is `cfg_reliable_123`.
+## Dynamic Model Selection
-2. **Use Config ID in `createHeaders`:**
+For dynamic model routing based on query complexity or task type, use **Portkey Configs** with conditional routing:
```python
from langchain_openai import ChatOpenAI
-from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders
-import os
-
-PORTKEY_API_KEY = os.environ.get("PORTKEY_API_KEY")
-PORTKEY_RELIABLE_CONFIG_ID = "cfg_reliable_123"
-
-portkey_reliable_headers = createHeaders(
- api_key=PORTKEY_API_KEY,
- provider="@openai"
+from portkey_ai import createHeaders
+
+# Define routing config (created in Portkey dashboard)
+config = {
+ "strategy": {
+ "mode": "conditional",
+ "conditions": [
+ {
+ "query": {"metadata.complexity": {"$eq": "simple"}},
+ "then": "cheap-model"
+ },
+ {
+ "query": {"metadata.complexity": {"$eq": "complex"}},
+ "then": "advanced-model"
+ }
+ ],
+ "default": "cheap-model"
+ },
+ "targets": [
+ {
+ "name": "cheap-model",
+ "override_params": {"model": "@openai-prod/gpt-4o-mini"}
+ },
+ {
+ "name": "advanced-model",
+ "override_params": {"model": "@openai-prod/o1"}
+ }
+ ]
+}
+
+model = ChatOpenAI(
+ model="gpt-4o", # Default model
+ base_url="https://api.portkey.ai/v1",
+ api_key="PORTKEY_API_KEY",
+ default_headers=createHeaders(config=config)
)
-llm_reliable = ChatOpenAI(
- api_key="placeholder_key",
- base_url=PORTKEY_GATEWAY_URL,
- default_headers=portkey_reliable_headers
+# Route to cheap model
+response1 = model.invoke(
+ "What is 2+2?",
+ config={"metadata": {"complexity": "simple"}}
)
-# response = llm_reliable.invoke("Poem on resilient AI.")
+# Route to advanced model
+response2 = model.invoke(
+ "Solve this differential equation...",
+ config={"metadata": {"complexity": "complex"}}
+)
```
-Offload complex logic to Portkey Configs, keeping Langchain code clean.
+### Use Cases
----
+**1. Cost Optimization**
-## 4. Full Observability
+Route by query complexity automatically:
-Routing Langchain `ChatOpenAI` via Portkey provides instant, comprehensive observability:
+```python
+def smart_invoke(prompt, complexity="simple"):
+ return model.invoke(
+ prompt,
+ config={"metadata": {"complexity": complexity}}
+ )
+
+# Automatic routing
+answer1 = smart_invoke("What is 2+2?", complexity="simple")
+answer2 = smart_invoke("Explain quantum mechanics", complexity="complex")
+```
-* **Logged Requests:** Detailed logs of requests, responses, latencies, costs.
-* **Tracing:** Understand call lifecycles.
-* **Performance Analytics:** Monitor metrics, track usage.
-* **Debugging:** Pinpoint errors quickly.
+**2. Model Specialization by Task**
-This is crucial for monitoring and optimizing production Langchain apps.
+Route different task types to specialized models:
-
-
-
+```python
+config = {
+ "strategy": {
+ "mode": "conditional",
+ "conditions": [
+ {"query": {"metadata.task": {"$eq": "code"}}, "then": "coding-model"},
+ {"query": {"metadata.task": {"$eq": "creative"}}, "then": "creative-model"}
+ ],
+ "default": "coding-model"
+ },
+ "targets": [
+ {
+ "name": "coding-model",
+ "override_params": {"model": "@openai-prod/gpt-4o"}
+ },
+ {
+ "name": "creative-model",
+ "override_params": {"model": "@anthropic-prod/claude-sonnet-4"}
+ }
+ ]
+}
----
+def route_by_task(prompt, task_type):
+ return model.invoke(
+ prompt,
+ config={"metadata": {"task": task_type}}
+ )
-## 5. Prompt Management
+code = route_by_task("Write a sorting algorithm", task_type="code")
+story = route_by_task("Write a sci-fi story", task_type="creative")
+```
-Portkey's Prompt Library helps manage prompts effectively:
+**3. Dynamic Agent Model Selection**
-* **Version Control:** Store and track prompt changes.
-* **Parameterized Prompts:** Use variables with [mustache templating](/product/prompt-library/prompt-templates#templating-engine).
-* **Sandbox:** Test prompts with different LLMs in Portkey.
+Use different models for different agent steps:
-### Using Portkey Prompts in Langchain
+```python
+from langchain.agents import create_agent
+from langchain.tools import tool
+
+@tool
+def complex_calculation(query: str) -> str:
+ """Perform complex calculations."""
+ return "42"
+
+model = ChatOpenAI(
+ model="gpt-4o",
+ base_url="https://api.portkey.ai/v1",
+ api_key="PORTKEY_API_KEY",
+ default_headers=createHeaders(config=config)
+)
-1. Create prompt in Portkey, get `Prompt ID`.
-2. Use Portkey SDK to render prompt with variables.
-3. Transform rendered prompt to Langchain message format.
-4. Pass messages to Portkey-configured `ChatOpenAI`.
+agent = create_agent(model, tools=[complex_calculation])
-```python
-import os
-from langchain_openai import ChatOpenAI
-from langchain_core.messages import SystemMessage, HumanMessage
-from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders, Portkey
+# Agent routes based on task complexity
+result = agent.invoke({
+ "messages": [{"role": "user", "content": "Calculate quantum probabilities"}],
+ "metadata": {"complexity": "complex"}
+})
+```
-PORTKEY_API_KEY = os.environ.get("PORTKEY_API_KEY")
-client = Portkey(api_key=PORTKEY_API_KEY)
+All routing decisions are tracked in Portkey with full observability—see which models were used, costs per model, and performance comparisons.
-PROMPT_ID = "pp-story-generator" # Your Portkey Prompt ID
+
+
-## Enabling Portkey Features
+That's it! You now get:
+- ✅ Full observability (costs, latency, logs)
+- ✅ Dynamic model selection per request
+- ✅ Automatic fallbacks and retries (via configs)
+- ✅ Budget controls per team/project
-By routing your LlamaIndex requests through Portkey, you get access to the following production-grade features:
+## Why Add Portkey to LlamaIndex?
-Call various LLMs like Anthropic, Gemini, Mistral, Azure OpenAI, Google Vertex AI, and AWS Bedrock with minimal code changes.
-Speed up your requests and save money on LLM calls by storing past responses in the Portkey cache. Choose between Simple and Semantic cache modes.
+Set up fallbacks between different LLMs or providers, load balance your requests across multiple instances or API keys, set automatic retries, and request timeouts.
+Portkey automatically logs all the key details about your requests, including cost, tokens used, response time, request and response bodies, and more. Send custom metadata and trace IDs for better analytics and debugging.
+Use Portkey as a centralized hub to store, version, and experiment with prompts across multiple LLMs, and seamlessly retrieve them in your LlamaIndex app for easy integration.
+Improve your LlamaIndex app by capturing qualitative & quantitative user feedback on your requests.
-Set budget limits on provider API keys and implement fine-grained user roles and permissions for both the app and the Portkey APIs.
-
-
-## Overriding a Saved Config
+
-
-
-[**Check out Observability docs here.**](/product/observability)
-
+Filter and analyze logs by metadata in the Portkey dashboard.
-## 5\. Prompt Management
+
-
-
-[**Read more about Portkey's Security & Enterprise offerings here**](/product/enterprise-offering).
-
-
-## Join Portkey Community
-
-Join the Portkey Discord to connect with other practitioners, discuss your LlamaIndex projects, and get help troubleshooting your queries.
+