Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -604,6 +604,39 @@ const realtime = new OpenAIRealtimeProvider({
});
```

### MCP Tool Servers

Connect your agent to any number of [Model Context Protocol](https://modelcontextprotocol.io/) servers. The SDK loads tool definitions from each server, namespaces them, and routes every invocation through MCP automatically.

```tsx
const realtime = new OpenAIRealtimeProvider({
apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY!,
mcpServers: [
{
id: "docs",
url: process.env.NEXT_PUBLIC_DOCS_MCP_URL!,
headers: {
Authorization: `Bearer ${process.env.NEXT_PUBLIC_DOCS_MCP_TOKEN!}`,
Comment on lines +613 to +619
Copy link

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example encourages putting NEXT_PUBLIC_OPENAI_API_KEY and NEXT_PUBLIC_DOCS_MCP_TOKEN into NEXT_PUBLIC_* environment variables, which are bundled into client-side JavaScript and fully exposed to any user of the app. An attacker can trivially extract these values from the browser, then reuse your OpenAI API key and MCP token to call your backends, exhaust quotas, or access data. To avoid leaking these secrets, keep them in server-only configuration (non-public env vars or a backend proxy) and do not expose them to untrusted clients.

Copilot uses AI. Check for mistakes.
},
toolNamePrefix: "docs__", // optional (defaults to `${id}__`)
},
{
id: "inventory",
url: process.env.NEXT_PUBLIC_INVENTORY_MCP_URL!,
},
],
});
```

What you get for free:

- Streamable HTTP transport management for each MCP server.
- Automatic pagination to fetch every declared tool.
- Guaranteed unique tool names via configurable prefixes so OpenAI can call them safely.
- Tool outputs are converted into OpenAI function responses, keeping the realtime session in sync.

You can mix MCP-provided tools with traditional `tools` functions to blend local logic and remote capabilities.

### Mock Provider (Development)

Perfect for testing without API keys:
Expand Down
45 changes: 44 additions & 1 deletion packages/core/src/types/realtime.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,48 @@
import { RealtimeMessage, Conversation, ChatStatus } from './conversation';
import { MouthState, PhonemeData } from './audio';

/**
* Configuration for connecting to an MCP server
*/
export interface MCPServerConfig {
/**
* Unique identifier for the server. Used for tool name prefixing.
*/
id: string;
/**
* Streamable HTTP endpoint for the MCP server.
*/
url: string;
/**
* Future-proofing for additional transports. Currently only streamable-http is supported.
*/
transport?: 'streamable-http';
/**
* Optional HTTP headers to send on every transport request (e.g., auth tokens).
*/
headers?: Record<string, string>;
/**
* Client identity used when registering with the MCP server.
*/
client?: {
name?: string;
version?: string;
/**
* Capabilities object forwarded directly to the MCP client.
*/
capabilities?: Record<string, unknown>;
};
/**
* Optional prefix used when exposing server tools inside the realtime provider.
* Defaults to `${id}__`.
*/
toolNamePrefix?: string | null;
/**
* Extra metadata for application-specific use.
*/
metadata?: Record<string, string>;
}

/**
* Tool definition for function calling
*/
Expand Down Expand Up @@ -31,6 +73,7 @@ export interface RealtimeConfig {
instructions?: string;
temperature?: number;
tools?: RealtimeTool[];
mcpServers?: MCPServerConfig[];
turnServers?: RTCIceServer[];
speed?: number;
enableLipSync?: boolean;
Expand Down Expand Up @@ -85,4 +128,4 @@ export interface RealtimeProvider extends RealtimeEvents {
enableMicrophone(): void;
disableMicrophone(): void;
isMicrophoneEnabled(): boolean;
}
}
35 changes: 34 additions & 1 deletion packages/providers/openai-realtime/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -312,6 +312,39 @@ const realtime = new OpenAIRealtimeProvider({
realtime.registerFunction(weatherTool);
```

### MCP Servers (Model Context Protocol)

You can mount any number of MCP servers and expose their tools directly to OpenAI. Each server describes its tools via the MCP spec; the provider handles the rest (connections, pagination, name-spacing, execution, and returning results to the realtime session).

```tsx
const realtime = new OpenAIRealtimeProvider({
apiKey: process.env.OPENAI_API_KEY || "",
mcpServers: [
{
id: "docs",
url: process.env.DOCS_MCP_URL!,
headers: {
Authorization: `Bearer ${process.env.DOCS_MCP_TOKEN!}`,
},
toolNamePrefix: "docs__", // optional (defaults to `${id}__`)
},
{
id: "inventory",
url: process.env.INVENTORY_MCP_URL!,
},
],
});
```

`MCPServerConfig` options:

- `id`: unique identifier. Also used as the default prefix for tools (`${id}__toolName`).
- `url`: Streamable HTTP endpoint for the server (e.g., `https://.../mcp`).
- `headers`: optional HTTP headers for auth.
- `toolNamePrefix`: override the default prefix (set to `null` to keep original names).
Copy link

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When toolNamePrefix is set to null to preserve original tool names, there's a risk of name collisions between tools from different MCP servers or between MCP tools and manual tools. The code handles this by overwriting (last one wins in the Map), but users aren't warned about this in the documentation. Consider adding a note about the collision risk when using null prefix.

Suggested change
- `toolNamePrefix`: override the default prefix (set to `null` to keep original names).
- `toolNamePrefix`: override the default prefix (set to `null` to keep original names). **Warning:** when set to `null`, tool names are not namespaced, so tools from different MCP servers or manual tools can collide; in case of a collision, the last registered tool overwrites earlier ones.

Copilot uses AI. Check for mistakes.

Remote tools are converted into OpenAI function definitions alongside any local `tools`, and responses are sent back through the MCP client automatically.

## 📱 Chat Status States

The `chatStatus` property provides real-time feedback:
Expand Down Expand Up @@ -426,4 +459,4 @@ MIT License - see [LICENSE](LICENSE) file for details.

## 🚀 Contributing

Contributions welcome! Please read our contributing guidelines and submit pull requests to our [GitHub repository](https://github.com/SolveServeSolution/khaveeai-sdk).
Contributions welcome! Please read our contributing guidelines and submit pull requests to our [GitHub repository](https://github.com/SolveServeSolution/khaveeai-sdk).
1 change: 1 addition & 0 deletions packages/providers/openai-realtime/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
"dev": "tsc --watch"
},
"dependencies": {
"@modelcontextprotocol/sdk": "^1.25.1",
"@khaveeai/core": "^0.1.5",
"uuid": "^13.0.0"
},
Expand Down
60 changes: 53 additions & 7 deletions packages/providers/openai-realtime/src/OpenAIRealtimeProvider.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
} from "@khaveeai/core";
import { v4 as uuidv4 } from "uuid";
import { ToolExecutor } from "./ToolExecutor";
import { MCPToolManager } from "./mcp/MCPToolManager";

export class OpenAIRealtimeProvider implements RealtimeProvider {
private config: RealtimeConfig;
Expand All @@ -20,6 +21,10 @@ export class OpenAIRealtimeProvider implements RealtimeProvider {
private audioContext: AudioContext | null = null;
private audioStream: MediaStream | null = null;
private toolExecutor: ToolExecutor;
private manualTools: RealtimeTool[] = [];
private mcpTools: RealtimeTool[] = [];
private mcpManager?: MCPToolManager;
private mcpInitialization?: Promise<void>;

// State
public isConnected = false;
Expand Down Expand Up @@ -64,9 +69,20 @@ export class OpenAIRealtimeProvider implements RealtimeProvider {

this.toolExecutor = new ToolExecutor();

// Register initial tools
if (this.config.tools) {
this.config.tools.forEach((tool) => this.registerFunction(tool));
this.config.tools.forEach((tool) => this.registerTool(tool, "manual"));
}

if (this.config.mcpServers && this.config.mcpServers.length > 0) {
this.mcpManager = new MCPToolManager(this.config.mcpServers);
this.mcpInitialization = this.mcpManager
.initialize()
.then((tools) => {
tools.forEach((tool) => this.registerTool(tool, "mcp"));
})
.catch((error) => {
console.error("Failed to initialize MCP servers:", error);
});
}
Comment on lines +76 to 86
Copy link

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MCP initialization is started in the constructor but not awaited, creating a fire-and-forget promise. If initialization fails quickly (e.g., invalid URL format), the error is logged but there's no way for calling code to know initialization has failed until they try to use the tools. Consider exposing the initialization promise or adding a ready state flag so consumers can check if MCP is fully initialized.

Copilot uses AI. Check for mistakes.
}

Expand Down Expand Up @@ -98,7 +114,7 @@ export class OpenAIRealtimeProvider implements RealtimeProvider {
this.dataChannel = dataChannel;

dataChannel.onopen = () => {
this.configureSession();
void this.configureSession();
Copy link

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The void operator here suppresses the promise but doesn't handle potential errors from configureSession. If MCP initialization fails or configureSession encounters an error, it will be silently ignored. Consider handling errors from the configureSession promise or at minimum logging them.

Copilot uses AI. Check for mistakes.
};
dataChannel.onmessage = (event) => {
this.handleDataChannelMessage(event);
Expand Down Expand Up @@ -247,15 +263,43 @@ export class OpenAIRealtimeProvider implements RealtimeProvider {
* Register a function/tool
*/
registerFunction(tool: RealtimeTool): void {
this.registerTool(tool, "manual");
}
Comment on lines 265 to +267
Copy link

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a potential race condition if registerFunction is called externally after connect but before MCP tools are initialized. The configureSession method waits for MCP initialization, but if someone calls registerFunction and it triggers a reconfiguration, the new manual tool won't be sent to OpenAI until the next session update. Consider documenting that tools should be registered before calling connect, or ensure session updates when tools are registered after initial configuration.

Copilot uses AI. Check for mistakes.

private registerTool(tool: RealtimeTool, origin: "manual" | "mcp"): void {
this.toolExecutor.register(tool.name, tool.execute);
const target = origin === "manual" ? this.manualTools : this.mcpTools;
const existingIndex = target.findIndex((existing) => existing.name === tool.name);

if (existingIndex >= 0) {
target[existingIndex] = tool;
} else {
target.push(tool);
}
}

private getAllRegisteredTools(): RealtimeTool[] {
const combined = new Map<string, RealtimeTool>();
[...this.manualTools, ...this.mcpTools].forEach((tool) => {
combined.set(tool.name, tool);
});
Comment on lines +283 to +285
Copy link

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When merging manual and MCP tools in getAllRegisteredTools, if both sources provide a tool with the same name, the later one (MCP tools) silently overwrites the earlier one (manual tools) due to Map behavior. This could lead to unexpected behavior if users define a manual tool with the same name as an MCP tool. Consider warning about or preventing name collisions, or documenting the precedence order.

Suggested change
[...this.manualTools, ...this.mcpTools].forEach((tool) => {
combined.set(tool.name, tool);
});
// First, register all manual tools.
this.manualTools.forEach((tool) => {
combined.set(tool.name, tool);
});
// Then, register MCP tools, warning on name collisions and
// preserving the existing behavior where MCP tools take precedence.
this.mcpTools.forEach((tool) => {
if (combined.has(tool.name)) {
// Name collision: MCP tool overrides a manual tool with the same name.
console.warn(
`[OpenAIRealtimeProvider] Tool name collision detected for "${tool.name}". ` +
"The MCP tool will override the previously registered manual tool."
);
}
combined.set(tool.name, tool);
});

Copilot uses AI. Check for mistakes.
return Array.from(combined.values());
}

/**
* Configure the OpenAI session
*/
private configureSession(): void {
private async configureSession(): Promise<void> {
if (!this.dataChannel) return;

if (this.mcpInitialization) {
try {
await this.mcpInitialization;
} catch (error) {
console.error("Failed to initialize MCP tools:", error);
}
}

const sessionConfig: any = {
modalities: ["text", "audio"],
input_audio_transcription: {
Expand All @@ -275,14 +319,16 @@ export class OpenAIRealtimeProvider implements RealtimeProvider {
},
};

const tools = this.getAllRegisteredTools();

// Only add tools and tool_choice if tools are provided
if (this.config.tools && this.config.tools.length > 0) {
sessionConfig.tools = this.config.tools.map((tool) => {
if (tools.length > 0) {
sessionConfig.tools = tools.map((tool) => {
// Build parameters object, removing the 'required' field from each property
const properties: any = {};
const requiredFields: string[] = [];

Object.entries(tool.parameters).forEach(
Object.entries(tool.parameters || {}).forEach(
Copy link

Copilot AI Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The safeguard 'tool.parameters || {}' is good, but this change suggests that tool.parameters might be undefined. However, the RealtimeTool type definition requires parameters to always be present (not optional). If parameters can indeed be undefined in practice, the type definition should be updated to reflect this, otherwise this null check is unnecessary.

Suggested change
Object.entries(tool.parameters || {}).forEach(
Object.entries(tool.parameters).forEach(

Copilot uses AI. Check for mistakes.
([key, param]: [string, any]) => {
// Extract 'required' flag before adding to properties
const { required, ...paramSchema } = param;
Expand Down
Loading