Skip to content

Commit e185161

Browse files
rmcwhorterrmcwhorter
committed
websocket docs
1 parent 8f479bc commit e185161

File tree

1 file changed

+207
-0
lines changed

1 file changed

+207
-0
lines changed

‎fern/calls/websocket-transport.mdx‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
---
2+
title: WebSocket Transport
3+
description: Stream audio directly via WebSockets for bidirectional real-time communication
4+
slug: calls/websocket-transport
5+
---
6+
7+
# WebSocket Transport
8+
9+
Vapi's WebSocket transport provides a powerful way to establish direct, bidirectional audio communication between your application and Vapi's AI assistants. Unlike traditional phone or web calls, this transport mechanism allows you to send and receive raw audio data in real time with minimal latency.
10+
11+
## Overview
12+
13+
The WebSocket transport offers several advantages:
14+
15+
- **Low Latency**: Direct streaming reduces processing delays
16+
- **Bidirectional Communication**: Simultaneous audio streaming in both directions
17+
- **Flexible Integration**: Implement in any environment that supports WebSockets
18+
- **Customizable Audio Format**: Configure sample rate and format to match your needs
19+
- **Sample Rate Conversion**: Automatic handling of different audio sample rates
20+
21+
## Creating a WebSocket Call
22+
23+
To create a call using the WebSocket transport:
24+
25+
```bash
26+
curl 'https://api.vapi.ai/call'
27+
-H 'authorization: Bearer YOUR_API_KEY'
28+
-H 'content-type: application/json'
29+
--data-raw '{
30+
"assistant": {
31+
"assistantId": "YOUR_ASSISTANT_ID"
32+
},
33+
"transport": {
34+
"provider": "vapi.websocket",
35+
"audioFormat": {
36+
"format": "pcm_s16le",
37+
"container": "raw",
38+
"sampleRate": 16000
39+
}
40+
}
41+
}'
42+
```
43+
44+
### Sample Response
45+
46+
```json
47+
{
48+
"id": "7420f27a-30fd-4f49-a995-5549ae7cc00d",
49+
"assistantId": "5b0a4a08-133c-4146-9315-0984f8c6be80",
50+
"type": "vapi.websocketCall",
51+
"createdAt": "2024-09-10T11:14:12.339Z",
52+
"updatedAt": "2024-09-10T11:14:12.339Z",
53+
"orgId": "eb166faa-7145-46ef-8044-589b47ae3b56",
54+
"cost": 0,
55+
"status": "queued",
56+
"transport": {
57+
"provider": "vapi.websocket",
58+
"websocketCallUrl": "wss://api.vapi.ai/7420f27a-30fd-4f49-a995-5549ae7cc00d/transport"
59+
}
60+
}
61+
```
62+
63+
## Audio Format Configuration
64+
65+
When creating a WebSocket call, you can configure the audio format:
66+
67+
| Parameter | Description | Default |
68+
|-----------|-------------|---------|
69+
| `format` | Audio encoding format | `pcm_s16le` (16-bit PCM) |
70+
| `container` | Audio container format | `raw` (Raw PCM) |
71+
| `sampleRate` | Sample rate in Hz | `16000` (16kHz) |
72+
73+
<Important>
74+
Currently, only raw PCM audio data (`pcm_s16le` format with `raw` container) is supported. Additional audio formats and container types may be supported in future releases.
75+
</Important>
76+
77+
<Note>
78+
Vapi automatically handles sample rate conversion between your specified rate and the model's required format. This means you can send audio at 8kHz, 44.1kHz, or other rates, and Vapi will convert it appropriately.
79+
</Note>
80+
81+
## Connecting to the WebSocket
82+
83+
After creating a call, connect to the WebSocket URL returned in the response:
84+
85+
```javascript
86+
// Using the websocketCallUrl from the response
87+
const socket = new WebSocket("wss://api.vapi.ai/7420f27a-30fd-4f49-a995-5549ae7cc00d/transport");
88+
89+
socket.onopen = () => {
90+
console.log("WebSocket connection established");
91+
};
92+
93+
socket.onclose = () => {
94+
console.log("WebSocket connection closed");
95+
};
96+
97+
socket.onerror = (error) => {
98+
console.error("WebSocket error:", error);
99+
};
100+
```
101+
102+
## Sending and Receiving Data
103+
104+
The WebSocket connection supports two types of messages:
105+
106+
1. **Binary audio data**: Raw PCM audio frames (16-bit signed little-endian format)
107+
2. **Text messages**: JSON-formatted control messages
108+
109+
### Sending Audio Data
110+
111+
```javascript
112+
// Send binary audio data
113+
function sendAudioChunk(audioBuffer) {
114+
if (socket.readyState === WebSocket.OPEN) {
115+
socket.send(audioBuffer);
116+
}
117+
}
118+
119+
// Example: Send audio from a microphone stream
120+
navigator.mediaDevices.getUserMedia({ audio: true })
121+
.then(stream => {
122+
const audioContext = new AudioContext();
123+
const source = audioContext.createMediaStreamSource(stream);
124+
const processor = audioContext.createScriptProcessor(1024, 1, 1);
125+
126+
processor.onaudioprocess = (e) => {
127+
const pcmData = e.inputBuffer.getChannelData(0);
128+
// Convert Float32Array to Int16Array (for pcm_s16le format)
129+
const int16Data = new Int16Array(pcmData.length);
130+
for (let i = 0; i < pcmData.length; i++) {
131+
int16Data[i] = Math.max(-32768, Math.min(32767, pcmData[i] * 32768));
132+
}
133+
sendAudioChunk(int16Data.buffer);
134+
};
135+
136+
source.connect(processor);
137+
processor.connect(audioContext.destination);
138+
});
139+
```
140+
141+
### Receiving Messages
142+
143+
```javascript
144+
socket.onmessage = (event) => {
145+
if (event.data instanceof Blob) {
146+
// Handle binary audio data
147+
event.data.arrayBuffer().then(buffer => {
148+
const audioData = new Int16Array(buffer);
149+
// Process audio data (e.g., play it back or analyze it)
150+
playAudio(audioData);
151+
});
152+
} else {
153+
// Handle JSON control messages
154+
try {
155+
const message = JSON.parse(event.data);
156+
console.log("Received control message:", message);
157+
handleControlMessage(message);
158+
} catch (error) {
159+
console.error("Error parsing message:", error);
160+
}
161+
}
162+
};
163+
```
164+
165+
### Sending Control Messages
166+
167+
```javascript
168+
function sendControlMessage(messageObj) {
169+
if (socket.readyState === WebSocket.OPEN) {
170+
socket.send(JSON.stringify(messageObj));
171+
}
172+
}
173+
174+
// Example: Send a hangup message
175+
function hangupCall() {
176+
sendControlMessage({ type: "hangup" });
177+
}
178+
```
179+
180+
## Ending the Call
181+
182+
To properly end a WebSocket call:
183+
184+
```javascript
185+
// Send hangup message
186+
sendControlMessage({ type: "hangup" });
187+
188+
// Close the WebSocket connection
189+
socket.close();
190+
```
191+
192+
## Comparison with Call Listen Feature
193+
194+
Vapi offers two WebSocket-related features that serve different purposes:
195+
196+
| WebSocket Transport | Call Listen Feature |
197+
|---------------------|---------------------|
198+
| Primary communication channel for the call | Secondary monitoring channel for an existing call |
199+
| Bidirectional audio streaming | One-way audio streaming (receive only) |
200+
| Replaces phone/web as the transport method | Supplements an existing phone/web call |
201+
| Created with `transport.provider: "vapi.websocket"` | Accessed via the `monitor.listenUrl` in a standard call |
202+
203+
See [Live Call Control](/calls/call-features) for more information about the Call Listen feature.
204+
205+
<Warning>
206+
When using the WebSocket transport, you cannot simultaneously use phone number parameters (`phoneNumber` or `phoneNumberId`). The transport method is mutually exclusive with phone-based calling.
207+
</Warning>

0 commit comments

Comments
 (0)