Skip to content

Commit 661b798

Browse files
renandincerthird774
authored and
RebeccaTamachiro
committed
Calls: Introduce simulcast API (#20371)
* Calls: Introduce simulcast API * Update Calls simulcast API definition after feedback on 2025-02-27 * Calls: Make UpdateTracksRequest schema to be a array of TrackObject * Calls: Replace Simulcast API fallbackStrategy with priorityOrdering and ridUnavailableStrategy * Calls: Update the Calls Simulcast API after Nils' comments * Calls: Add Simulcast documentation text * Update src/content/docs/calls/simulcast.mdx Co-authored-by: Kevin Kipp <[email protected]> --------- Co-authored-by: Kevin Kipp <[email protected]>
1 parent 6c4722e commit 661b798

File tree

2 files changed

+253
-0
lines changed

2 files changed

+253
-0
lines changed

‎public/realtime/static/calls-api-2024-05-21.yaml‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -140,6 +140,17 @@ paths:
140140
sessionId: 2a45361d5fd7cc14eface0587c276c94
141141
trackName: generated-audio
142142
kind: "audio"
143+
remote_track_with_simulcast:
144+
description: Pull a remote track with simulcast preferences
145+
value:
146+
tracks:
147+
- location: remote
148+
sessionId: 2a45361d5fd7cc14eface0587c276c94
149+
trackName: simulcast-video-track
150+
simulcast:
151+
preferredRid: "h"
152+
priorityOrdering: "asciibetical"
153+
ridUnavailableStrategy: "nextPriority"
143154
security:
144155
- secret: []
145156
parameters:
@@ -207,6 +218,32 @@ paths:
207218
a=rtpmap:96 VP8/90000
208219
...
209220
type: offer
221+
remote_tracks_with_simulcast:
222+
value:
223+
requiresImmediateRenegotiation: true
224+
tracks:
225+
- sessionId: 2a45361d5fd7cc14eface0587c276c94
226+
trackName: simulcast-video-track
227+
mid: "5"
228+
simulcast:
229+
preferredRid: "h"
230+
priorityOrdering: "asciibetical"
231+
ridUnavailableStrategy: "nextPriority"
232+
sessionDescription:
233+
sdp: |
234+
v=0
235+
o=- 0 0 IN IP4 127.0.0.1
236+
s=-
237+
c=IN IP4 127.0.0.1
238+
t=0 0
239+
m=video 4002 RTP/AVP 96
240+
a=rtpmap:96 VP8/90000
241+
a=simulcast:recv f;h;q
242+
a=rid:f recv
243+
a=rid:h recv
244+
a=rid:q recv
245+
...
246+
type: offer
210247
/apps/{appId}/sessions/{sessionId}/renegotiate:
211248
put:
212249
tags:
@@ -329,6 +366,73 @@ paths:
329366
requiresImmediateRenegotiation: false
330367
tracks:
331368
- mid: "7"
369+
/apps/{appId}/sessions/{sessionId}/tracks/update:
370+
put:
371+
tags:
372+
- Change tracks
373+
summary: Change tracks by reusing existing transceivers
374+
requestBody:
375+
content:
376+
application/json:
377+
schema:
378+
$ref: "#/components/schemas/UpdateTracksRequest"
379+
examples:
380+
reuse_transceiver:
381+
description: Reuse an existing transceiver for a new track
382+
value:
383+
tracks:
384+
- location: "remote"
385+
sessionId: "2a45361d5fd7cc14eface0587c276c94"
386+
trackName: "other-track-name"
387+
mid: "7"
388+
reuse_with_simulcast:
389+
description: Reuse an existing transceiver with new simulcast preferences
390+
value:
391+
tracks:
392+
- location: "remote"
393+
sessionId: "2a45361d5fd7cc14eface0587c276c94"
394+
trackName: "simulcast-track"
395+
mid: "8"
396+
simulcast:
397+
preferredRid: "h"
398+
priorityOrdering: "asciibetical"
399+
ridUnavailableStrategy: "nextPriority"
400+
security:
401+
- secret: []
402+
parameters:
403+
- in: path
404+
name: appId
405+
schema:
406+
type: string
407+
required: true
408+
description: WebRTC application ID
409+
- in: path
410+
name: sessionId
411+
schema:
412+
type: string
413+
required: true
414+
description: Current PeerConnection session ID
415+
responses:
416+
"200":
417+
description: OK
418+
headers:
419+
vary:
420+
schema:
421+
type: string
422+
example: Origin
423+
content:
424+
application/json:
425+
schema:
426+
$ref: "#/components/schemas/UpdateTracksResponse"
427+
examples:
428+
success:
429+
value:
430+
requiresImmediateRenegotiation: false
431+
tracks:
432+
- mid: "7"
433+
sessionId: "2a45361d5fd7cc14eface0587c276c94"
434+
trackName: "new-track-name"
435+
332436
/apps/{appId}/sessions/{sessionId}:
333437
get:
334438
tags:
@@ -412,6 +516,27 @@ components:
412516
kind:
413517
type: string
414518
description: Give a hint to the SFU about the transceiver kind. This is required when the SFU generates the offer
519+
simulcast:
520+
type: object
521+
description: Simulcast configuration for the track
522+
properties:
523+
preferredRid:
524+
type: string
525+
description: Preferred RID for simulcast streams
526+
priorityOrdering:
527+
type: string
528+
enum:
529+
- none
530+
- asciibetical
531+
default: none
532+
description: Controls what happens if there is not enough network resources available to send the preferredRid. 'none' means keep sending even if not enough bandwidth, 'asciibetical' uses a-z order to determine priority where a is most desirable and z is least desirable.
533+
ridUnavailableStrategy:
534+
type: string
535+
enum:
536+
- none
537+
- nextPriority
538+
default: none
539+
description: Controls what happens when the rid currently being used or preferredRid is no longer being sent by the publisher. 'none' means do nothing, 'nextPriority' uses the next on the priorityOrdering.
415540
CloseTrackObject:
416541
type: object
417542
properties:
@@ -543,3 +668,44 @@ components:
543668
type: string
544669
sessionDescription:
545670
$ref: "#/components/schemas/SessionDescription"
671+
ChangeTracksRequest:
672+
type: object
673+
properties:
674+
tracks:
675+
type: object
676+
additionalProperties:
677+
allOf:
678+
- $ref: "#/components/schemas/TrackObject"
679+
description: Map of track IDs to track objects for changing tracks
680+
sessionDescription:
681+
$ref: "#/components/schemas/SessionDescription"
682+
UpdateTracksRequest:
683+
type: object
684+
properties:
685+
tracks:
686+
type: array
687+
items:
688+
$ref: "#/components/schemas/TrackObject"
689+
description: Array of track objects for updating tracks
690+
sessionDescription:
691+
$ref: "#/components/schemas/SessionDescription"
692+
UpdateTracksResponse:
693+
type: object
694+
properties:
695+
errorCode:
696+
type: string
697+
errorDescription:
698+
type: string
699+
requiresImmediateRenegotiation:
700+
type: boolean
701+
tracks:
702+
type: array
703+
items:
704+
allOf:
705+
- $ref: "#/components/schemas/TrackObject"
706+
- properties:
707+
errorCode:
708+
type: string
709+
errorDescription:
710+
type: string
711+
description: Array of track objects with results

‎src/content/docs/realtime/simulcast.mdx‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
---
2+
pcx_content_type: get-started
3+
title: Simulcast
4+
sidebar:
5+
order: 8
6+
---
7+
8+
Simulcast is a feature of WebRTC that allows a publisher to send multiple streams of the same media at different qualities. For example, this is useful for scenarios where you want to send a high quality stream for desktop users and a lower quality stream for mobile users.
9+
10+
```mermaid
11+
graph LR
12+
A[Publisher] -->|Low quality| B[Cloudflare Calls SFU]
13+
A -->|Medium quality| B
14+
A -->|High quality| B
15+
B -->|Low quality| C@{ shape: procs, label: "Subscribers"}
16+
B -->|Medium quality| D@{ shape: procs, label: "Subscribers"}
17+
B -->|High quality| E@{ shape: procs, label: "Subscribers"}
18+
19+
```
20+
21+
### How it works
22+
23+
Simulcast in WebRTC allows a single media source, like a camera or screen share, to be encoded at multiple quality levels and sent simultaneously, which is beneficial for subscribers with varying network conditions and device capabilities. The media source is encoded into multiple streams, each identified by RIDs (RTP Stream Identifiers) for different quality levels, such as low, medium, and high. These simulcast streams are described in the SDP you send to Cloudflare Calls SFU. It's the responsibility of the Cloudflare Calls SFU to ensure that the appropriate quality stream is delivered to each subscriber based on their network conditions and device capabilities.
24+
25+
Cloudflare Calls SFU will automatically handle the simulcast configuration based on the SDP you send to it from the publisher. The SFU will then automatically switch between the different quality levels based on the subscriber's network conditions. You can control the quality switching behavior using the `simulcast` configuration object when you send an API call to start pulling a remote track.
26+
27+
### Quality Control
28+
29+
The `simulcast` configuration object in the API call when you start pulling a remote track allows you to specify:
30+
31+
- `preferredRid`: The preferred stream (RID for the simulcast stream. [RIDs can be specified by the publisher.](https://developer.mozilla.org/en-US/docs/Web/API/RTCRtpSender/setParameters#encodings))
32+
- `priorityOrdering`: Controls how the SFU handles bandwidth constraints
33+
- `none`: Keep sending the preferred layer even if there's not enough bandwidth
34+
- `asciibetical`: Use alphabetical ordering (a-z) to determine priority, where 'a' is most desirable and 'z' is least desirable
35+
- `ridUnavailableStrategy`: Controls what happens when the preferred RID is no longer available, for example when the publisher stops sending it
36+
37+
- `none`: Do nothing
38+
- `nextPriority`: Switch to the next available RID based on the priority ordering
39+
40+
You will likely want to order the asciibetical RIDs based on your desired metric, such as higest resoltion to lowest or highest bandwidth to lowest.
41+
42+
### Bandwidth Management across media tracks
43+
44+
Cloudflare Calls treats all media tracks equally at the transport level. For example, if you have multiple video tracks (cameras, screen shares, etc.), they all have equal priority for bandwidth allocation. This means:
45+
46+
1. Each track's simulcast configuration is handled independently
47+
1. The SFU performs automatic bandwidth estimation and layer switching based on network conditions independently for each track
48+
49+
### Layer Switching Behavior
50+
51+
When a layer switch is requested (through updating `preferredRid`) with the `/tracks/update` API:
52+
53+
1. The SFU will automatically generate a Picture Loss Indication (PLI)
54+
2. Layer switching only occurs when a keyframe arrives on the target layer
55+
3. PLI generation is debounced to prevent excessive requests
56+
57+
### Publisher Configuration
58+
59+
For publishers (local tracks), you only need to include the simulcast attributes in your SDP. The SFU will automatically handle the simulcast configuration based on the SDP. For example, the SDP should contain a section like this:
60+
61+
```sdp
62+
a=simulcast:send f;h;q
63+
a=rid:f send
64+
a=rid:h send
65+
a=rid:q send
66+
```
67+
68+
You can include these by specifying `sendEncodings` when creating the transceiver:
69+
70+
```js
71+
const transceiver = peerConnection.addTransceiver(track, {
72+
direction: "sendonly",
73+
sendEncodings: [
74+
{ scaleResolutionDownBy: 1, rid: "a" },
75+
{ scaleResolutionDownBy: 2, rid: "b" },
76+
{ scaleResolutionDownBy: 4, rid: "c" }
77+
]
78+
});
79+
## Example
80+
81+
Here's an example of how to use simulcast with Cloudflare Calls:
82+
83+
1. Create a new local track with simulcast configuration. There should be a section in the SDP with `a=simulcast:send`.
84+
2. Use the [Cloudflare Calls API](/calls/https-api) to push this local track, by calling the /tracks/new endpoint.
85+
3. Use the [Cloudflare Calls API](/calls/https-api) to start pulling a remote track (from another browser or device), by calling the /tracks/new endpoint and specifying the `simulcast` configuration object along with the remote track ID you get from step 2.
86+
87+
For more examples, check out the [Calls Examples GitHub repository](https://github.com/cloudflare/calls-examples/tree/main/simulcast).

0 commit comments

Comments
 (0)