Calls: Introduce simulcast API

[renandincer]

This is a first stab at introducing simulcast into the Cloudflare Calls SFU API.

Examples

Example 1: Using the /tracks/change endpoint to reuse a transceiver

This example shows how to reuse an existing transceiver (with mid "2") to receive a different track from a remote session.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/change" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tracks": {
      "video-track-1": {
        "location": "remote",
        "sessionId": "2a45361d5fd7cc14eface0587c276c94",
        "trackName": "new-video-track",
        "mid": "2"
      }
    }
  }'

Example 2: Using the /tracks/change endpoint with simulcast preferences

This example demonstrates how to reuse a transceiver while specifying simulcast preferences, requesting the high-quality stream ("h") and falling back to the least bandwidth option if that's not available.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/change" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tracks": {
      "simulcast-video": {
        "location": "remote",
        "sessionId": "2a45361d5fd7cc14eface0587c276c94",
        "trackName": "simulcast-camera",
        "mid": "3",
        "simulcast": {
          "preferredRid": "h",
          "preferredRidNotAvailable": "leastbandwidth"
        }
      }
    }
  }'

Example 3: Changing multiple tracks simultaneously

This example shows how to change multiple tracks in a single request, reusing different transceivers for different remote tracks.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/change" \
 -H "Authorization: Bearer YOUR_API_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
   "tracks": {
     "video-track": {
       "location": "remote",
       "sessionId": "2a45361d5fd7cc14eface0587c276c94",
       "trackName": "camera-feed",
       "mid": "2"
     },
     "audio-track": {
       "location": "remote",
       "sessionId": "2a45361d5fd7cc14eface0587c276c94",
       "trackName": "microphone",
       "mid": "0"
     }
   }
 }'

Example 4: Using the /tracks/new endpoint with simulcast

This example demonstrates adding a new local track with simulcast capabilities, specifying the SDP with simulcast attributes and indicating a preference for the high-quality stream.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/new" \
-H "Authorization: Bearer YOUR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
  "sessionDescription": {
    "sdp": "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=-\r\nc=IN IP4 127.0.0.1\r\nt=0 0\r\nm=video 4002 RTP/AVP 96\r\na=rtpmap:96 VP8/90000\r\na=simulcast:send f;h;q\r\na=rid:f send\r\na=rid:h send\r\na=rid:q send\r\n",
    "type": "offer"
  },
  "tracks": [
    {
      "location": "local",
      "trackName": "simulcast-camera-feed",
      "mid": "0",
      "simulcast": {
        "preferredRid": "h",
        "preferredRidNotAvailable": "leastbandwidth"
      }
    }
  ]
}'

Example 5: Using the /tracks/change endpoint with a different fallback strategy

This example shows how to request a specific quality level (in this case, the lowest quality "q") with a fallback strategy of "none", which means no fallback will be attempted if the preferred RID is not available.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/change" \
 -H "Authorization: Bearer YOUR_API_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
   "tracks": {
     "simulcast-video": {
       "location": "remote",
       "sessionId": "2a45361d5fd7cc14eface0587c276c94",
       "trackName": "simulcast-camera",
       "mid": "3",
       "simulcast": {
         "preferredRid": "q",
         "preferredRidNotAvailable": "none"
       }
     }
   }
 }'

Open Questions

The following case is not handled: "I REALLY want the RID I just told you, but if the RID is no longer being published, then downgrade. However I'm okay with packet loss with a higher rid I can't handle", because we are using preferredRidNotAvailable for both "RID stopped published" and also "you don't have enough bandwidth for this RID"

[github-actions]

Preview URL: https://2b5ea9a9.preview.developers.cloudflare.com
Preview Branch URL: https://introduce-simulcast-api.preview.developers.cloudflare.com

Files with changes (up to 15)

Original Link	Updated Link
https://developers.cloudflare.com/realtime/simulcast/	https://introduce-simulcast-api.preview.developers.cloudflare.com/realtime/simulcast/

[nils-ohlmeier]

Some initial quick points:

Example 4 is not what we had been discussing. When you push a track the tracks object should have no simulcast in it at all. Conceptually I don't even know what a preferredRid from the publishers point of view would mean.
But this does raise an interesting question: should we respond with an error if someone makes an API call like that, or just ignore the simulcast part?

Should we use /track/change or /track/update?
I don't have a strong preference either way, just wanted to mention here in case someone cares about it.
But I would prefer if we use PUT for the /track/change (or /update) endpoint.

Regarding the open question "I REALLY want the RID I just told you, but if the RID is no longer being published, then downgrade. However I'm okay with packet loss with a higher rid I can't handle":

• the first part of your quote makes sense and the current proposal allows the client to express this (even more detailed by saying how to downgrade).
• The second sentence in quote doesn't really compute for me. Are you trying to say "Please upgrade me, rather downgrade"? If so I don't think we currently have a way to request an upgrade. The part which doesn't make sense to me is specifically the packet loss part. Packet loss on video tracks will result in retransmissions, which overall ends up using more bandwidth. Where you trying to say "However I'm okay with using more bandwidth with a higher RID"?. The important part is that if we talk about bandwidth: if the higher bandwidth RID doesn't fit through the downlink it really doesn't make sense to try to push it through. It will only result in frozen/stuttering video. I'll stop rambling now and let you clarify what you mean. :-)

[renandincer]

Example 4 is not what we had been discussing. When you push a track the tracks object should have no simulcast in it at all. Conceptually I don't even know what a preferredRid from the publishers point of view would mean.

This is an oversight from my side. What I mean was new remote track with simulcast capabilities, not local. When you pull a brand new track, there should be an option to specify simulcast options though. We can't just depend on the change endpoint to handle simulcast.

But this does raise an interesting question: should we respond with an error if someone makes an API call like that, or just ignore the simulcast part?

I'm leaning towards erroring because it's a blatant issue with the API definition, push with simulcast doesn't make sense.

Should we use /track/change or /track/update?
I don't have a strong preference either way, just wanted to mention here in case someone cares about it.
But I would prefer if we use PUT for the /track/change (or /update) endpoint.

I would also prefer PUT at /track or /tracks as well

The second sentence in quote doesn't really compute for me.

What if you wanted to ignore all bandwidth estimation changes but keep the changes when a rid disappears entirely?

[nils-ohlmeier]

Example 4 is not what we had been discussing. When you push a track the tracks object should have no simulcast in it at all. Conceptually I don't even know what a preferredRid from the publishers point of view would mean.

This is an oversight from my side. What I mean was new remote track with simulcast capabilities, not local. When you pull a brand new track, there should be an option to specify simulcast options though. We can't just depend on the change endpoint to handle simulcast.

Yes when you pull a new track the /track/new endpoint will support the simulcast object for sure.

But this does raise an interesting question: should we respond with an error if someone makes an API call like that, or just ignore the simulcast part?

I'm leaning towards erroring because it's a blatant issue with the API definition, push with simulcast doesn't make sense.

Okay.

Should we use /track/change or /track/update?
I don't have a strong preference either way, just wanted to mention here in case someone cares about it.
But I would prefer if we use PUT for the /track/change (or /update) endpoint.

I would also prefer PUT at /track or /tracks as well

Great. We already have PUT for /track/close, which makes sense to me because again this attempts to modify an existing resource.

The second sentence in quote doesn't really compute for me.

What if you wanted to ignore all bandwidth estimation changes but keep the changes when a rid disappears entirely?

Ahh. Yeah I think that scenario highlights your concern if we need two different parameters:

• one for not available (any more)
• one for how to respond to issues with the downlink of the pulling client
  Which probably makes this easier to comprehend, rather than trying to squeeze these two dimensions into on parameter.

[astroza]

Do you agree we have a consensus about the HTTP method and endpoint?

curl -X PUT "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/update"

[astroza]

Regarding the request body I lean towards having schema/TracksRequest for it OR a subset of it that does not include the sdp and autoDiscover fields.

TracksRequest/subset should be updated to have the simulcast field

The subset could be look like:

    TracksUpdate:
      type: object
      properties:
        tracks:
          type: array
          items:
            $ref: "#/components/schemas/TrackObject"

and TrackObject should have the recently added simulcast field

[astroza]

In relation to the simulcast field:

• I think there is no doubt about keeping preferredRid ✅
• preferredRidNotAvailable
  • The name may not suggest what it does
  • It should not be exclusive for preferredRid: Any other option should default to this field
    Example: {preferredRid: "h" }, { minWidth: 1024 }, { maxBandwidth: 1000000 } (the last 2 are potential fields, we are not defining them here)
    If they can't be meet we should use a backup criteria, so what could be that field name?
    • { fallback: "leastbandwidth" }
    • { backupCriteria: "auto" } // auto would send the best quality as possible
    • something else

[renandincer]

@astroza

and TrackObject should have the recently added simulcast field

This is already reflected in https://github.com/cloudflare/cloudflare-docs/pull/20371/files#diff-dfab4b0d8e0e3e2d7230fcf1c1ac833de68f48d0512d9b956781a594c5b5549bR506. Were you thinking of something else?

Updating the PR now to make the corrections we discussed.

[renandincer]

{ backupCriteria: "auto" } // auto would send the best quality as possible

Shouldn't this be called bestQualityForAvailableBandwidth then?

Because imagine a scenario where there are many tiny users on screen. The client can set "send me simulcast that is maxWidth 180x180. Then if this criteria fails (say there isn't a layer that is this small), what would we do? Is this API expressive enough for this kind of case?

[nils-ohlmeier]

{ backupCriteria: "auto" } // auto would send the best quality as possible

Shouldn't this be called bestQualityForAvailableBandwidth then?

Because imagine a scenario where there are many tiny users on screen. The client can set "send me simulcast that is maxWidth 180x180. Then if this criteria fails (say there isn't a layer that is this small), what would we do? Is this API expressive enough for this kind of case?

Several points on this:

• I'm not sure if we should already here discuss things like resolution constraints.
• If we add things like resolution constraints, we should probably not use something like maxWidth because that would require a hard failure if we can't match that. I would propose something like idealResolution instead, which would give us a little bit more wiggle room, e.g. the stream is 5 pixel bigger than requested. And yes even in that case you still need a threshold.
• In the end there is always a scenario where the SFU could decide to stop sending a video stream (except if explictly asked to not interfere). But I think for that scenario we should have some kind of event interface so that the client is aware of these decisions.

[nils-ohlmeier]

I think overall it looks good. Quite a few details still to figure out.

[nils-ohlmeier]

I was not planing on echoing back in the response the simulcast parameters.
What is the use case or advantage of doing that?

[renandincer]

no advantage. can remove 👍

[renandincer]

but maybe echo shows it is applied? example we'd not echo if not a simulcast track?

[nils-ohlmeier]

Does the new-track-name only apply to the use case of replacing a track?
For the simulcast scenario the track stays the same. So I assumed we would not return a trackName in that case.

[renandincer]

I think we should try to return what we have here. I don't see why returning is a bad thing?

[third774]

I think "new-track-name" is confusing. It's fine to keep trackName though IMO.

Suggested change

	trackName: "new-track-name"
	trackName: "track-name"

[nils-ohlmeier]

• It should not be exclusive for preferredRid: Any other option should default to this field
  Example: {preferredRid: "h" }, { minWidth: 1024 }, { maxBandwidth: 1000000 } (the last 2 are potential fields, we are not defining them here)
  If they can't be meet we should use a backup criteria, so what could be that field name?

  • { fallback: "leastbandwidth" }
  • { backupCriteria: "auto" } // auto would send the best quality as possible
  • something else

I think the new name fallbackStrategy is a lot better compared to preferredRidNotAvailable.

But I think the field fallbackStrategy inside the simulcast object should only address the scenario which is specific to simulcast: that the publisher can decided to stop sending a RID at any given time. This behavior is unique to simulcast.

What is not unique to simulcast are things like idealResolution or maxBandwidth. These constraints will also apply to SVC tracks in the future. And they all apply to the downlink from the SFU to the pulling endpoint.

Thus I would propose that the simulcast object should contain a fallbackStrategy parameter, which only describes what the SFU should do in case the requested RID is not available or becomes unavailable.

At a later time we should define a strategy parameter outside of the simulcast object, probably on the same level as simulcast, which describes strategies and constraints for the downlink part.
So the only action we would need to take on this PR here is to change the description of what fallbackStrategy inside the simulcast object means.

[renandincer]

I want to surface additional points made:

1. It's unclear if preferredRid just a soft preference, and the SFU will automatically switch down when bandwidth can't support the layer or if it is this a hard preference, where no matter what the packet loss/delay on the receiver is Calls SFU will continue to forward this layer if the publisher is forwarding it.

2. Relation of different video tracks in bandwidth selection is unclear.

3. It’s unclear if the Calls SFU performs probing for a higher layer switch periodically, for a better bandwidth estimation prediction or if the expectation that the user of the calls API performs probing on their side, by setting preferredRid (This seems a bit cumbersome, for the client to do)

4. On the publisher side (new track API with "local" track), it’s unclear if the customer needs to specify anything different on the /new tracks API regarding simulcast, or if the SDP signalling a=simulcast enough? The API spec re-uses the same new track object on that side as well.

5. It’s unclear how the determination of "best" and "worst" layer is done. ONLY based on the "f","h", and "q" RID convention? Or is this something more general, where the SFU looks at the a=simulcast:recv f;h;q  line and does the ordering based on that? Or is it even more general, like alphabetical ordering? This should be a configuration variable.

6. Does the Calls SFU implicitly generate a PLI, when a layer switch is requested (through updating preferredRid), and only switch layers when a keyframe arrives on the target layer? If yes, does the PLI debouncing/ or any other rule to limit the rate of PLIs here also apply?

To solve issue (1, 5):

Underneath the simulcast option, have two new fields next to the preferredRid field.

1. To control "what happens if there is no enough network resources available to send the preferredRid. This would be called priorityOrdering, with options none (do nothing - keep sending even if not enough bandwidth) and asciibetical(use as order a-z to determine priority where a is most desirable and z is least desirable)
2. To control "what happens when the rid currently being used or prefrerredRid is no longer being sent by the publisher". This would be called ridUnavailableStrategy, with options none (do nothing) and nextPriority(use the next on the priorityOrdering.

To solve issue (2, 3):

We should document that Cloudflare assumes equal priority for all video tracks on the transport level. For example, if there is X video tracks and Y screen shares, these should be treated equally to start. Later, we can add additional options to prioritize certain tracks over others.

We should document how often and exact behavior

Remote and local track definitions are diverging with the addition of simulcast field.

To solve issue (4):

Consider separating push track and pull track objects.

To solve issue (6):

Document PLI behavior when a simulcast layer switch is requested.

[renandincer]

Updated the API definition after the conversations above.

[hyperlint-ai]

Howdy and thanks for contributing to our repo. The Cloudflare team reviews new, external PRs within two (2) weeks. If it's been two weeks or longer without any movement, please tag the PR Assignees in a comment.

We review internal PRs within 1 week. If it's something urgent or has been sitting without a comment, start a thread in the Developer Docs space internally.

---

PR Change Summary

Introduced simulcast functionality in the Cloudflare Calls SFU API documentation, providing detailed explanations and examples for users.

• Added a new section on simulcast, explaining its purpose and functionality.
• Included multiple examples demonstrating how to use the simulcast API effectively.
• Outlined quality control and bandwidth management strategies for simulcast streams.

Added Files

• src/content/docs/calls/simulcast.mdx

---

How can I customize these reviews?

Check out the Hyperlint AI Reviewer docs for more information on how to customize the review.

If you just want to ignore it on this PR, you can add the hyperlint-ignore label to the PR. Future changes won't trigger a Hyperlint review.

Note specifically for link checks, we only check the first 30 links in a file and we cache the results for several hours (for instance, if you just added a page, you might experience this). Our recommendation is to add hyperlint-ignore to the PR to ignore the link check for this PR.