Processing pipeline: Guidelines on handling backpressure

[tidoust]

The backpressure mechanism in WHATWG Streams takes some getting used to, but appears simple and powerful after a while. It remains difficult to reason about backpressure in video processing pipelines because, by definition, this backpressure mechanism stops whenever something else than WHATWG Streams are used:

• WebRTC uses MediaStreamTrack by default.
• The VideoEncoder and VideoDecoder classes in WebCodecs have their own queueing mechanism.
• VideoTrackGenerator and MediaStreamTrackProcessor create a bridge between WebRTC and WebCodecs, with specific queueing rules.

There are good reasons that explain the divergence of approaches regarding streams handling across technologies. For example, see Decoupling WebCodecs from Streams. From a developer perspective, this makes mixing technologies harder. It also creates more than one way to build the same pipeline with no obvious right approach to queueing and backpressure. In processing pipelines, when should developers rely on the backpressure mechanism provided by Streams? When should developers rather rely on internal queueing mechanisms?

Would it be worthwhile to create guidelines on the ins and outs of the different approaches, as a way to inform developers and perhaps further evolutions of underlying web technologies?

Related sample code:

• Sample code that uses internal VideoEncoder / VideoDecoder queueing: Add WebCodecs in Worker sample webcodecs#583
• Sample code that relies on stream's backpressure mechanism: Experimenting with video processing pipelines on the web

[aboba]

The backpressure issue becomes critical when transporting media over the network. If the media is being generated at a bitrate that exceeds the bottleneck bandwidth, queues will build. There are a few ways to adjust:

Short term

• Retransmission timeouts: When using partial reliability or datagram transport, a maximum retransmission timer can be set, after which the datagram is not retransmitted, or the stream (for frame/stream transport) is aborted. This technique can work well in concert with Scalable Video Coding (SVC), where "non-discardable" frames (e.g. base layer frames) have larger retransmission timers than "discardable" frames).

• Layer drop. After several retransmission timeouts, an SVC layer can be dropped. Currently WebCodecs only supports temporal scalability, so this would imply halving the framerate. The WebCodecs encoder configuration need not necessarily be changed, the dropped layer could still be generated, just not sent. It is trickier to figur out when the dropped layer can be re-enabled since in an "application limited" regime, the true bottleneck bandwidth may not be easily estimated. Also, it is optimal to only re-start sending an upper layer at an "upswitch point" so the upper layer frames can be decoded.

• Per-frame QP. This is not yet supported in WebCodecs, but the proposed API enables per-frame adjustments.

Longer term

• Resolution, bitrate, framerate adjustment. In WebCodecs, the aspects of the encoder configuration can be changed without requiring generation of a keyframe.