Processing Pipeline: Timing Model

[aboba]

Dom & Francois have demonstrated a way of tracking timing at each stage of the processing pipeline, by inserting a timestamp overlay.

It would be nice to have a consistent timing model built in so as to make this more straightforward.

The requestVideoFrameCallback specification defines VideoFrameCallbackMetadata, which includes some timing information.

It has been proposed that some fields from VideoFrameCallbackMetadata be added to VideoFrameMetadata:
w3c/webcodecs#601

However, currently timing metadata is not consistently exposed across media processing APIs. For example:

• VideoFrame.captureTime is not exposed within the mediacapture-transform API (e.g. in a VideoFrame produced from a MediaStreamTrack)
• When aVideoFrame is passed to WebCodecs Encoder, the timing metadata is not copied to the encoded Chunk

Related slides: https://docs.google.com/presentation/d/1DEk4urMPNufIYX88jzRqXddxoGRZXv10iOTGbKvLe5s/edit#slide=id.g159ea124a3f_8_221