Expose in VideoFrameMetadata some fields from VideoFrameCallbackMetadata

[youennf]

It seems useful to expose some fields in VideoFrameMetadata like:

• https://wicg.github.io/video-rvfc/#dom-videoframecallbackmetadata-capturetime
• https://wicg.github.io/video-rvfc/#dom-videoframecallbackmetadata-receivetime
• https://wicg.github.io/video-rvfc/#dom-videoframecallbackmetadata-rtptimestamp

This could be added as an entry in VideoFrameMetadata registry.

[dalecurtis]

These sound fine to me, but @aboba points out that https://wicg.github.io/video-rvfc/ may not meet the registry requirements yet.

I know at TPAC there was some talk of moving rVFC into the HTML spec. Do we still want to do that?

@chrisn @tguilbert-google

[youennf]

I think the idea would be to write a registry entry that would define these values without referencing rvfc. Instead, it would reference webrtc-pc and/or media capture-main.

[tguilbert-google]

Yes, I was still planning on moving rVFC to the HTML spec. I haven't found the time to do so yet. Embedding the spec into the HTML spec is more work than just referencing it.

Previous (closed) PR tracking this work:
whatwg/html#5332 (comment)

[chrisn]

I agree, adding this into HTML makes sense.

[aboba]

The RVFC metadata items seem quite useful to me, but they also raise some questions for the behavior of other APIs:

1. In the VideoFrame(s) produced by MediaStreamTrackProcessor, should we expect capturetime metadata to be present?
2. Is there an expectation that the WebCodecs encoder will carry Videoframe metadata items through to encodedChunk metadata? (e.g. capturetime)
3. Is there an expectation that the WebCodecs decoder will carry encodedChunk metadata items through to VideoFrame metadata? (e.g. receivetime)

Without an explicit change to the WebCodecs API, I'm assuming that the answer to questions 2 and 3 is "no". But what I'd really like to be able to do is to trace performance through each stage of the receive and send pipeline.

[youennf]

1. In the VideoFrame(s) produced by MediaStreamTrackProcessor, should we expect capturetime metadata to be present?

It depends on MediaStreamTrack's source. If source is getUserMedia, yes.

2. Is there an expectation that the WebCodecs encoder will carry Videoframe metadata items through to encodedChunk metadata? (e.g. capturetime)

EncodedVideoChunk has no metadata so I would think that it would not be carried automatically.
webrtc encoded transform could decide to expose it in RTCEncodedVideoFrameMetadata but this is out of scope of web codecs.

3. Is there an expectation that the WebCodecs decoder will carry encodedChunk metadata items through to VideoFrame metadata? (e.g. receivetime)

My understanding is that the decoder will not have that data so will not set it.
The application wrapping the decoder will have this data (say the WebRTC pipeline) and can cheaply attach it to the video frame.

[aboba]

We now have a proposal to add some timing information to the RTCEncodedVideoFrameMetadata: w3c/webrtc-encoded-transform#173

Would it make sense to define a metadata registry for encodedChunk and add similar info there?

[guidou]

Is it possible to make progress with this?
Recently, we have received requests from developers asking for this, in particular capture time, to be exposed via MediaStreamTrackProcessor.

[Djuffin]

Why can't we use regular VideoFrame.timestamp for capture time when frames are coming from a capture device.

[tguilbert-google]

@guidou reached out to me last Friday about surfacing capture timestamps, proposing VideoFrame.timestamp. My apologies, I didn't follow up after discussing with @Djuffin, and how VideoFrame.timestamp was a valid path forward.

The MediaStreamTrackProcessor spec could define how the timestamp is populated, and might not need a WebCodecs spec change?