Skip to content

Capture, receive, and RTP timestamp concept definitions & normative requirements for gUM/gDM #156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Oct 31, 2024
Merged

Capture, receive, and RTP timestamp concept definitions & normative requirements for gUM/gDM #156

Changes from 3 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
6f40605
Define captureTime.
handellm Sep 9, 2024
ae946a9
Goldplate
handellm Sep 11, 2024
2dc8783
Address comments from internal review.
handellm Sep 11, 2024
41da924
Updates after offline review
handellm Sep 20, 2024
069793f
Updates after TPAC joint Media/WebRTC session.
handellm Sep 27, 2024
88a2e13
Clarify nature and utility of timestamps.
handellm Sep 27, 2024
15e2399
Address reviewer comments.
handellm Sep 27, 2024
958a9bd
Update index.html
handellm Oct 7, 2024
c79520a
Update index.html
handellm Oct 7, 2024
4d9e139
Update index.html
handellm Oct 7, 2024
584ace5
Update index.html
handellm Oct 7, 2024
8d59ae2
Update index.html
handellm Oct 7, 2024
f5d3490
Update index.html
handellm Oct 7, 2024
d915bd5
Adress reviewer comments.
handellm Oct 7, 2024
a8fc811
Address reviewer comments.
handellm Oct 8, 2024
b5778b0
Clarify nature of presentation time, introduce offset in webcodecs vi…
handellm Oct 9, 2024
012628d
fix
handellm Oct 9, 2024
794cd2d
Update index.html
handellm Oct 10, 2024
fcafe1c
Add a note about presentation/capture timestamps being potentially th…
handellm Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 39 additions & 1 deletion index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
// See https://github.com/w3c/respec/wiki/ for how to configure ReSpec
var respecConfig = {
group: "webrtc",
xref: ["geometry-1", "html", "infra", "permissions", "dom", "image-capture", "mediacapture-streams", "webaudio", "webcodecs", "webidl"],
xref: ["geometry-1", "html", "infra", "permissions", "dom", "hr-time", "image-capture", "mediacapture-fromelement", "mediacapture-streams", "screen-capture", "webaudio", "webcodecs", "webidl"],
edDraftURI: "https://w3c.github.io/mediacapture-extensions/",
editors: [
{name: "Jan-Ivar Bruaroey", company: "Mozilla Corporation", w3cid: 79152},
Expand Down Expand Up @@ -58,6 +58,10 @@ <h2>Terminology</h2>
<p>The terms [=permission state=], [=request permission to use=], and
<a data-cite="permissions">prompt the user to choose</a> are defined in
[[!permissions]].</p>
<p>
{{Performance.timeOrigin}} and {{Performance.now()}} are defined in
[[!hr-time]].
</p>
</section>
<section id="conformance">
</section>
Expand Down Expand Up @@ -1151,6 +1155,40 @@ <h2>Constrainable Properties</h2>
</tbody>
</table>
</section>
<section>
<h2>Capture timestamps</h2>
<div>
Some video sources can supply information about when a video frame was captured.
This information is useful for example for AV sync and end-to-end delay measurement.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does it help with AV sync if it's only on video frames?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolute audio capture timestamps are available via other web APIs right?
To try to answer the overarching question, presentation timestamps alone are insufficient if you want to do accurate AV sync on a local (think MediaRecorder) or on a remote receiver, especially in the presence of delays in the capture process, or insert VTGs in the capture pipeline to do video processing.
Is this answer enough or do you want to see further changes to clarify this in the spec text based on latest uploaded revision?

</p>
<p>
{{VideoFrameMetadata}}.{{VideoFrameMetadata/captureTime}} MUST be set by the user
agent in video frames sourced from {{MediaDevices/getUserMedia()}},
{{MediaDevices/getDisplayMedia()}}, and {{HTMLMediaElement/captureStream()}}.
</p>
<section>
<h3>{{VideoFrameMetadata}}</h3>
<pre class="idl">
partial dictionary VideoFrameMetadata {
DOMHighResTimeStamp captureTime;
};</pre>
<section class="notoc">
<h4>Members</h4>
<dl class="dictionary-members" data-link-for="VideoFrameMetadata" data-dfn-for="VideoFrameMetadata">
<dt><dfn><code>captureTime</code></dfn> of type <span
class="idlMemberType">DOMHighResTimeStamp, readonly</span></dt>
<dd>
<p>
The capture time of the frame, defined as {{Performance.timeOrigin}} + {{Performance.now()}}.
This is the user agent's best estimate of the instant the frame content was captured or
generated.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the videoFrame.timestamp of these frames? Why do we need another one?

This is in milliseconds when the videoFrame.timestamp is in microseconds. That seems inconsistent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @jan-ivar - I was a little trigger happy in uploading this PR, I will upload changes soon.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

captureTime is absolute which enables delay measurements in a video pipeline. Also, video capture OS APIs typically provide media sampling timestamps which I've measured to be sometimes tens of milliseconds before they're available in a capture sink. I've tried to update what the use of the timestamps are in non-normative text.

I don't know why VideoFrame.timestamp was specified in micros. DOMHighResTimestamp with resolution around 0.1ms seems fine.

</p>
</dd>
</dl>
</section>
</section>
</div>
</section>
<section>
<h2>Exposing MediaStreamTrack source heuristic reactions support</h2>
<div>
Expand Down