Skip to content

Commit ceded2d

Browse files
alvestrandalvestrand
authored
Merge pull request #22 from alvestrand/switch-idl
Import IDL from code
2 parents e4d2155 + 402e6bb commit ceded2d

File tree

2 files changed

+475
-213
lines changed

2 files changed

+475
-213
lines changed

‎index.bs

Lines changed: 95 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ Group: webrtc
77
Repository: alvestrand/webrtc-media-streams/
88
URL: https://alvestrand.github.io/webrtc-media-streams/
99
Editor: Harald Alvestrand, Google https://google.com, [email protected]
10+
Editor: Guido Urdaneta, Google https://google.com, [email protected]
1011
Abstract: This API defines an API surface for manipulating the bits on
1112
Abstract: MediaStreamTracks being sent via an RTCPeerConnection.
1213
Markup Shorthands: css no, markdown yes
@@ -36,6 +37,9 @@ spec: WEB-CODECS; urlPrefix: https://github.com/WICG/web-codecs/
3637
}
3738
}
3839
</pre>
40+
<pre class=link-defaults>
41+
spec:streams; type:interface; text:ReadableStream
42+
</pre>
3943

4044
# Introduction # {#introduction}
4145

@@ -57,6 +61,10 @@ This specification gives an interface that builds on [[WEB-CODECS]] to
5761
provide access to such functionality while retaining the setup flow of
5862
RTCPeerConnection.
5963

64+
This iteration of the specification provides access to encoded media,
65+
which is the output of the encoder part of a codec and the input to the
66+
decoder part of a codec.
67+
6068
# Terminology # {#terminology}
6169

6270
<p noexport>
@@ -69,50 +77,97 @@ The IDL terms <dfn type="idl-name">AudioEncoder</dfn>, <dfn>VideoEncoder</dfn>,
6977
The Streams definition doesn't use WebIDL much, but the WebRTC spec does.
7078
This specification shows the IDL extensions for WebRTC.
7179

72-
It uses an extension to RTCConfiguration in order to pass the extra
73-
decorators for encoder and decoder initialization to an {{RTCPeerConnection}}.
80+
It uses an extension to RTCConfiguration in order to notify the
81+
{{RTCPeerConnection}} that insertable streams will be used, and uses
82+
an additional API on {{RTCRtpSender}} and {{RTCRtpReceiver}} to
83+
insert the processing into the pipeline.
84+
85+
<pre class="idl">
86+
// New dictionary.
87+
dictionary RTCInsertableStreams {
88+
ReadableStream readableStream;
89+
WritableStream writableStream;
90+
};
91+
92+
// New enum for video frame types. Will eventually re-use the equivalent defined
93+
// by WebCodecs.
94+
enum RTCEncodedVideoFrameType {
95+
"empty",
96+
"key",
97+
"delta",
98+
};
99+
100+
dictionary RTCVideoFrameMetadata {
101+
long synchronizationSource;
102+
sequence&lt;long&gt; contributingSources;
103+
};
104+
105+
// New interfaces to define encoded video and audio frames. Will eventually
106+
// re-use or extend the equivalent defined in WebCodecs.
107+
interface RTCEncodedVideoFrame {
108+
readonly attribute RTCEncodedVideoFrameType type;
109+
readonly attribute unsigned long long timestamp;
110+
attribute ArrayBuffer data;
111+
RTCVideoFrameMetadata getMetadata();
112+
};
74113

75-
<pre class='idl'>
76-
typedef (AudioEncoder or VideoEncoder) Encoder;
77-
typedef (AudioDecoder or VideoDecoder) Decoder;
114+
dictionary RTCAudioFrameMetadata {
115+
long synchronizationSource;
116+
sequence&lt;long&gt; contributingSources;
117+
};
78118

79-
dictionary Config {
80-
// To Be Defined
81-
};
119+
interface RTCEncodedAudioFrame {
120+
readonly attribute unsigned long long timestamp;
121+
attribute ArrayBuffer data;
122+
RTCAudioFrameMetadata getMetadata();
123+
};
82124

83-
callback EncoderDecorator = Encoder(Encoder encoder, optional Config config);
84-
callback DecoderDecorator = Decoder(Decoder encoder, optional Config config);
85125

126+
// New fields in RTCConfiguration
86127
partial dictionary RTCConfiguration {
87-
EncoderDecorator encoderFactory;
88-
DecoderDecorator decoderFactory;
128+
boolean forceEncodedVideoInsertableStreams = false;
129+
boolean forceEncodedAudioInsertableStreams = false;
130+
};
131+
132+
// New methods for RTCRtpSender and RTCRtpReceiver
133+
partial interface RTCRtpSender {
134+
RTCInsertableStreams createEncodedVideoStreams();
135+
RTCInsertableStreams createEncodedAudioStreams();
136+
};
137+
138+
partial interface RTCRtpReceiver {
139+
RTCInsertableStreams createEncodedVideoStreams();
140+
RTCInsertableStreams createEncodedAudioStreams();
89141
};
90142
</pre>
91143

92144
## Extension operation ## {#operation}
93-
At creation of an RTCPeerConnection, the following steps are added to
94-
the creation algorithm:
95-
96-
* Let the RTCPeerConnection object have two internal slots named [[\EncoderFactory]] and [[\DecoderFactory]], initialized to null.
97-
* If the RTCConfiguration parameter contains a value for "encoderFactory", store that in [[\EncoderFactory]].
98-
* If the RTCConfiguration parameter contains a value for "decoderFactory", store that in [[\DecoderFactory]].
99-
100-
At the time when a codec is initialized as part of the encoder, run
101-
the following steps:
102-
103-
* If the unencoded data source does not permit access, abort these steps. (OPEN ISSUE: How is this error surfaced?)
104-
* Let the unencoded data source be represented by a ReadableStream called "source".
105-
* Let the encoded data sink be represented by a WriteableStream called
106-
* "sink".
107-
* Let the internal encoder object be called "internalEncoder". "internalEncoder" will have a WritableStream property called "writable" (by virtue of being an instance of WebCodec).
108-
* "internalEncoder" will have a ReadableStream property called "readable".
109-
* If the PeerConnection's [[\EncoderFactory]] is null, pipe "source" to "writable", and pipe "readable" to "sink", and skip the rest of these steps.
110-
* Call the function stored in [[\EncoderFactory]], using the newly initialized encoder and its parameters as arguments.
111-
* Let the return value from the function be "encoder".
112-
* If "encoder" has an attribute "readable", pipe it to "sink". Otherwise, pipe the "internalEncoder"'s "readable" to "sink".
113-
* If "encoder" has an attribute "writable", pipe "source" to it. Otherwise, pipe "source" to the "internalEncoder"'s "writable".
114-
115-
The media will then be processed according to the rules of [[WEB-CODECS]].
145+
146+
At the time when a codec is initialized as part of the encoder, and the
147+
corresponding flag is set in the {{RTCPeerConnection}}'s {{RTCConfiguration}}
148+
argument, ensure that the codec is disabled and produces no output.
149+
150+
Let the {{RTCRtpSender}} or {{RTCRtpReceiver}} have an internal slot,
151+
[[\Streams]], initialized to null.
152+
153+
When {{RTCRtpSender/createEncodedVideoStreams}}() or {{RTCRtpSender/createEncodedAudioStreams}}() is
154+
called, run the following steps:
155+
156+
* If the kind of the sender does not match, throw a {{TypeError}} and abort these steps.
157+
* If the data source does not permit access, throw an {{InvalidAccessError}} and abort these steps.
158+
* If [[\Streams]] is not null, throw an {{InvalidStateError}}.
159+
* Create an {{RTCInsertableStreams}} object 's'.
160+
* Set s.readableStream to a ReadableStream representing the encoded data source.
161+
* Set s.writableStream to a WritableStream representing the encoded data sink.
162+
* Enable the encoded data source.
163+
* Store 's' in the internal slot [[\Streams]].
164+
* Return 's'
165+
166+
When a frame is produced from the encoded data source, place it on the
167+
[[\Streams]].readableStream'.
168+
169+
When a frame appears on the [[\Streams]].writableStream, process it as if it came
170+
directly from the encoded data source.
116171

117172
# Privacy and security considerations # {#privacy}
118173

@@ -123,7 +178,12 @@ However, streams that are isolated (as specified in
123178
[[WEBRTC-IDENTITY]]) or tainted with another origin, cannot be
124179
accessed using this API, since that would break the isolation rule.
125180

181+
The API will allow access to some aspects of timing information that are
182+
otherwise unavailable, which allows some fingerprinting surface.
183+
126184

127185
# Examples # {#examples}
128186

187+
See the explainer document.
188+
129189

0 commit comments

Comments
 (0)