Skip to content

Commit 9e56e54

Browse files
alvestrandalvestrand
committed
Import IDL from code
1 parent e4d2155 commit 9e56e54

File tree

2 files changed

+433
-214
lines changed

2 files changed

+433
-214
lines changed

‎index.bs

Lines changed: 88 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ Group: webrtc
77
Repository: alvestrand/webrtc-media-streams/
88
URL: https://alvestrand.github.io/webrtc-media-streams/
99
Editor: Harald Alvestrand, Google https://google.com, [email protected]
10+
Editor: Guido Urdaneta, Google https://google.com, [email protected]
1011
Abstract: This API defines an API surface for manipulating the bits on
1112
Abstract: MediaStreamTracks being sent via an RTCPeerConnection.
1213
Markup Shorthands: css no, markdown yes
@@ -36,6 +37,9 @@ spec: WEB-CODECS; urlPrefix: https://github.com/WICG/web-codecs/
3637
}
3738
}
3839
</pre>
40+
<pre class=link-defaults>
41+
spec:streams; type:interface; text:ReadableStream
42+
</pre>
3943

4044
# Introduction # {#introduction}
4145

@@ -57,6 +61,10 @@ This specification gives an interface that builds on [[WEB-CODECS]] to
5761
provide access to such functionality while retaining the setup flow of
5862
RTCPeerConnection.
5963

64+
This iteration of the specification provides access to encoded media,
65+
which is the output of the encoder part of a codec and the input to the
66+
decoder part of a codec.
67+
6068
# Terminology # {#terminology}
6169

6270
<p noexport>
@@ -69,50 +77,90 @@ The IDL terms <dfn type="idl-name">AudioEncoder</dfn>, <dfn>VideoEncoder</dfn>,
6977
The Streams definition doesn't use WebIDL much, but the WebRTC spec does.
7078
This specification shows the IDL extensions for WebRTC.
7179

72-
It uses an extension to RTCConfiguration in order to pass the extra
73-
decorators for encoder and decoder initialization to an {{RTCPeerConnection}}.
80+
It uses an extension to RTCConfiguration in order to notify the
81+
{{RTCPeerConnection}} that insertable streams will be used, and uses
82+
an additional API on {{RTCRtpSender}} and {{RTCRtpReceiver}} to
83+
insert the processing into the pipeline.
7484

75-
<pre class='idl'>
76-
typedef (AudioEncoder or VideoEncoder) Encoder;
77-
typedef (AudioDecoder or VideoDecoder) Decoder;
85+
<pre class="idl">
86+
// New dictionary.
87+
dictionary RTCInsertableStreams {
88+
ReadableStream readable;
89+
WritableStream writable;
90+
};
7891

79-
dictionary Config {
80-
// To Be Defined
81-
};
92+
// New enum for video frame types. Will eventually re-use the equivalent defined
93+
// by WebCodecs.
94+
enum RTCEncodedVideoFrameType {
95+
"empty",
96+
"key",
97+
"delta",
98+
};
8299

83-
callback EncoderDecorator = Encoder(Encoder encoder, optional Config config);
84-
callback DecoderDecorator = Decoder(Decoder encoder, optional Config config);
100+
dictionary RTCFrameMetadata {
101+
long synchronizationSource;
102+
sequence&lt;long&gt; contributingSources;
103+
};
85104

105+
// New interfaces to define encoded video and audio frames. Will eventually
106+
// re-use or extend the equivalent defined in WebCodecs.
107+
// The additionalData fields contain metadata about the frame and will
108+
// eventually be exposed differently.
109+
interface RTCEncodedVideoFrame {
110+
readonly attribute RTCEncodedVideoFrameType type;
111+
readonly attribute unsigned long long timestamp;
112+
attribute ArrayBuffer data;
113+
RTCFrameMetadata getMetadata();
114+
};
115+
116+
interface RTCEncodedAudioFrame {
117+
readonly attribute unsigned long long timestamp;
118+
attribute ArrayBuffer data;
119+
RTCFrameMetadata getMetadata();
120+
};
121+
122+
123+
// New fields in RTCConfiguration
86124
partial dictionary RTCConfiguration {
87-
EncoderDecorator encoderFactory;
88-
DecoderDecorator decoderFactory;
125+
boolean forceEncodedVideoInsertableStreams = false;
126+
boolean forceEncodedAudioInsertableStreams = false;
127+
};
128+
129+
// New methods for RTCRtpSender and RTCRtpReceiver
130+
partial interface RTCRtpSender {
131+
RTCInsertableStreams createEncodedVideoStreams();
132+
RTCInsertableStreams createEncodedAudioStreams();
133+
};
134+
135+
partial interface RTCRtpReceiver {
136+
RTCInsertableStreams createEncodedVideoStreams();
137+
RTCInsertableStreams createEncodedAudioStreams();
89138
};
90139
</pre>
91140

92141
## Extension operation ## {#operation}
93-
At creation of an RTCPeerConnection, the following steps are added to
94-
the creation algorithm:
95-
96-
* Let the RTCPeerConnection object have two internal slots named [[\EncoderFactory]] and [[\DecoderFactory]], initialized to null.
97-
* If the RTCConfiguration parameter contains a value for "encoderFactory", store that in [[\EncoderFactory]].
98-
* If the RTCConfiguration parameter contains a value for "decoderFactory", store that in [[\DecoderFactory]].
99-
100-
At the time when a codec is initialized as part of the encoder, run
101-
the following steps:
102-
103-
* If the unencoded data source does not permit access, abort these steps. (OPEN ISSUE: How is this error surfaced?)
104-
* Let the unencoded data source be represented by a ReadableStream called "source".
105-
* Let the encoded data sink be represented by a WriteableStream called
106-
* "sink".
107-
* Let the internal encoder object be called "internalEncoder". "internalEncoder" will have a WritableStream property called "writable" (by virtue of being an instance of WebCodec).
108-
* "internalEncoder" will have a ReadableStream property called "readable".
109-
* If the PeerConnection's [[\EncoderFactory]] is null, pipe "source" to "writable", and pipe "readable" to "sink", and skip the rest of these steps.
110-
* Call the function stored in [[\EncoderFactory]], using the newly initialized encoder and its parameters as arguments.
111-
* Let the return value from the function be "encoder".
112-
* If "encoder" has an attribute "readable", pipe it to "sink". Otherwise, pipe the "internalEncoder"'s "readable" to "sink".
113-
* If "encoder" has an attribute "writable", pipe "source" to it. Otherwise, pipe "source" to the "internalEncoder"'s "writable".
114-
115-
The media will then be processed according to the rules of [[WEB-CODECS]].
142+
143+
At the time when a codec is initialized as part of the encoder, and the
144+
corresponding flag is set in the {{RTCPeerConnection}}'s {{RTCConfiguration}}
145+
argument, ensure that the codec is disabled and produces no output.
146+
147+
When {{RTCRtpSender/createEncodedVideoStreams}}() or {{RTCRtpSender/createEncodedAudioStreams}}() is
148+
called, run the following steps:
149+
150+
* If the kind of the sender does not match, throw a {{TypeError}} and abort these steps.
151+
* If the data source does not permit access, throw an {{InvalidAccessError}} and abort these steps.
152+
* Create an {{RTCInsertableStreams}} object 's'.
153+
* Set s.readable to a ReadableStream representing the encoded data source.
154+
* Set s.writable to a WritableStream representing the encoded data sink.
155+
* Enable the encoded data source.
156+
* Store 's' in an internal slot [[\streams]].
157+
* Return 's'
158+
159+
When a frame is produced from the encoded data source, place it on the
160+
[[\streams]].readable' stream.
161+
162+
When a frame appears on the [[\streams]].writable stream, process it as if it came
163+
directly from the encoded data source.
116164

117165
# Privacy and security considerations # {#privacy}
118166

@@ -123,7 +171,12 @@ However, streams that are isolated (as specified in
123171
[[WEBRTC-IDENTITY]]) or tainted with another origin, cannot be
124172
accessed using this API, since that would break the isolation rule.
125173

174+
The API will allow access to some aspects of timing information that are
175+
otherwise unavailable, which allows some fingerprinting surface.
176+
126177

127178
# Examples # {#examples}
128179

180+
See the explainer document.
181+
129182

0 commit comments

Comments
 (0)