Skip to content

Commit f9e6dc1

Browse files
guidouguidou
authored
Merge pull request #51 from youennf/transform-model
Add transform model
2 parents 06540f9 + 36e24fa commit f9e6dc1

File tree

2 files changed

+1087
-1483
lines changed

2 files changed

+1087
-1483
lines changed

‎index.bs

Lines changed: 140 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -115,13 +115,17 @@ partial dictionary RTCConfiguration {
115115
boolean encodedInsertableStreams = false;
116116
};
117117

118+
typedef (SFrameTransform or RTCRtpScriptTransform) RTCRtpTransform;
119+
118120
// New methods for RTCRtpSender and RTCRtpReceiver
119121
partial interface RTCRtpSender {
120122
RTCInsertableStreams createEncodedStreams();
123+
attribute RTCRtpTransform? transform;
121124
};
122125

123126
partial interface RTCRtpReceiver {
124127
RTCInsertableStreams createEncodedStreams();
128+
attribute RTCRtpTransform? transform;
125129
};
126130
</pre>
127131

@@ -134,31 +138,147 @@ argument, ensure that the codec is disabled and produces no output.
134138

135139
### Stream creation ### {#stream-creation}
136140

137-
Let the {{RTCRtpSender}} or {{RTCRtpReceiver}} have an internal slot,
138-
`[[Streams]]`, initialized to null.
141+
At construction of each {{RTCRtpSender}} or {{RTCRtpReceiver}}, run the following steps:
142+
1. Initialize [=this=].`[[Streams]]` to null.
143+
2. Initialize [=this=].`[[transform]]` to null.
144+
3. Initialize [=this=].`[[readable]]` to the result of <a dfn for="ReadableStream">creating</a> a {{ReadableStream}}. `[[readable]]` is provided frames using the [=readEncodedData=] algorithm given |this| as parameter.
145+
4. Set [=this=].`[[readable]]`.`[[owner]]` to |this|.
146+
5. Initialize [=this=].`[[writable]]` to the result of [=WritableStream/creating=] a {{WritableStream}}, its [=WritableStream/create/writeAlgorithm=] set to [=writeEncodedData=] given |this| as parameter.
147+
6. Set [=this=].`[[writable]]`.`[[owner]]` to |this|.
148+
7. Initialize [=this=].`[[pipeToController]]` to null.
149+
8. Initialize [=this=].`[[lastReceivedFrameTimestamp]]` to zero.
150+
9. If the {{RTCPeerConnection}}'s configuration does not have {{RTCConfiguration/encodedInsertableStreams}} set to "true", queue a task to run the following steps:
151+
1. If [=this=].`[[pipeToController]]` is not null, abort these steps.
152+
2. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}.
153+
<!-- FIXME: Use pipeTo algorithm when available. -->
154+
3. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with [=this=].`[[readable]]`, [=this=].`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`.signal.
155+
156+
The <dfn method for="RTCRtpSender">createEncodedStreams()</dfn> method steps are:
157+
158+
1. If the {{RTCPeerConnection}}'s configuration does not have {{RTCConfiguration/encodedInsertableStreams}} set to "true", throw an "{{InvalidAccessError}}" {{DOMException}} and abort these steps.
159+
2. If the data source does not permit access, throw an "{{InvalidAccessError}}" {{DOMException}} and abort these steps.
160+
3. If [=this=].`[[Streams]]` is not null, throw an "{{InvalidAccessError}}" {{DOMException}}.
161+
4. If [=this=].`[[pipeToController]]` is not null, throw an "{{InvalidAccessError}}" {{DOMException}}.
162+
5. Set [=this=].`[[Streams]]` to an {{RTCInsertableStreams}} object.
163+
6. Set [=this=].`[[Streams]]`.{{RTCInsertableStreams/readable}} to [=this=].`[[readable]]`.
164+
7. Set [=this=].`[[Streams]]`.{{RTCInsertableStreams/writable}} to [=this=].`[[writable]]`.
165+
8. Enable the encoded data source.
166+
10. Return [=this=].`[[Streams]]`.
139167

140-
When {{RTCRtpSender/createEncodedStreams}}() is
141-
called, run the following steps:
168+
### Stream processing ### {#stream-processing}
142169

143-
* If the {{RTCPeerConnection}}'s configuration does not have {{RTCConfiguration/encodedInsertableStreams}} set to "true", throw an {{InvalidStateError}} and abort these steps.
144-
* If the data source does not permit access, throw an {{InvalidAccessError}} and abort these steps.
145-
* If `[[Streams]]` is not null, throw an {{InvalidStateError}}.
146-
* Create an {{RTCInsertableStreams}} object |s|.
147-
* Set |s|.{{RTCInsertableStreams/readable}} to a {{ReadableStream}} representing the encoded data source.
148-
* Set |s|.{{RTCInsertableStreams/writable}} to a {{WritableStream}} representing the encoded data sink.
149-
* Enable the encoded data source.
150-
* Store |s| in the internal slot `[[Streams]]`.
151-
* Return |s|
170+
The <dfn>readEncodedData</dfn> algorithm is given a |rtcObject| as parameter. It is defined by running the following steps:
171+
1. Wait for a frame to be produced by |rtcObject|'s encoder if it is a {{RTCRtpSender}} or |rtcObject|'s packetizer if it is a {{RTCRtpReceiver}}.
172+
2. Let |frame| be the newly produced frame.
173+
3. Set |frame|.`[[owner]]` to |rtcObject|.
174+
4. [=ReadableStream/enqueue=] |frame| in |rtcObject|.`[[readable]]`.
175+
176+
The <dfn>writeEncodedData</dfn> algorithm is given a |rtcObject| as parameter and a |frame| as input. It is defined by running the following steps:
177+
1. If |frame|.`[[owner]]` is not equal to |rtcObject|, abort these steps and return [=a promise resolved with=] undefined. A processor cannot create frames, or move frames between streams.
178+
2. If the |frame|'s {{RTCEncodedVideoFrame/timestamp}} is equal to or larger than |rtcObject|.`[[lastReceivedFrameTimestamp]]`, abort these steps and return [=a promise resolved with=] undefined. A processor cannot reorder frames, although it may delay them or drop them.
179+
3. Set |rtcObject|.`[[lastReceivedFrameTimestamp]]` to the |frame|'s {{RTCEncodedVideoFrame/timestamp}}.
180+
4. Enqueue the frame for processing as if it came directly from the encoded data source, by running one of the following steps:
181+
* If |rtcObject| is a {{RTCRtpSender}}, enqueue it to |rtcObject|'s packetizer, to be processed [=in parallel=].
182+
* If |rtcObject| is a {{RTCRtpReceiver}}, enqueue it to |rtcObject|'s decoder, to be processed [=in parallel=].
183+
5. Return [=a promise resolved with=] undefined.
184+
185+
## Extension attribute ## {#attribute}
186+
187+
A RTCRtpTransform has two private slots called `[[readable]]` and `[[writable]]`.
188+
189+
The <dfn attribute for="RTCRtpSender,RTCRtpReceiver">transform</dfn> getter steps are:
190+
1. Return [=this=].`[[transform]]`.
191+
192+
The `transform` setter steps are:
193+
2. Let |transform| be the argument to the setter.
194+
3. Let |checkedTransform| set to |transform| if it is not null or to an [=identity transform stream=] otherwise.
195+
3. Let |reader| be the result of [=ReadableStream/getting a reader=] for |checkedTransform|.`[[readable]]`.
196+
4. Let |writer| be the result of [=WritableStream/getting a writer=] for |checkedTransform|.`[[writable]]`.
197+
5. Initialize |newPipeToController| to a new {{AbortController}}.
198+
6. If [=this=].`[[pipeToController]]` is not null, run the following steps:
199+
1. [=AbortSignal/Add=] the [=chain transform algorithm=] to [=this=].`[[pipeToController]]`.signal.
200+
2. [=AbortSignal/signal abort=] [=this=].`[[pipeToController]]`.signal.
201+
7. Else, run the [=chain transform algorithm=] steps.
202+
8. Set [=this=].`[[pipeToController]]` to |newPipeToController|.
203+
9. Set [=this=].`[[transform]]` to |transform|.
204+
205+
The <dfn>chain transform algorithm</dfn> steps are defined as:
206+
1. If |newPipeToController|'s [=AbortSignal/aborted flag=] is true, abort these steps.
207+
2. [=ReadableStreamDefaultReader/Release=] |reader|.
208+
3. [=WritableStreamDefaultWriter/Release=] |writer|.
209+
4. Assert that |newPipeToController| is the same object as |rtcObject|.`[[pipeToController]]`.
210+
<!-- FIXME: Use pipeTo algorithm when available. -->
211+
5. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|.signal.
212+
6. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|.signal.
213+
214+
This algorithm is defined so that transforms can be updated dynamically.
215+
There is no guarantee on which frame will happen the switch from the previous transform to the new transform.
216+
217+
If a web application sets the transform synchronously at creation of the {{RTCRtpSender}} (for instance when calling addTrack), the transform will receive the first frame generated by the {{RTCRtpSender}}'s encoder.
218+
Similarly, if a web application sets the transform synchronously at creation of the {{RTCRtpReceiver}} (for instance when calling addTrack, or at track event handler), the transform will receive the first full frame generated by the {{RTCRtpReceiver}}'s packetizer.
219+
220+
# SFrameTransform # {#sframe}
152221

153-
### Stream processing ### {#stream-processing}
222+
<pre class="idl">
223+
enum SFrameTransformRole {
224+
"encrypt",
225+
"decrypt"
226+
};
227+
228+
dictionary SFrameTransformOptions {
229+
SFrameTransformRole role = "encrypt";
230+
};
231+
232+
[Exposed=(Window,DedicatedWorker)]
233+
interface SFrameTransform {
234+
constructor(optional SFrameTransformOptions options);
235+
// FIXME: add key handling methods.
236+
};
237+
SFrameTransform includes GenericTransformStream;
238+
</pre>
154239

155-
When a frame is produced from the encoded data source, place it on the
156-
`[[Streams]]`.{{RTCInsertableStreams/readable}}.
240+
The <dfn constructor for="SFrameTransform" lt="SFrameTransform(options)"><code>new SFrameTransform(<var>options</var>)</code></dfn> constructor steps are:
241+
1. Let |transformAlgorithm| be an algorithm which takes a |frame| as input and runs the <a href="#sframe-transform-algorithm">SFrame transform algorithm</a> with |this| and |frame|.
242+
2. Set |this|.`[[transform]]` to the result of [=TransformStream/creating=] a {{TransformStream}}, with [=TransformStream/create/transformAlgorithm=] set to |transformAlgorithm|.
243+
3. Let |options| be the method's first argument.
244+
4. Set |this|.`[[role]]` to |options|["{{SFrameTransportOptions/role}}"].
245+
5. Set |this|.`[[readable]]` to |this|.`[[transform]]`.`[[readable]]`.
246+
6. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`.
247+
248+
## SFrame transform algorithm ## {#sframe-transform-algorithm}
249+
250+
The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |frame|, runs these steps:
251+
1. Let |role| be |sframe|.`[[role]]`.
252+
2. If |frame|.`[[rtcObject]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'.
253+
3. If |frame|.`[[rtcObject]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'.
254+
4. Let |data| be undefined.
255+
5. If |frame| is an ArrayBuffer, set |data| to |frame|. // FIXME Support BufferSource
256+
6. If |frame| is an {{RTCEncodedAudioFrame}}, set |data| to |frame|.{{RTCEncodedAudioFrame/data}}
257+
7. If |frame| is an {{RTCEncodedVideoFrame}}, set |data| to |frame|.{{RTCEncodedVideoFrame/data}}
258+
8. If |data| is undefined, abort these steps.
259+
9. Let |buffer| be the result of running the SFrame algorithm with |data| and |role| as parameters. This algorithm is defined by the <a href="https://datatracker.ietf.org/doc/draft-omara-sframe/">SFrame specification</a>.
260+
10. If |frame| is an ArrayBuffer, set |frame| to |buffer|.
261+
11. If |frame| is an {{RTCEncodedAudioFrame}}, set |frame|.{{RTCEncodedAudioFrame/data}} to |buffer|.
262+
12. If |frame| is an {{RTCEncodedVideoFrame}}, set |frame|.{{RTCEncodedVideoFrame/data}} to |buffer|.
263+
13. [=ReadableStream/Enqueue=] |frame| in |sframe|.`[[transform]]`.
264+
265+
# RTCRtpScriptTransform # {#scriptTransform}
266+
267+
<pre class="idl">
268+
[Exposed=(Window)]
269+
interface RTCRtpScriptTransform {
270+
constructor(Worker worker, optional object options);
271+
// FIXME: add messaging methods.
272+
};
273+
</pre>
157274

158-
When a frame appears on the `[[Streams]]`.{{RTCInsertableStreams/writable}}, do the following:
159-
* Check that the frame is a a valid frame that has been created by the encoded data source; if it is not, discard it. A processor cannot create frames, or move frames between streams.
160-
* Check that the frame's {{RTCEncodedVideoFrame/timestamp}} is equal to or larger than any previously received frame. A processor cannot reorder frames, although it may delay them or drop them.
161-
* Process the frame as if it came directly from the encoded data source.
275+
The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(<var>worker</var>, <var>options</var>)</code></dfn> constructor steps are:
276+
1. Set |t1| to an [=identity transform stream=].
277+
2. Set |t2| to an [=identity transform stream=].
278+
3. Set |this|.`[[writable]]` to |t1|.`[[writable]]`.
279+
4. Set |this|.`[[readable]]` to |t2|.`[[readable]]`.
280+
5. FIXME: transfer |t1|.`[[readable]]` and |t2|.`[[writable]]` to the dedicated worker.
281+
6. FIXME: Create counterpart of |this| in dedicated worker, for instance expose transfered |t1|.`[[readable]]` and |t2|.`[[writable]]`.
162282

163283
# Privacy and security considerations # {#privacy}
164284

@@ -176,5 +296,3 @@ otherwise unavailable, which allows some fingerprinting surface.
176296
# Examples # {#examples}
177297

178298
See the [explainer document](https://github.com/w3c/webrtc-insertable-streams/blob/master/explainer.md#code-examples).
179-
180-

0 commit comments

Comments
 (0)