Skip to content

Commit d0d557c

Browse files
youennfyouennf
committed
Add API and algorithm to expose transform to workers
1 parent bc9ae2d commit d0d557c

File tree

1 file changed

+99
-75
lines changed

1 file changed

+99
-75
lines changed

‎index.bs

Lines changed: 99 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -62,69 +62,13 @@ an additional API on {{RTCRtpSender}} and {{RTCRtpReceiver}} to
6262
insert the processing into the pipeline.
6363

6464
<pre class="idl">
65-
// New dictionary.
66-
dictionary RTCInsertableStreams {
67-
ReadableStream readable;
68-
WritableStream writable;
69-
};
70-
71-
// New enum for video frame types. Will eventually re-use the equivalent defined
72-
// by WebCodecs.
73-
enum RTCEncodedVideoFrameType {
74-
"empty",
75-
"key",
76-
"delta",
77-
};
78-
79-
dictionary RTCEncodedVideoFrameMetadata {
80-
long long frameId;
81-
sequence&lt;long long&gt; dependencies;
82-
unsigned short width;
83-
unsigned short height;
84-
long spatialIndex;
85-
long temporalIndex;
86-
long synchronizationSource;
87-
sequence&lt;long&gt; contributingSources;
88-
};
89-
90-
// New interfaces to define encoded video and audio frames. Will eventually
91-
// re-use or extend the equivalent defined in WebCodecs.
92-
[Exposed=Window]
93-
interface RTCEncodedVideoFrame {
94-
readonly attribute RTCEncodedVideoFrameType type;
95-
readonly attribute unsigned long long timestamp;
96-
attribute ArrayBuffer data;
97-
RTCEncodedVideoFrameMetadata getMetadata();
98-
};
99-
100-
dictionary RTCEncodedAudioFrameMetadata {
101-
long synchronizationSource;
102-
sequence&lt;long&gt; contributingSources;
103-
};
104-
105-
[Exposed=Window]
106-
interface RTCEncodedAudioFrame {
107-
readonly attribute unsigned long long timestamp;
108-
attribute ArrayBuffer data;
109-
RTCEncodedAudioFrameMetadata getMetadata();
110-
};
111-
112-
113-
// New fields in RTCConfiguration
114-
partial dictionary RTCConfiguration {
115-
boolean encodedInsertableStreams = false;
116-
};
117-
11865
typedef (SFrameTransform or RTCRtpScriptTransform) RTCRtpTransform;
11966

120-
// New methods for RTCRtpSender and RTCRtpReceiver
12167
partial interface RTCRtpSender {
122-
RTCInsertableStreams createEncodedStreams();
12368
attribute RTCRtpTransform? transform;
12469
};
12570

12671
partial interface RTCRtpReceiver {
127-
RTCInsertableStreams createEncodedStreams();
12872
attribute RTCRtpTransform? transform;
12973
};
13074
</pre>
@@ -139,39 +83,27 @@ argument, ensure that the codec is disabled and produces no output.
13983
### Stream creation ### {#stream-creation}
14084

14185
At construction of each {{RTCRtpSender}} or {{RTCRtpReceiver}}, run the following steps:
142-
1. Initialize [=this=].`[[Streams]]` to null.
14386
2. Initialize [=this=].`[[transform]]` to null.
144-
3. Initialize [=this=].`[[readable]]` to the result of <a dfn for="ReadableStream">creating</a> a {{ReadableStream}}. `[[readable]]` is provided frames using the [=readEncodedData=] algorithm given |this| as parameter.
87+
3. Initialize [=this=].`[[readable]]` to the result of <a dfn for="ReadableStream">creating</a> a {{ReadableStream}}. [=this=].`[[readable]]` is provided frames using the [=readEncodedData=] algorithm given |this| as parameter.
14588
4. Set [=this=].`[[readable]]`.`[[owner]]` to |this|.
14689
5. Initialize [=this=].`[[writable]]` to the result of [=WritableStream/creating=] a {{WritableStream}}, its [=WritableStream/create/writeAlgorithm=] set to [=writeEncodedData=] given |this| as parameter.
14790
6. Set [=this=].`[[writable]]`.`[[owner]]` to |this|.
14891
7. Initialize [=this=].`[[pipeToController]]` to null.
14992
8. Initialize [=this=].`[[lastReceivedFrameTimestamp]]` to zero.
150-
9. If the {{RTCPeerConnection}}'s configuration does not have {{RTCConfiguration/encodedInsertableStreams}} set to "true", queue a task to run the following steps:
93+
9. [=Queue a task=] to run the following steps:
15194
1. If [=this=].`[[pipeToController]]` is not null, abort these steps.
15295
2. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}.
15396
<!-- FIXME: Use pipeTo algorithm when available. -->
15497
3. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with [=this=].`[[readable]]`, [=this=].`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`.signal.
15598

156-
The <dfn method for="RTCRtpSender">createEncodedStreams()</dfn> method steps are:
157-
158-
1. If the {{RTCPeerConnection}}'s configuration does not have {{RTCConfiguration/encodedInsertableStreams}} set to "true", throw an "{{InvalidAccessError}}" {{DOMException}} and abort these steps.
159-
2. If the data source does not permit access, throw an "{{InvalidAccessError}}" {{DOMException}} and abort these steps.
160-
3. If [=this=].`[[Streams]]` is not null, throw an "{{InvalidAccessError}}" {{DOMException}}.
161-
4. If [=this=].`[[pipeToController]]` is not null, throw an "{{InvalidAccessError}}" {{DOMException}}.
162-
5. Set [=this=].`[[Streams]]` to an {{RTCInsertableStreams}} object.
163-
6. Set [=this=].`[[Streams]]`.{{RTCInsertableStreams/readable}} to [=this=].`[[readable]]`.
164-
7. Set [=this=].`[[Streams]]`.{{RTCInsertableStreams/writable}} to [=this=].`[[writable]]`.
165-
8. Enable the encoded data source.
166-
10. Return [=this=].`[[Streams]]`.
16799

168100
### Stream processing ### {#stream-processing}
169101

170102
The <dfn>readEncodedData</dfn> algorithm is given a |rtcObject| as parameter. It is defined by running the following steps:
171103
1. Wait for a frame to be produced by |rtcObject|'s encoder if it is a {{RTCRtpSender}} or |rtcObject|'s packetizer if it is a {{RTCRtpReceiver}}.
172104
2. Let |frame| be the newly produced frame.
173105
3. Set |frame|.`[[owner]]` to |rtcObject|.
174-
4. [=ReadableStream/enqueue=] |frame| in |rtcObject|.`[[readable]]`.
106+
4. [=ReadableStream/Enqueue=] |frame| in |rtcObject|.`[[readable]]`.
175107

176108
The <dfn>writeEncodedData</dfn> algorithm is given a |rtcObject| as parameter and a |frame| as input. It is defined by running the following steps:
177109
1. If |frame|.`[[owner]]` is not equal to |rtcObject|, abort these steps and return [=a promise resolved with=] undefined. A processor cannot create frames, or move frames between streams.
@@ -245,7 +177,7 @@ The <dfn constructor for="SFrameTransform" lt="SFrameTransform(options)"><code>n
245177
5. Set |this|.`[[readable]]` to |this|.`[[transform]]`.`[[readable]]`.
246178
6. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`.
247179

248-
## SFrame transform algorithm ## {#sframe-transform-algorithm}
180+
## Algorithm ## {#sframe-transform-algorithm}
249181

250182
The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |frame|, runs these steps:
251183
1. Let |role| be |sframe|.`[[role]]`.
@@ -275,20 +207,112 @@ The <dfn method for="SFrameTransform">setEncryptionKey(|key|, |keyID|)</dfn> met
275207
# RTCRtpScriptTransform # {#scriptTransform}
276208

277209
<pre class="idl">
210+
// New enum for video frame types. Will eventually re-use the equivalent defined
211+
// by WebCodecs.
212+
enum RTCEncodedVideoFrameType {
213+
"empty",
214+
"key",
215+
"delta",
216+
};
217+
218+
dictionary RTCEncodedVideoFrameMetadata {
219+
long long frameId;
220+
sequence&lt;long long&gt; dependencies;
221+
unsigned short width;
222+
unsigned short height;
223+
long spatialIndex;
224+
long temporalIndex;
225+
long synchronizationSource;
226+
sequence&lt;long&gt; contributingSources;
227+
};
228+
229+
// New interfaces to define encoded video and audio frames. Will eventually
230+
// re-use or extend the equivalent defined in WebCodecs.
231+
[Exposed=Window]
232+
interface RTCEncodedVideoFrame {
233+
readonly attribute RTCEncodedVideoFrameType type;
234+
readonly attribute unsigned long long timestamp;
235+
attribute ArrayBuffer data;
236+
RTCEncodedVideoFrameMetadata getMetadata();
237+
};
238+
239+
dictionary RTCEncodedAudioFrameMetadata {
240+
long synchronizationSource;
241+
sequence&lt;long&gt; contributingSources;
242+
};
243+
244+
[Exposed=Window]
245+
interface RTCEncodedAudioFrame {
246+
readonly attribute unsigned long long timestamp;
247+
attribute ArrayBuffer data;
248+
RTCEncodedAudioFrameMetadata getMetadata();
249+
};
250+
251+
252+
// New interfaces to expose JavaScript-based transforms.
253+
254+
[Global=(Worker,DedicatedWorker),Exposed=DedicatedWorker]
255+
interface RTCTransformEvent : Event {
256+
readonly attribute RTCRtpScriptTransformer transformer;
257+
};
258+
259+
[Global=(Worker,DedicatedWorker),Exposed=DedicatedWorker]
260+
partial interface DedicatedWorkerGlobalScope : WorkerGlobalScope {
261+
attribute EventHandler onrtctransform;
262+
};
263+
264+
[Global=(Worker,DedicatedWorker),Exposed=DedicatedWorker]
265+
interface RTCRtpScriptTransformer {
266+
readonly attribute ReadableStream readable;
267+
readonly attribute WritableStream writable;
268+
readonly attribute any options;
269+
};
270+
278271
[Exposed=(Window)]
279272
interface RTCRtpScriptTransform {
280-
constructor(Worker worker, optional object options);
273+
constructor(Worker worker, optional any options);
281274
// FIXME: add messaging methods.
282275
};
283276
</pre>
284277

278+
## Operations ## {#RTCRtpScriptTransform-operations}
279+
285280
The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(<var>worker</var>, <var>options</var>)</code></dfn> constructor steps are:
286281
1. Set |t1| to an [=identity transform stream=].
287282
2. Set |t2| to an [=identity transform stream=].
288283
3. Set |this|.`[[writable]]` to |t1|.`[[writable]]`.
289284
4. Set |this|.`[[readable]]` to |t2|.`[[readable]]`.
290-
5. FIXME: transfer |t1|.`[[readable]]` and |t2|.`[[writable]]` to the dedicated worker.
291-
6. FIXME: Create counterpart of |this| in dedicated worker, for instance expose transfered |t1|.`[[readable]]` and |t2|.`[[writable]]`.
285+
5. Let |serializedOptions| be the result of [$StructuredSerialize$](|object|).
286+
6. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
287+
7. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
288+
8. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
289+
1. Let |transformerOptions| be the result of [$StructuredDeserialize$](|serializedOptions|, the current Realm).
290+
2. Let |readable| be the result of [$StructuredDeserialize$](|serializedReadable|, the current Realm).
291+
3. Let |writable| be the result of [$StructuredDeserialize$](|serializedWritable|, the current Realm).
292+
4. Let |transformer| be a new {{RTCRtpScriptTransformer}}.
293+
5. Set |transformer|.`[[options]]` to |transformerOptions|.
294+
6. Set |transformer|.`[[readable]]` to |readable|.
295+
7. Set |transformer|.`[[writable]]` to |writable|.
296+
8. Let |event| be the result of [=creating an event=] with {{RTCTransformEvent}}.
297+
9. Set |event|.type attribute to "rtctransform".
298+
10. Set |event|.transformer to |transformer|.
299+
11. Dispatch |event| on |worker|’s global scope.
300+
301+
// FIXME: Describe error handling (worker closing flag true at RTCRtpScriptTransform creation time. And worker being terminated while transform is processing data).
302+
303+
## Attributes ## {#RTCRtpScriptTransformer-attributes}
304+
305+
A RTCRtpScriptTransformer has three private slots called `[[options]]`, `[[readable]]` and `[[writable]]`.
306+
307+
The <dfn attribute for="RTCRtpScriptTransformer">options</dfn> getter steps are:
308+
1. Return [=this=].`[[options]]`.
309+
310+
The <dfn attribute for="RTCRtpScriptTransformer">readable</dfn> getter steps are:
311+
1. Return [=this=].`[[readable]]`.
312+
313+
The <dfn attribute for="RTCRtpScriptTransformer">writable</dfn> getter steps are:
314+
1. Return [=this=].`[[writable]]`.
315+
292316

293317
# Privacy and security considerations # {#privacy}
294318

0 commit comments

Comments
 (0)