[audioworklet] Number of channels in output buffers

[hoch]

Consider the following:

// In AudioWorkletProcessor class definition.
process(inputs, outputs, parameters) {
  for (let i = 0; i < inputs.length; ++i) {
    let inputBuffer = inputs[i];
    let inputChannelCount = inputBuffer.channel;
  }
  for (let o = 0; o < outputs.length; ++o) {
    let outputBuffer = outputs[o];
    let outputChannelCount = outputBuffer.channel;
  }
}

In the example above, what you get from inputChannelCount is determined by the channel count computing algorithm. (spec) However, what you get for outputChannelCount is not well-defined.

The simplest approach is making the matching input/output buffers have the same channel count. For example, if inputChannelCount gives you n, you can expected outputChannelCount to be n.

Although there are more elegant solutions for this, I am against creating AudioBuffer objects (filling in outputs every render quantum) in the process method because that will generate garbages fast and furious.

[hoch]

We can utilize AudioWorkletNodeOptions to handle the channel configuration. Consider this example:

dictionary AudioWorkletNodeOptions : AudioNodeOptions {
  unsigned long numberOfInputs = 1;
  unsigned long numberOfOutputs = 1;
  sequence<unsigned long> outputChannelCount = [];
  record<DOMString, double> parameterData;
}

Consider these 4 cases:

1 input, 1 output

Similar to GainNode. The input channel count is computed by AudioNodeOptions values and the output channel count follows it.

n inputs, 1 output

Similar to ChannelMergerNode. The input channel count will be identical for all inputs, once it's computed by AudioNodeOptions values. The output channel count must be specified by outputChannelCount with the length of 1.

1 input, m outputs

Similar to ChannelSplitterNode. The input channel count is computed by AudioNodeOptions values. The channel count of each output must be specified by outputChannelCount with the length of m.

n input, m outputs

The input channel count will be identical for all inputs, computed by AudioNodeOptions values. The channel count of each output must be specified by outputChannelCount with the length of m.

To sum up, the dynamic channel configuration will not be supported on the output side when you have multiple inputs or outputs. Also outputChannelCount MUST be specified for such case.

[padenot]

Looks like it would work. Does this extend to 0 input n output (multi-channel source), or n input 0 output (multi-channel analyser)? I think it would works well.

[hoch]

1. We should throw on (0 input && 0 output).
2. For 0 input and m outputs, nothing to do with inputs. The output channel count is configured by the user-supplied info.
3. For n inputs and 0 output, the input channel count is computed by the node options. Also nothing to do with outputs.

Yeah, I think it works correctly.

[joeberkovitz]

The thrust of this seems right, but it feels underspecified to me. I think we need to say what happens across this matrix of possibilities in two different cases: 1) where outputChannelCount is specified in AudioWorkletOptions, and 2) where outputChannelCount isn't specified, in which case either an error gets thrown or a default has to be supplied. And in one case (the 1:m case) , there is a reasonable default I think that was overlooked.

So I would propose that the combinations behave like this:

0 input, 0 output

throw an error

0 input, m outputs

use outputChannelCount with length m if specified.
throw if specified, but length doesn't match m
throw if outputChannelCount is not specified.

1 input, 1 output

(these cases are the same as per your observation that AudioNode
use outputChannelCount with length 1 if specified.
throw if specified, but length isn't 1
use computed input channel count if outputChannelCount not specified.

n inputs, 1 output

use outputChannelCount with length 1 if specified.
throw an error if specified, but length isn't 1
throw if outputChannelCount is not specified.

(NOTE: Your comment "The input channel count will be identical for all inputs, once it's computed by AudioNodeOptions values" doesn't seem right -- in the max or clamped-max modes, different inputs on a node can have different numbers of channels according to the channel mixing spec. We cannot to take this behavior away from AudioWorkletNodes -- which means there is no workable default.)

1 input, m outputs

use outputChannelCount with length m if specified
throw an error if specified, but length isn't m
use the computed input channel count for each output if outputChannelCount not specified.

n input, m outputs

use outputChannelCount with length m if specified
throw an error if specified, but length isn't m
throw if outputChannelCount is not specified.

[joeberkovitz]

One side point: why is the options dictionary called AudioWorkletNodeOptions rather than AudioWorkletOptions? This is different from every other options dictionary.

[hoch]

The thrust of this seems right, but it feels underspecified to me.

We were merely exchanging ideas here, but your last comment shaped this thread into the actual spec text.

NOTE: Your comment "The input channel count will be identical for all inputs, once it's computed by AudioNodeOptions values" doesn't seem right -- in the max or clamped-max modes, different inputs on a node can have different numbers of channels according to the channel mixing spec.

Well, the spec of "up/downmixing" section is not really clear on how multiple inputs are handled. So my interpretation was to compute the channel count once based on the options, and apply that single value for all the inputs. This is where my confusion begins:

"Once this number is computed, all of the connections will be up or down-mixed to that many channels."

I think your interpretation is correct - and we should add more text in the spec to clarify that each input gets to have its own computed channel count.

[joeberkovitz]

But it seems to me the spec is actually pretty clear that each input has a distinct count. I read the text here: https://webaudio.github.io/web-audio-api/#channel-up-mixing-and-down-mixing

And at the end, what I see is a clear:

"For each input of an AudioNode, an implementation must:

1. Compute computedNumberOfChannels.
2. For each connection to the input:
   [etc.]"

The words for each input to me convey that the computation of channel count is separate for each input.

[hoch]

@joeberkovitz Oh, I missed that part. Then I think your comment above nicely sums up the behavior. Should we proceed to an actual PR with the more refined text?

By the way, I think we might encounter some issues because of this dynamic mechanism. This means every render quantum will get a newly created object of sequence<sequence<Float32Array>> which in turn it will generate garbages. The channel count changes dynamically, so I think there is no way to optimize this.

[joeberkovitz]

@hoch: I don't agree that garbage is inevitable under this dynamic approach. There is a way to optimize this, and I think it's important to include something like the following as a non-normative guideline:

The minimization of garbage in the audio thread is of great importance to the efficiency of AudioNode implementations. In the absence of connectivity changes to an AWN that affect the number of input channels, implementations are strongly encouraged to pool buffers, only allocating new ones when the channel counts are different from those in the previous quantum.

If UAs generate garbage every quantum, it'll be a big step backward.

[hoch]

@joeberkovitz Consider this example:

let evilGlobal;

class myProcessor extends AudioWorkletProcessor {
  constructor() {
    super();
  }
  
  process(inputs, outputs, parameters) {
    if (!evilGlobal)
      evilGlobal = inputs[0]
  }
}

That's where "buffer pooling" fails. Without creating new sequence and Float32Array objects every render quantum, user can cache the reference in the global scope and inspect/use it any time.

From my point of view, The "let's make allocation more efficient" approach doesn't really help this situation. The allocation is allocation, and it'll accumulate rapidly when it happens.

The good news is only one process method can be invoked at any given moment. There is no concurrency here, so we can reuse the pool of Float32Arrays without worrying about garbages. However, as shown above, user can see the array content being changed by the implementation.

Perhaps we should open another issue for this matter.

[joeberkovitz]

@hoch I'm still not seeing how this presents a problem (and it could be because I'm just plain slow).

If we're worried that people will write broken code that just holds onto the buffer, that can be defeated in the UA by switching between different (but still pooled) sets of buffers, or by re-allocating them every N render quanta where N is large enough to really cut down on the garbage, but small enough to break code that tries to cache buffers.

The sin of allowing code to cache buffers seems far outweighed by the sin of generating lots of garbage on every quantum.

[hoch]

@joeberkovitz I completely agree with you on this, but I am only making the counter argument for the sake of discussion.

The sin of allowing code to cache buffers seems far outweighed by the sin of generating lots of garbage on every quantum.

My concern is whether this "implicit yet visible" change in the buffer is acceptable from the architecture/specification perspective. I haven't seen any other Web API allows this behavior.

Also randomizing underlying scope or storage seems to be rather critical due to the compat/interop issue:

• Non-determinism of PaintWorklet
• Blink's intent on PaintWorklet

I don't think AudioWorklet poses such problem, but at least we have to address it before too late.