Apply ambisonics to an audio file then to a video file result to a non playable video

[bbaudel]

Here is the code using it. I will add a sample online soon (tested with Omnitone 1.0.6 and 1.2.1) :
https://github.com/gopro/forgejs/tree/alps-700-omnitone-fix
https://github.com/gopro/forgejs-samples/tree/alps-700-omnitone-fix

[bbaudel]

Here is the sample to execute on a localhost:
https://www.dropbox.com/s/ok3c101ej0r71mk/sample.zip?dl=0

Here is the sample online:
http://www.kolor.com/private_forgejs/projects/ambisonic/ to begin by an image scene with ambisonics audio (first scene)
http://www.kolor.com/private_forgejs/projects/ambisonic/#first-scene&uid=scene-2 to begin by a video with ambisonics (3rd scene)

[bbaudel]

Notes on the samples

• I've also noticed that the sound volume is not always the same and noise can happen in some cases.
• The global AudioContext used is the same for wav and mp4 files, and when a new scene is requested the FOARenderer is connected to it and disconnected from the previous scene.

[hoch]

So sorry for the late response! (Was on vacation when the issue was filed...)

I am not sure if I understood the problem correctly. This is what I did:

1. Go to http://www.kolor.com/private_forgejs/projects/ambisonic/

• The audio plays correctly. (scene 1)

2. Press ">" on the right.

• The audio still plays correctly. (scene 2)

3. Press ">" on the right again.

• No audio from the player. (scene 3)

But if I access the scene 3 with the direct link, it plays the audio correctly.

Did I get your problem right?

[bbaudel]

Yes exactly.
It can be linked to our audio implementation.
For image scene with ambisonics we instanciate an audio tag and use a global webaudio context which are destroyed and "unlinked" when leaving the scene.
For video scene with embedded ambisonics we instanciate a video tag and use a global webaudio context which are destroyed and "unlinked" when leaving the scene.

[hoch]

Have not looked at the code yet...

Can you keep the audio context and the Omnitone renderer? Why do you have to re-instantiate the whole thing for every scene?

The Omnitone renderer is simple and transparent, so all you need to do is the reset the rotation matrix to put it back to the initial state. There will be a tail of 256 samples from the convolution, but that's less than ~7ms. It should be trivial to reconstruct the renderer from scratch, but still there is a cost and you might want to avoid that.

For each scene, all you have to do is to create MediaElementAudioSourceNode for audio or video tag and feed it into the renderer.

Keep it mind that an AudioNode must be connected to the node belongs to the same audio context.

[bbaudel]

Thanks for your feedback.

I've tested with a SoundManager class which create a FOARenderer and connect it to the "global" audio context.
Then the Sound and Video classes now only connect the sound element source to the FOARenderer input when they are instanciated.

But I've exactly the same result.

Note: on some sounds or videos I'll be happy to switch between FOA and HOA renderer also, it's why I was creating and instanciating the Ambisonics renderer on each sound or video where ambisonics are required on my first test.

[hoch]

Could you try it again with the latest version of Canary? I think it might have been a bug in Chrome's implementation and I fixed the issue a couple of days ago.

[bbaudel]

Same result with a few tests with the latest Canary version.

[hoch]

Okay, I think this is related to an implementation bug in Chrome. Have you had a chance to try this out in FireFox?

[hoch]

My bet is changing the source of MediaElement dynamically does not go well with the attached MediaElementSourceNode (which is a Chrome bug). For the workaround/hack, I suggest you to do this:

1. Create VideoElement
2. Set src.
3. Create MESN with the video element.
4. Connect to the Omnitone.

When changing the source:

1. Disconnect MESN from Omnitone.
2. Give it ~20ms.
3. Change the new src in the element.
4. Connect to MESN the Omnitone again.

[hoch]

One more finding: if you change .src on MediaElement while the element was being played, you'll get an error. I had to call .play() after I set the source, otherwise I get the silent output.

[hoch]

@bbaudel The example above does not work correctly any more because of the recent Autoplay policy change. Could you check it out? I would like to revisit this because the problem might have been fixed in Chrome.