How to Include AVPlayer Audio in SCNRecorder Recordings?

[giomurru]

Hi there,

I'm encountering an issue with SCNRecorder while trying to record an SCNView. The view is displaying a video that is mapped as a texture. The video is played using an AVPlayer, and its visuals and audio are perfectly synchronized when displayed.

However, when I use SCNRecorder to record the scene, only the video is captured — the audio is missing.

From my exploration, I noticed that SCNRecorder includes an AudioEngine class for playing soundtracks. However, this approach seems tailored to a custom player that doesn't support video playback. Additionally, trying to synchronize a separate audio playback manually with the video being rendered by AVPlayer feels overly complex and prone to issues.

I'm wondering... is it feasible to extend SCNRecorder to initialize audio recording from an AVPlayer instance that is already in use?

Any insights, suggestions, or pointers would be greatly appreciated!

[v-grigoriev]

Hi @giomurru, I believe this can be done by slightly modifying SCNRecroder.

In general, we can't do anything with AVPlayer itself.

I see that SCNView provides us with audioEngine reference.

If SCNView uses the engine to play any audio you can try attaching to it.
So you can try the code below modifying SCNRecorder according to comments I made.

    sceneView.prepareForRecording()

    do {
      guard let recorder = sceneView.recorder else { return }
      recorder.useAudioEngine = true

      // You need to make audioFormat public to access it
      recorder.audioInput.audioFormat = /* determ your format */

      sceneView.audioEngine.mainMixerNode.installTap(
        onBus: 0,
        bufferSize: 4096,
        format: nil
      ) { [weak recorder] (buffer, time) in
        guard let recorder else { return }

        do {
          let sampleBuffer = try Self.createAudioSampleBuffer(from: buffer, time: time)

          // You need to define new audioEngine(_: AVAudioEngine, didOutputAudioSampleBuffer: CMSampleBuffer) handler.
          recorder.audioInput.audioEngine(sceneView.audioEngine, didOutputAudioSampleBuffer: sampleBuffer)
        }
        catch {
          // Catch errors on your own.
        }
      }
    }
    
static func createAudioSampleBuffer(from buffer: AVAudioPCMBuffer, time: AVAudioTime) throws -> CMSampleBuffer {
    let audioBufferList = buffer.mutableAudioBufferList
    let streamDescription = buffer.format.streamDescription.pointee
    let timescale = CMTimeScale(streamDescription.mSampleRate)
    let format = try CMAudioFormatDescription(audioStreamBasicDescription: streamDescription)
    let sampleBuffer = try CMSampleBuffer(
      dataBuffer: nil,
      formatDescription: format,
      numSamples: CMItemCount(buffer.frameLength),
      sampleTimings: [
        CMSampleTimingInfo(
          duration: CMTime(value: 1, timescale: timescale),
          presentationTimeStamp: CMTime(
            seconds: AVAudioTime.seconds(forHostTime: time.hostTime),
            preferredTimescale: timescale
          ),
          decodeTimeStamp: .invalid
        )
      ],
      sampleSizes: []
    )
    try sampleBuffer.setDataBuffer(fromAudioBufferList: audioBufferList)
    return sampleBuffer
  }

[giomurru]

Hi @v-grigoriev, thank you for your response, and apologies for the delayed reply — I believe I missed the notification.

I'm happy to share that I was able to resolve the issue. If you're interested in how I extended SCNRecorder to enable audio recording from an AVPlayer playing video, I’ve just pushed the changes to my fork.

You can compare the branches here:
master...giomurru:SCNRecorder:spherualizer

To use it in the client app, declare:

var audioEngineVideoTapper : AudioEngine.AVPlayerTapper!

Then, before calling startRecording, set it up like this:

audioEngineVideoTapper = AudioEngine.AVPlayerTapper(player: avPlayer)
sceneView.recorder?.useAudioEngine = true
audioEngineVideoTapper.recorder = sceneView.recorder

and on finishRecording you release it like this:

self.audioEngineVideoTapper.recorder = nil
self.audioEngineVideoTapper = nil

Let me know if you have any questions or feedback. Since the issue is now resolved, feel free to close it on your end. Thanks again for building and maintaining such a great framework—really appreciate your work!