Google Meet audio nearly inaudible when recording

[tleyden]

When I try to record mic audio with cidre while in a Google Meet meeting, both the incoming and outgoing audio become barely audible. The audio I hear from other participants through my speakers drops to near-silence, and other participants can barely hear me.

This only happens while the cidre recording is active. When I stop the recording, both audio streams return to normal volume. It happens regardless if I set output.vp_set_bypass_voice_processing(true)?; or not.

Cidre code below. Any tips or ideas would be appreciated!

System

• macbook air m4 running tahoe

Steps to repro

1. Join a google meet in chrome, with audio coming into built-in mic and coming out of built-in speakers.
2. Run cli app that calls cidre to record (code below)

Actual: the audio coming from google meet becomes barely audible. It's both ways too, the other participants can barely hear me.
Expected: the audio from google meet shouldn't be affected

Cidre code

#[cfg(target_os = "macos")]
use cidre::{
    arc,
    at::{
        self, au,
        audio::component::{InitializedState, UninitializedState},
    },
    av, ns, os,
};
use log::{error, info};

#[cfg(target_os = "macos")]
pub struct CidreRecorder {
    context: Option<Box<RecordingContext>>,
}

#[cfg(target_os = "macos")]
struct RecordingContext {
    file: arc::R<av::AudioFile>,
    format: arc::R<av::AudioFormat>,
    output: Option<au::Output<InitializedState>>,
    data: Vec<f32>,
}

#[cfg(target_os = "macos")]
impl Drop for RecordingContext {
    fn drop(&mut self) {
        if let Some(mut output) = self.output.take() {
            if let Err(e) = output.stop() {
                error!("Failed to stop audio output: {:?}", e);
            }
        }
        self.file.close();
        info!("RecordingContext dropped");
    }
}

#[cfg(target_os = "macos")]
impl RecordingContext {
    fn new(file: arc::R<av::AudioFile>, format: arc::R<av::AudioFormat>) -> Self {
        Self {
            file,
            format,
            output: None,
            data: Vec::new(),
        }
    }

    fn start(
        &mut self,
        mut output: au::Output<UninitializedState>,
        use_voiceprocessor_io: bool,
    ) -> os::Result<()> {
        info!(
            "Setting up audio output configuration (use_voiceprocessor_io: {})",
            use_voiceprocessor_io
        );
        output.set_io_enabled(au::Scope::INPUT, 1, true)?;
        output.set_io_enabled(au::Scope::OUTPUT, 0, false)?;
        output.set_should_allocate_input_buf(false)?;
        output.set_should_allocate_output_buf(false)?;

        if !use_voiceprocessor_io {
            info!("Bypassing voice processing");
            output.vp_set_bypass_voice_processing(true)?;
        }

        output.set_input_cb(RecordingContext::input_cb, self as *mut Self)?;

        info!("Allocating audio resources");
        let output = output.allocate_resources()?;
        let max_frames = output.unit().max_frames_per_slice()? as usize;
        info!("Max frames per slice: {}", max_frames);
        self.data = vec![0f32; max_frames];
        self.output = Some(output);

        let output = unsafe { self.output.as_mut().unwrap_unchecked() };
        info!("Starting audio output");
        output.start()
    }

    extern "C-unwind" fn input_cb(
        ctx: *mut RecordingContext,
        _io_action_flags: &mut au::RenderActionFlags,
        _in_timestamp: &at::AudioTimeStamp,
        _in_bus_num: u32,
        in_number_frames: u32,
        _io_data: *mut at::AudioBufList<1>,
    ) -> os::Status {
        if ctx.is_null() {
            error!("input_cb called with null context");
            return au::err::NO_CONNECTION.into();
        }
        let ctx = unsafe { &mut *ctx };

        if ctx.output.is_none() {
            error!("input_cb called with no output");
            return au::err::NO_CONNECTION.into();
        }

        let output = unsafe { ctx.output.as_mut().unwrap_unchecked() };

        let mut buf_list = at::AudioBufList::<1>::new();
        buf_list.buffers[0] = at::AudioBuf {
            number_channels: 1,
            data_bytes_size: std::mem::size_of_val(&ctx.data[..]) as u32,
            data: ctx.data.as_mut_ptr() as *mut _,
        };

        if let Err(e) = output.render(in_number_frames, &mut buf_list, 1) {
            error!("Failed to render audio: {:?}", e);
            return e.status();
        }

        let buf = match av::AudioPcmBuf::with_buf_list_no_copy(&ctx.format, &buf_list, None) {
            Some(buf) => buf,
            None => {
                error!("Failed to create PCM buffer");
                return au::err::INVALID_PARAM.into();
            }
        };

        if let Err(e) = ctx.file.write(&buf) {
            error!("Failed to write audio to file: {:?}", e);
            return au::err::INVALID_PARAM.into();
        }

        os::Status::NO_ERR
    }
}

#[cfg(target_os = "macos")]
impl CidreRecorder {
    pub fn new() -> Self {
        info!("Creating new CidreRecorder");
        Self { context: None }
    }

    pub fn start_recording(
        &mut self,
        output_path: &str,
        use_voiceprocessor_io: bool,
    ) -> Result<(), String> {
        info!(
            "Starting recording to: {} (use_voiceprocessor_io: {})",
            output_path, use_voiceprocessor_io
        );

        let output = au::Output::new_apple_vp().map_err(|e| {
            error!("Failed to create audio output: {:?}", e);
            format!("Failed to create audio output: {:?}", e)
        })?;

        let input_device = output.input_device().map_err(|e| {
            error!("Failed to get input device: {:?}", e);
            format!("Failed to get input device: {:?}", e)
        })?;

        let asbd = output.input_stream_format(1).map_err(|e| {
            error!("Failed to get input stream format: {:?}", e);
            format!("Failed to get input stream format: {:?}", e)
        })?;

        info!("Input stream format: {:?}", asbd);

        let format = av::AudioFormat::with_asbd(&asbd).ok_or_else(|| {
            error!("Failed to create audio format");
            "Failed to create audio format".to_string()
        })?;

        let device_name = input_device
            .name()
            .map_err(|e| format!("Failed to get device name: {:?}", e))?;
        info!("Input device: {}", device_name);

        let url = ns::Url::with_fs_path_str(output_path, false);
        info!("Opening file for writing: {:?}", url);

        let file = av::AudioFile::open_write_common_format(
            &url,
            &format.settings(),
            av::audio::CommonFormat::PcmF32,
            format.is_interleaved(),
        )
        .map_err(|e| {
            error!("Failed to open audio file for writing: {:?}", e);
            format!("Failed to open audio file for writing: {:?}", e)
        })?;

        info!("Audio file opened successfully");

        let mut ctx = Box::new(RecordingContext::new(file, format));
        ctx.start(output, use_voiceprocessor_io).map_err(|e| {
            error!("Failed to start recording context: {:?}", e);
            format!("Failed to start recording context: {:?}", e)
        })?;

        self.context = Some(ctx);
        info!("Recording started successfully");
        Ok(())
    }

    pub fn stop_recording(&mut self) -> Result<(), String> {
        info!("Stopping recording");
        if let Some(_ctx) = self.context.take() {
            info!("Recording stopped successfully");
            Ok(())
        } else {
            error!("No active recording to stop");
            Err("No active recording to stop".to_string())
        }
    }

    pub fn is_recording(&self) -> bool {
        self.context.is_some()
    }
}

#[cfg(not(target_os = "macos"))]
pub struct CidreRecorder;

#[cfg(not(target_os = "macos"))]
impl CidreRecorder {
    pub fn new() -> Self {
        Self
    }

    pub fn start_recording(&mut self, _output_path: &str) -> Result<(), String> {
        Err("Audio recording is only supported on macOS".to_string())
    }

    pub fn stop_recording(&mut self) -> Result<(), String> {
        Err("Audio recording is only supported on macOS".to_string())
    }

    pub fn is_recording(&self) -> bool {
        false
    }
}

[yury]

Thank you for your feedback.

I added vp_set_other_audio_ducking_cfg method to control ducking others audio. In my tests it helps a little bit. Please give it a try.

I updated example too.

[tleyden]

Oh great, thank you I will test it in the next few days! Do you know why it automatically does the audio "ducking"?

I thought maybe it was two echo cancellations fighting each other.

[yury]

I'm not an expert here either—just trying to learn as I go! 😊

VoiceIO enables audio ducking quite aggressively by default (as far as I understand).
For two-way voice chats, it's better to set au::VoiceIoOtherAudioDuckingCfg::enable_advanced_ducking to true. This prevents it from ducking audio coming from other outputs (though it will definitely use more CPU).

There's a very helpful WWDC session on VoiceIO ducking: Manage Audio Ducking with VoiceIO.

[tleyden]

I tried it, but no luck. Didn't seem to make any difference.

    fn start(
        &mut self,
        mut output: au::Output<UninitializedState>,
        use_voiceprocessor_io: bool,
    ) -> os::Result<()> {
        info!(
            "Setting up audio output configuration (use_voiceprocessor_io: {})",
            use_voiceprocessor_io
        );
        output.set_io_enabled(au::Scope::INPUT, 1, true)?;
        output.set_io_enabled(au::Scope::OUTPUT, 0, false)?;
        output.set_should_allocate_input_buf(false)?;
        output.set_should_allocate_output_buf(false)?;

        info!("Configuring audio ducking with minimum level");
        let ducking_cfg = au::VoiceIoOtherAudioDuckingCfg {
            enable_advanced_ducking: true,
            ducking_level: au::voice_io_other_audio_ducking_level::MIN,
        };
        info!(
            "Applying audio ducking config: enable_advanced_ducking={}, ducking_level={} (MIN=10)",
            ducking_cfg.enable_advanced_ducking, ducking_cfg.ducking_level
        );
        output.vp_set_other_audio_ducking_cfg(&ducking_cfg)?;
        info!("Audio ducking configured successfully");

(with cidre 0.11.8)

Thanks for the tips and the video link!

[yury]

Turns out there are private api for ducking.

Chrome use it too.

I added AudioDeviceDuck api.

If you don't plan to pass Apple Review process - use "private" feature. Otherwise Device::duck will use dynamic symbols to find that function and call it. I also updated example.

[wangfu91]

Actual: the audio coming from google meet becomes barely audible. It's both ways too, the other participants can barely hear me.

@tleyden I encountered a similar problem in one of my audio projects when using the VPIO(Voice Processing IO) for echo cancellation.
For other apps that are using the microphone simultaneously, the audio input dropped to an extremely low level.
It turns out this is a bug specific to Apple and only occurs when using the built-in microphone of the MacBook.

mozilla/cubeb-coreaudio-rs#239 (comment)

As you have found, this is an Apple issue with the VoiceProcessingIO audio unit. If the builtin mic is hooked up to VPIO its volume will be very low in any concurrent non-VPIO uses. Even in other processes. Try simultaneously using the builtin mic in Safari and Chrome or System Settings.
This seems true for at least those pretty modern Macbook Pros with a mic array under the hood.