iOS: No audio playback on first launch - AudioPlaybackStatusChanged handler is web-only

[vladt99]

Bug Description

On iOS, the first connection to an ElevenLabs agent produces no audible audio. The agent's speech data is received (visible in onDebug callbacks as audio_event with alignment data), but nothing plays through the speaker. Disconnecting and reconnecting fixes it - audio works on the second attempt and all subsequent attempts.

Root Cause

The AudioPlaybackStatusChanged handler in livekit_manager.dart (line 105) is effectively web-only:

  ..on<AudioPlaybackStatusChanged>((event) async {
    // Handle audio playback issues (especially for iOS)
    if (!_room!.canPlaybackAudio) {
      try {
        await _room!.startAudio();
      ...

On iOS:

• Room.canPlaybackAudio is initialized to true and never changes (it only tracks Web Audio Context state)
• Room.startAudio() is a no-op on native platforms - it only resumes HTML audio elements

So this handler never triggers on iOS, despite the comment saying "especially for iOS".

The actual iOS audio issue is that RTCAudioSession (WebRTC's audio session wrapper) starts in soloAmbient category, which doesn't support simultaneous playback + recording. LiveKit's automatic audio management (RemoteAudioManagementMixin) does eventually reconfigure to playAndRecord when tracks start, but there's a race condition on first launch: audio data flows before the reconfiguration takes effect.

On second launch, RTCAudioSession retains the playAndRecord state from the first attempt, so audio works immediately.

Suggested Fix

Pre-configure the iOS audio session in LiveKitManager.connect() before _room!.connect() (line 126):

  // Before connecting to LiveKit room
  if (Platform.isIOS) {
    await Native.configureAudio(
      NativeAudioConfiguration(
        appleAudioCategory: AppleAudioCategory.playAndRecord,
        appleAudioMode: AppleAudioMode.voiceChat,
        appleAudioCategoryOptions: {
          AppleAudioCategoryOption.defaultToSpeaker,
          AppleAudioCategoryOption.allowBluetooth,
          AppleAudioCategoryOption.allowBluetoothA2DP,
        },
        preferSpeakerOutput: true,
      ),
    );
  }

  await _room!.connect(serverUrl, token);

This uses the same RTCAudioSession code path as LiveKit's internal audio management, ensuring the session is ready before WebRTC starts processing audio.

Reproduction Steps

1. Fresh iOS app launch (kill app from recents first)
2. Connect to an ElevenLabs conversational agent
3. Agent speaks - no audio heard
4. Disconnect and reconnect
5. Agent speaks - audio works

Environment

• elevenlabs_agents: ^0.3.0
• livekit_client: 2.5.4 (transitive)
• iOS 17/18, physical device
• Flutter 3.x stable

Current Workaround

Call Native.configureAudio() from app code before startSession():

  import 'package:livekit_client/src/support/native.dart';
  import 'package:livekit_client/src/support/native_audio.dart';

  // Before calling client.startSession()
  await Native.configureAudio(
    NativeAudioConfiguration(
      appleAudioCategory: AppleAudioCategory.playAndRecord,
      appleAudioMode: AppleAudioMode.voiceChat,
      appleAudioCategoryOptions: {
        AppleAudioCategoryOption.defaultToSpeaker,
        AppleAudioCategoryOption.allowBluetooth,
        AppleAudioCategoryOption.allowBluetoothA2DP,
      },
      preferSpeakerOutput: true,
    ),
  );

This requires importing internal livekit_client files, hence the request for a proper fix in the SDK.

[kraenhansen]

Thanks for reporting this. I haven't yet reproduced this myself, but it seems doable from your thorough description 👍 Would it be possible for you to push a repo with a minimal reproduction as well?