Skip to content

CI infra: Android tests fail with DEVICE_NOT_FOUND on 8/10 builds #3610

@mattleibow

Description

@mattleibow

Problem

Android tests fail on 8 out of 10 recent main builds with:

XHarness exit code: 81 (DEVICE_NOT_FOUND)
crit: Failed to find compatible device: x86_64

The XHarness runner retries 9 times over ~2 minutes, but no Android emulator is ever found on the CI agent.

Root Cause

The CI agent (Android (Linux) job) doesn't have a running Android emulator. XHarness requires an attached device or running emulator to deploy and run tests. This appears to be an infrastructure issue — the emulator either isn't being started, or the startup is racing with the test runner.

Impact

The Android test job fails on nearly every build, contributing to the perpetual red Tests stage. Since the tests never actually run (no device), there's zero test coverage for Android in CI currently.

Recommendations

  1. Verify emulator startup in CI pipeline — Check if the Azure DevOps pipeline has a step to start the Android emulator before the test task. If not, add one with appropriate wait time.

  2. Add emulator health check — Before running XHarness, run adb devices and wait for the emulator to appear:

    adb wait-for-device
    adb shell getprop sys.boot_completed | grep -q 1
  3. Increase XHarness retry timeout — The current 9 retries with ~10s delay (~90s total) may not be enough for emulator cold boot. Consider increasing to 5 minutes.

  4. Consider skipping if no device — If the emulator can't be reliably started, mark the Android test job as optional (allowed to fail) rather than blocking the entire build.

Data

8/10 recent main builds (156223–156430) have DEVICE_NOT_FOUND in the Android test logs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    New

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions