[android] Android.Device_Emulator.JIT.Test failing on emulators with CoreCLR

[ivanpovazan]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=953058
Build error leg or test failing: Android.Device_Emulator.JIT.Test
Found in: #112547

Error log

[21:40:21] dbug: ADBRunner using ADB.exe supplied from /datadisks/disk1/work/C3800A6B/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25103.1/tools/net8.0/any/../../../runtimes/any/native/adb/linux/adb
[21:40:21] dbug: Full resolved path:'/datadisks/disk1/work/C3800A6B/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25103.1/runtimes/any/native/adb/linux/adb'
[21:40:21] info: Will attempt to find device supporting architectures: 'x86_64'
[21:40:21] dbug: Executing command: '/datadisks/disk1/work/C3800A6B/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25103.1/runtimes/any/native/adb/linux/adb start-server'
[21:40:21] dbug: 
[21:40:21] info: Finding attached devices/emulators...
[21:40:21] dbug: Executing command: '/datadisks/disk1/work/C3800A6B/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25103.1/runtimes/any/native/adb/linux/adb devices -l'
[21:40:21] dbug: Found 1 possible devices
[21:40:21] dbug: Evaluating output line for device serial: emulator-5554          device product:sdk_phone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:1
[21:40:21] dbug: Executing command: '/datadisks/disk1/work/C3800A6B/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25103.1/runtimes/any/native/adb/linux/adb -s emulator-5554 shell getprop ro.product.cpu.abilist'
[21:40:21] fail: No attached device supports one of required architectures x86_64
[21:40:21] dbug: No suitable devices found
[21:40:21] crit: Failed to find compatible device: x86_64
[21:40:21] dbug: Saving diagnostics data to '/datadisks/disk1/work/C3800A6B/w/B81A09C5/e/diagnostics.json'
XHarness exit code: 81 (DEVICE_NOT_FOUND)
+ exit 81
+ export _commandExitCode=81
+ /usr/bin/python3 -u /datadisks/disk1/work/C3800A6B/w/B81A09C5/u/xharness-event-processor.py
Reporting 1 events from diagnostics file `/datadisks/disk1/work/C3800A6B/w/B81A09C5/e/diagnostics.json`
Analyzing android/test@None (81)
    Encountered DEVICE_NOT_FOUND
    If this occurs repeatedly, please check for architectural mismatch, e.g. sending arm64_v8a APKs to an x86_64 / x86 only queue

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "crit: Failed to find compatible device: x86_64",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=953058
Error message validated: [crit: Failed to find compatible device: x86_64]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/17/2025 9:53:47 AM UTC

Report

Build	Definition	Test	Pull Request
1142185	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#119464
1142088	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#119379
1141932	dotnet/runtime	Android.Device_Emulator.JIT.Static.Test.WorkItemExecution	#119456
1140942	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution	#119414
1140676	dotnet/runtime	System.Xml.Linq.SDMSample.Tests.WorkItemExecution
1140545	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#114969
1140486	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#117168
1140329	dotnet/runtime	Android.Device_Emulator.JIT.Static.Test.WorkItemExecution	#119388
1138715	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#119177
1138674	dotnet/runtime	System.Xml.Linq.Misc.Tests.WorkItemExecution
1137495	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1136374	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#119234
1135287	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#119160
1134960	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#119203
1134913	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#119201
1134051	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#119098
1133920	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#119138
1130721	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1129417	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118978
1129286	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118958
1128392	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#115766
1127687	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#118546
1127241	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118862
1127147	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#118914
1126342	dotnet/runtime	System.Private.Xml.Tests.WorkItemExecution	#118880
1125965	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118864
1125925	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#117072
1125691	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1124841	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
1124730	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118810
1123558	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118203
1123543	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#118720
1123449	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1123391	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118651
1123349	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#118729
1123312	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118718
1123274	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#116422
1123156	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118614
1123073	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118750
1123053	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118546
1122072	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http2.SslOptions_InvalidName_Throws	#118660
1122991	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118753
1122941	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118499
1122923	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#118750
1122918	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#106099
1122857	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118709
1122832	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118708
1122824	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#117182
1122798	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#118726
1122785	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118710
1122782	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#118715
1122699	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#118611
1122614	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118629
1122557	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118729
1122300	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpRequestErrorTest_Http20.SecureConnectionError	#114826
1122215	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http2.SslOptions_InvalidName_Throws	#118716
1122011	dotnet/runtime	System.Globalization.Tests.WorkItemExecution	#118546
1122018	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#118493
1121957	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#118156
1120356	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118634
1121949	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118611
1121962	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118707
1121914	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118699
1121896	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118413
1121879	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#118704
1121856	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution	#118691
1121781	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#117088
1121763	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118698
1121748	dotnet/runtime	System.Net.Mail.Tests.SmtpClientTlsTest_Send.AuthenticationException_Propagates	#118546
1121705	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#118694
1121682	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118413
1121651	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118691
1121639	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118687
1121492	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_SocketsHttpHandler_SecurityTest_Http2.SslOptions_InvalidName_Throws	#118684
1121468	dotnet/runtime	System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpRequestErrorTest_Http20.SecureConnectionError	#118050
1121435	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118682
1121238	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution
1121223	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#118640
1121204	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#118156
1121200	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118668
1121121	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118664
1120981	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#118657
1120922	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118301
1120929	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118418
1120857	dotnet/runtime	HardwareIntrinsics_General_ro.WorkItemExecution	#118611
1119790	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118418
1120683	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#116422
1120675	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#118413
1120598	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118493
1119768	dotnet/runtime	System.Runtime.InteropServices.Tests.WorkItemExecution	#118607
1120505	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118617
1120472	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118633
1120465	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118640
1120360	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#118514
1120286	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118633
1120265	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#117502
1120235	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#118629
1120208	dotnet/runtime	System.Net.Mail.Functional.Tests.WorkItemExecution	#118617
1119319	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#118301
1120146	dotnet/runtime	System.Net.Http.Functional.Tests.WorkItemExecution	#118628
Displaying 100 of 114 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	11	114

[ivanpovazan]

@kotlarmilos there is a failure on #112547 indicating that for CoreCLR on Android we use a queue which has only x86 emulators, causing the test to fail as we target x64 specifically.

Do we have queues with x64 emulators only that we could use?

[kotlarmilos]

I am not aware of a separate queue with x64-only emulators. This failure appears to be machine-specific. In successful runs, there are two emulators available, and one is selected. I assume one is x64 and the other is x86. Here is an example of a successful run: https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-112625-merge-5c5e4bed3d564f6eb6/Android.Device_Emulator.JIT.Test.Attempt.2/1/console.7eba4a2f.log?helixlogtype=result

Alternatively, we could restart this specific machine.

/cc: @matouskozak

[matouskozak]

I am not aware of a separate queue with x64-only emulators. This failure appears to be machine-specific. In successful runs, there are two emulators available, and one is selected. I assume one is x64 and the other is x86. Here is an example of a successful run: https://helixr1107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-112625-merge-5c5e4bed3d564f6eb6/Android.Device_Emulator.JIT.Test.Attempt.2/1/console.7eba4a2f.log?helixlogtype=result

I think queues with Android API <= 29 have both x64 and x86 simulators installed and it appears that XHarness is trying to connect to both even though we need just the x64 version. I think re-run would make the test pass.

Alternatively, we could restart this specific machine.

I think these are VMs but you can try asking in First Responders as it seems that this specific VM is on the top of the failures list (in the past 3 days).

[ivanpovazan]

I think queues with Android API <= 29 have both x64 and x86 simulators installed

Yes, I think that is the problem, or better to say that on this particular failure there aren't two emulators installed:

[21:40:21] dbug: Found 1 possible devices
[21:40:21] dbug: Evaluating output line for device serial: emulator-5554          device product:sdk_phone_x86 model:Android_SDK_built_for_x86 device:generic_x86 transport_id:1
[21:40:21] dbug: Executing command: '/datadisks/disk1/work/C3800A6B/p/microsoft.dotnet.xharness.cli/10.0.0-prerelease.25103.1/runtimes/any/native/adb/linux/adb -s emulator-5554 shell getprop ro.product.cpu.abilist'
[21:40:21] fail: No attached device supports one of required architectures x86_64

Via xharness we:

• first fetch available devices via:
  adb devices -l
• Then for each of them we check the supported architectures:
  adb -s <device-name> shell getprop ro.product.cpu.abilist
• If we find the match, we pick that emulator.
• If we don't find the match, we throw:
  https://github.com/dotnet/xharness/blob/32f83a8881e5142504a9089bac4aa98382a08f75/src/Microsoft.DotNet.XHarness.Android/AdbRunner.cs#L769

I think this host machine just does not have a x64 emulator installed and that is why it fails. The question is, if it is the only host with this problem.

[steveisok]

@ivanpovazan did anyone raise this with FR? They should be able to go into the machines and determine if it's all installed properly.

[ivanpovazan]

I updated the ErrorMessage pattern to: fail: No attached device supports one of required architectures x86_64 as the previous one was picking up cases when there are no devices at all.

[ivanpovazan]

The problematic machines have been deleted. I suggest monitoring this issue for some time, to see if we catch some other misbehaviors.