test_build_details Failing in Testbed

[johnzhou721]

Describe the bug

test_build_details fails in testbed using 3.14-b2.

Steps to reproduce

Unzip the support package and python3.14 testbed run -- test test_build_details -v

Expected behavior

It either passes or gets skipped.

Screenshots

No response

Environment

• Operating System:
• Python version:
• Software versions:
  • Briefcase:
  • Toga:
  • ...

Logs

(base) johnzhou@Johns-MBP Python-3.14-iOS-support.b2 % python3.14 testbed run -- test test_build_details -v
Updating plist... done.
Obtaining lock on simulator creation...
Simulator lock acquired.
Starting xcodebuild...
Writing result bundle at path:
	/Users/johnzhou/Downloads/Python-3.14-iOS-support.b2/testbed/20250524-210756.xcresult

--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:iOS Simulator, id:AAFDACDC-69F5-48B4-A523-4E0F2F416DEB, OS:18.3.1, name:iPhone SE (3rd generation) }
{ platform:iOS Simulator, id:AAFDACDC-69F5-48B4-A523-4E0F2F416DEB, OS:18.3.1, name:iPhone SE (3rd generation) }
2025-05-24 21:07:58.193 xcodebuild[13670:8697834]  DVTDeveloperAccountManager: Failed to load credentials for zhoushiyi2017@icloud.com: Error Domain=DVTDeveloperAccountCredentialsError Code=0 "Invalid credentials in keychain for zhoushiyi2017@icloud.com, missing Xcode-Token" UserInfo={NSLocalizedDescription=Invalid credentials in keychain for zhoushiyi2017@icloud.com, missing Xcode-Token}
2025-05-24 21:08:20: New test simulator detected
UDID: 4855D104-E90E-4A23-A1AF-04B517B7FD3F
Using Parallel Testing Device Clones Device Set: '/Users/johnzhou/Library/Developer/XCTestDevices'
getpwuid_r did not find a match for uid 501
Filtering the log data using "senderImagePath ENDSWITH "/iOSTestbedTests.xctest/iOSTestbedTests" OR senderImagePath ENDSWITH "/Python.framework/Python""
Timestamp               Ty Process[PID:TID]
=== Messages dropped during live streaming (use `log show` to see what they were)
2025-05-24 21:08:34.105 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) Test command: (
    test,
    "test_build_details",
    "-v"
)
2025-05-24 21:08:34.105 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) Configuring isolated Python...
2025-05-24 21:08:34.106 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) Pre-initializing Python runtime...
2025-05-24 21:08:34.107 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) PythonHome: /Users/johnzhou/Library/Developer/XCTestDevices/4855D104-E90E-4A23-A1AF-04B517B7FD3F/data/Containers/Bundle/Application/36D59CC7-7EDE-4CD3-9E19-42B184D51A08/iOSTestbed.app/python
2025-05-24 21:08:34.107 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) Configure argc/argv...
2025-05-24 21:08:34.107 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) Initializing Python runtime...
2025-05-24 21:08:34.131 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) App packages path: /Users/johnzhou/Library/Developer/XCTestDevices/4855D104-E90E-4A23-A1AF-04B517B7FD3F/data/Containers/Bundle/Application/36D59CC7-7EDE-4CD3-9E19-42B184D51A08/iOSTestbed.app/app_packages
2025-05-24 21:08:34.131 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) App path: /Users/johnzhou/Library/Developer/XCTestDevices/4855D104-E90E-4A23-A1AF-04B517B7FD3F/data/Containers/Bundle/Application/36D59CC7-7EDE-4CD3-9E19-42B184D51A08/iOSTestbed.app/app
2025-05-24 21:08:34.131 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) ---------------------------------------------------------------------------
== CPython 3.14.0a6 (main, Mar 19 2025, 07:04:19) [Clang 15.0.0 (clang-1500.3.9.4)]
== iOS-18.3.1-arm64-64bit little-endian
== Python build: release framework=Python
== cwd: /Users/johnzhou/Library/Developer/XCTestDevices/4855D104-E90E-4A23-A1AF-04B517B7FD3F/data/Containers/Data/Application/2BE405E5-C27D-4598-B581-EF71476FF2C0/tmp/test_python_worker_14986æ
== CPU count: 10
== encodings: locale=US-ASCII FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 285157725
0:00:00 load avg: 177.67 Run 1 test sequentially in a single process
0:00:00 load avg: 177.67 [1/1] test_build_details
test_base_interpreter (test.test_build_details.CPythonBuildDetailsTests.test_base_interpreter) ...
skipped 'Android and iOS run tests via a custom testbed method that changes sys.executable'
test_base_prefix (test.test_build_details.CPythonBuildDetailsTests.test_base_prefix) ...
ok
test_implementation (test.test_build_details.CPythonBuildDetailsTests.test_implementation) ...

  test_implementation (test.test_build_details.CPythonBuildDetailsTests.test_implementation) (part='_multiarch') ... FAIL
test_language_version (test.test_build_details.CPythonBuildDetailsTests.test_language_version) ...
ok
test_language_version_info (test.test_build_details.CPythonBuildDetailsTests.test_language_version_info) ...
ok
test_location (test.test_build_details.CPythonBuildDetailsTests.test_location) ...
ok
test_parse (test.test_build_details.CPythonBuildDetailsTests.test_parse) ...
ok
test_platform (test.test_build_details.CPythonBuildDetailsTests.test_platform) ...
FAIL
test_top_level_container (test.test_build_details.CPythonBuildDetailsTests.test_top_level_container) ...
ok

======================================================================
FAIL: test_implementation (test.test_build_details.CPythonBuildDetailsTests.test_implementation) (part='_multiarch')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/johnzhou/Library/Developer/XCTestDevices/4855D104-E90E-4A23-A1AF-04B517B7FD3F/data/Containers/Bundle/Application/36D59CC7-7EDE-4CD3-9E19-42B184D51A08/iOSTestbed.app/python/lib/python3.14/test/test_build_details.py", line 84, in test_implementation
    self.assertEqual(getattr(sys.implementation, key), value)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'arm64-iphonesimulator' != 'x86_64-iphonesimulator'
- arm64-iphonesimulator
? ^^^
+ x86_64-iphonesimulator
? ^^^^


======================================================================
FAIL: test_platform (test.test_build_details.CPythonBuildDetailsTests.test_platform)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/johnzhou/Library/Developer/XCTestDevices/4855D104-E90E-4A23-A1AF-04B517B7FD3F/data/Containers/Bundle/Application/36D59CC7-7EDE-4CD3-9E19-42B184D51A08/iOSTestbed.app/python/lib/python3.14/test/test_build_details.py", line 59, in test_platform
    self.assertEqual(self.key('platform'), sysconfig.get_platform())
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'ios-13.0-x86_64-iphonesimulator' != 'ios-13.0-arm64-iphonesimulator'
- ios-13.0-x86_64-iphonesimulator
?          ^^^^
+ ios-13.0-arm64-iphonesimulator
?          ^^^


----------------------------------------------------------------------
Ran 9 tests in 0.005s

FAILED (failures=2, skipped=1)
test test_build_details failed
0:00:00 load avg: 177.67 [1/1/1] test_build_details failed (2 failures)

== Tests result: FAILURE ==

1 test failed:
    test_build_details

Total duration: 59 ms
Total tests: run=9 failures=2 skipped=1
Total test files: run=1/1 failed=1
Result: FAILURE
2025-05-24 21:08:34.791 Df iOSTestbed[14986:84cbf4] (iOSTestbedTests) ---------------------------------------------------------------------------
2025-05-24 21:08:47.919 xcodebuild[13670:8697728] [MT] IDETestOperationsObserverDebug: 41.547 elapsed -- Testing started completed.
2025-05-24 21:08:47.919 xcodebuild[13670:8697728] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2025-05-24 21:08:47.919 xcodebuild[13670:8697728] [MT] IDETestOperationsObserverDebug: 41.547 sec, +41.547 sec -- end
Failing tests:
	-[iOSTestbedTests testPython]

** TEST FAILED **

Testing started
Test suite 'iOSTestbedTests' started on 'Clone 1 of iPhone SE (3rd generation) - iOSTestbed (14986)'
Test case '-[iOSTestbedTests testPython]' failed on 'Clone 1 of iPhone SE (3rd generation) - iOSTestbed (14986)' (0.687 seconds)
(base) johnzhou@Johns-MBP Python-3.14-iOS-support.b2 % 

Additional context

This is related to merging the archs for a stdlib in each SDK. It might make sense to skip this test, or patch the test to detect build-details.json at a different path. If build scripts like PAS are planned to be provided upstream, this will need to be done as well there.

[freakboy3742]

For anyone investigating this; we're now in a position to actually test_build_details as part of CI, by adding test_build_details to the cpython-testbed minimal testing set.

[freakboy3742]

This should be resolved by python/cpython#138176. The underlying issue is that build-details.json is platform specific, but doesn't include a platform specific identifier in the name, so we can't have 2 versions that reflect 2 different architectures in a fat build. The XCframework restructuring that occurs in python/cpython#138176 means this problem will no longer exist.