gh-138171: Migrate iOS testbed location and add Apple build script #138176
Conversation
freakboy3742
requested review from
a team,
erlend-aasland,
AA-Turner and
corona10
as code owners
freakboy3742
added
OS-ios
needs backport to 3.13
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as outdated.
This comment was marked as outdated.
|
The buildbot initially failed because the buildbot configuration references the older |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
!buildbot iOS |
|
🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit 68d671b 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F138176%2Fmerge The command will test the builders whose names match following regular expression: The builders matched are:
|
|
@freakboy3742 I imagine the CODEOWNERS entries will need updating in this PR? |
.gitignore
Outdated
| Apple/testbed/Python.xcframework/*-*/bin | ||
| Apple/testbed/Python.xcframework/*-*/include | ||
| Apple/testbed/Python.xcframework/*-*/lib | ||
| Apple/testbed/Python.xcframework/*-*/Python.framework |
There was a problem hiding this comment.
Do we need *-* or would a simpler * pattern work?
There was a problem hiding this comment.
Hrm - you may be right. I used the *-* pattern because there are files in that folder that need to be version controlled (most notably, Info.plist), but when combined with the suffixes, they should be unique.
|
|
||
|
|
||
| def unpack_deps(host, prefix_dir, cache_dir=None): | ||
| deps_url = "https://github.com/beeware/cpython-apple-source-deps/releases/download" |
There was a problem hiding this comment.
Are there plans to move this under https://github.com/python/?
There was a problem hiding this comment.
Yes. @emmatyping started a pre-PEP discussion recently about consolidating the various repos we have into a single "binary dependency management repo", so that we have a single source for all (or as many as possible) of the build systems as possible. iOS and Android binaries (which are both currently BeeWare hosted) are part of those discussions.
If there's a more pressing interest for an interim solution, I'm happy to transfer ownership of both the iOS and Android repos to the Python org, or to give maintainer access to the existing repos any core team member who wants it.
Co-authored-by: Hugo van Kemenade <[email protected]>
|
I've just pushed an update that makes the final distribution size significantly better - it's now ~75MB compressed (down from ~140MB). This has been achieved by:
This means there is essentially no duplicated content in the distribution artefact; and we have a lot more liberty to modify XCframework structure over time, as we can update the build script in parallel with any structural changes in the XCframework. It also means we can move the dylib-Info-template.plist file into the XCframework as a build resource, rather than requiring the user to explicitly add it to their own project. I've updated the user documentation to describe how to use the build script, which significantly simplifies those usage instructions. |
There was a problem hiding this comment.
Thanks for addressing the previous comments. I built and tested this commit using python3.13 Apple build iOS and python3.13 Apple build test with the iOS 26 SDK on an Apple Silicon Mac running macOS Tahoe (26.0) and Xcode 26.0 as well as on an Intel Mac running macOS Sequoia (15.6.1) and Xcode (16.4). There were no Python test suite failures on either build.
freakboy3742
removed
needs backport to 3.13
|
After a discussion with @ambv, I think we don't need to backport this after all. The motivation to backport was so that we buildbots could use the new directory structure and build scripts. This is required because the buildbot can't currently differentiate PR builds based on the underlying Python version. However, if we fix that underlying problem, we don't need to backport, as we can retain "old style" builds for pre-3.15, and use new-style builds going forward. On that basis, I'll merge this to main, but not backport, and target adding iOS binaries for 3.15. |
|
!buildbot iOS |
|
🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit ebbbf32 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F138176%2Fmerge The command will test the builders whose names match following regular expression: The builders matched are:
|
|
!buildbot iOS |
|
🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit e0f6b5a 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F138176%2Fmerge The command will test the builders whose names match following regular expression: The builders matched are:
|
| "--timeout=-1", | ||
| # Adding Python options requires the use of a subprocess to | ||
| # start a new Python interpreter. | ||
| "--dont-add-python-opts", |
There was a problem hiding this comment.
--dont-add-python-opts may reduce the effectiveness of the tests: see #138805 (comment).
However, resolving this in the Android testbed required a fairly significant refactoring, so since this PR doesn't make the iOS situation any worse, it's probably best left to a separate PR.
There was a problem hiding this comment.
I'm not sure I agree it's reducing effectiveness. It's preventing the use of a code branch that would otherwise be invoked as a consequence of using --fast-ci or --slow-ci; if that problem code branch is invoked, the test suite crashes. Using --fast-ci/--slow-ci improves testing consistency; the options that aren't being processed as a result of using this flag weren't being used previously, either.
I agree that the Python options that are enabled by that code branch should be factored into the iOS testbed - but as you have flagged, that would requires a much more substantial refactor, and it can be handled independently.
There was a problem hiding this comment.
Sounds good. FYI, the reason why I made the Android testbed accept Python options on the command line, rather than hard-coding the options expected by the CPython test suite, is that some of those options might not be appropriate for third-party package tests run by cibuildwheel.
There was a problem hiding this comment.
Sure - that makes sense. I've got some ideas for how this could be managed in the context of the iOS testbed - we can read options out of the Info.plist file to separate test suite options from interpreter options.
There was a problem hiding this comment.
I found it simpler not to separate them at all, but to let PyConfig_SetBytesArgv handle both of them at once, just like in a normal Python executable.
|
|
Thanks @freakboy3742 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14. |
|
Sorry, @freakboy3742, I could not cleanly backport this to |
|
After a conversation with @hugovk, it might be worth back porting this to 3.14 so that it is in 3.14 final; even if we don't do a binary release until a 3.14.1 or later - or at all - having the directory renamed will make other maintenance and back porting easier. |
…ipt (python#138176) Adds tooling to generate and test an iOS XCframework, in a way that will also facilitate adding other XCframework targets for other Apple platforms (tvOS, watchOS, visionOS and even macOS, potentially). --------- Co-authored-by: Hugo van Kemenade <[email protected]> (cherry picked from commit 35c7e52)
|
GH-139204 is a backport of this pull request to the 3.14 branch. |
…ipt (python#138176) Adds tooling to generate and test an iOS XCframework, in a way that will also facilitate adding other XCframework targets for other Apple platforms (tvOS, watchOS, visionOS and even macOS, potentially). --------- Co-authored-by: Hugo van Kemenade <[email protected]>
…ript (#138176) (#139204) Co-authored-by: Hugo van Kemenade <[email protected]>
This script simplifies the process of configuring, compiling and packaging an XCframework for an Apple platform. This is an analog of the scripts used to build Android and Emscripten artefacts. It is a precursor to adding GitHub Action builds for iOS, and producing iOS build artefacts.
At present, it only supports iOS, but it has been constructed so that it could be used on any Apple platform (including, potentially, a redistributable macOS XCframework).
The simplest usage of this script is:
which will:
XCframework, merging binaries into a "fat" binary if necessary
This is the complete sequence that would be needed in CI to build and test a candidate release artefact. The output of the command will detect if it's running in a GitHub Actions and use output groups to make the log output easier to digest.
Each individual step can be invoked individually - there are commands to
clean,configure-build,make-build,configure-host,make-host,package, andtest.There is also a
buildcommand that can be used to combine the configure and make steps for the build Python, an individual host, all hosts, or all builds.One of the steps is to manage the download of binary artefacts; these artefacts are downloaded to the
cross-build/downloadsfolder by default, but an alternate cache location can be provided.There are three potentially controversial parts to this PR:
Moving
iOStoApple/iOSAs part of recognising that iOS is one of (potentially) many Apple platforms, and to prevent future proliferation of top-level directories for other Apple platforms, it migrates the iOS folder into a top level Apple folder. This change isn't strictly required, but if we don't make this change now, there is potential for a proliferation of directories if/when we add support for tvOS, watchOS, visionOS, macCatalyst. It also provides a convenient home for "cross platform" apple concerns, like the testbed script and the build script itself, as well as place for the existing content in the
Macdirectory to migrate to as part of a cleanup of macOS builds - something that @ned-deily has expressed an interest in doing 1Universal simulator binaries
This PR generates a build that includes x86_64 simulator binaries. The iOS simulator slice of the XCframework is effectively a "universal" build covering both x86_64 and ARM64; although it has to be compiled in 2 passes and binaries merged after the fact. It's possible to both compile and test x86_64 binaries on ARM64 machines (pass
--simulator "iPhone 16e,arch=x86_64"to theciortesttarget, or to the testbed script), although running the test suite in emulator mode currently causes some test failure related tobuild-details.jsonnot being fully cross-platform aware.x86_64 isn't a Tier 3 platform for Python due to the difficulties of commissioning a buildbot; but x86_64 is still a supported platform for Apple. If we're going to produce official binaries, it seems to me like we should still support x86_64.
If we choose not to support x86_64, then there will be some changes required to the build script and testbed, as the name of the simulator slice will change from
ios_arm64_x86_64_simulatortoios_arm64_simulator. It would also allow removing the most complicated part of this script - the part that does the merging of binaries.Backporting to 3.13 and 3.14
I've proposed this for backport to 3.13 and 3.14. This is primarily so that the buildbots can be migrated to use this script, and CI jobs can be added for iOS. The directory reorganization is a significant change to make in a backport, especially in the year-old 3.13 release; however, when I floated this idea at the CPython core team summit, @Yhg1s suggested 1 that while it is a big change, the set of affected iOS users would be small, so the impact wasn't a huge concern.
NOTE: This PR currently contains a duplicate of the
iOS/Resources/bindirectory; this is needed because the location is hard-coded into the buildbots. Once the location change is confirmed, I can add the new path to the buildbot configuration, and remove the duplicate bin scripts.📚 Documentation preview 📚: https://cpython-previews--138176.org.readthedocs.build/
Footnotes
https://pyfound.blogspot.com/2025/06/python-language-summit-2025-python-on-mobile.html ↩ ↩2