Skip to content

Commit f2b9275

Browse files
FeodorFitsnerFeodorFitsner
authored
Packaging v2 (#111)
* Upgraded Android plugin to Python 3.12 * Format exception * Return formatted exception * Some cleanup * Moved to getPythonError() * Package Python for arm64-v8a * Fix yaml * Fix rsync command * Add build_script section * Manually push artifacts * Build Python for Android for all ABIs * Do not strip libpythonbundle.so * build.gradble changes - Download distros for all ABIs * Do not build Python for Android x86 Flutter doesn't support x86 * Pack/unpack site-packages * Packaging python for iOS - v1 * install_name_tool * Try real packaging for iOS * Fix yaml * Fix call to package-python-for-ios.sh * Fix rsync * Use build and dist dirs * Fix android packaging * Add _sysconfigdata__*.py for all platforms * Make self-contained and reusable create_xcframework_from_dylibs * Use full Python version 3.12.3 to package pythons * Save a bit * Fix site-packages * prepare macos script * iOS works! * Project changes * Update serious_python_darwin.podspec * Build Python for macOS * Emscripten * Build python with new repos * Fix iOS dist * Implement simple PyPI server in Dart * Re-build pythons * Added supported tags for pip install * iOS project can be compiled with Python * site-packages processing works for iOS * Remove packaging scripts * Re-worked package command - darft * Calculating app archive hash * Run on macos-sonoma * Shutdown Pyodide PyPI server * Re-enable cache * Update junk list * remove bin * Add python xcframeworks into Flutter app bundle * Make the script idempotent, cleanup. * iOS packaging works! * Fix for Android * Improve archive unpacking performance/effectiveness * iOS: update site-packages on every build * Leave "bin" folder on for desktop while packaging * macOS support * Do not delete *_dist folders * Merge macOS site packages * Use extra index on Gemfury * Flet example updated * Migrated to Python 3.12 * Download stripped python for build purposes * Linux migrated to python 3.12 * Update links to python-build repo * Try test iOS * Fixed SERIOUS_PYTHON_SITE_PACKAGES * Test on Android * Fix packaging command for Android * Test on Windows, Linux and macOS * Test macOS on Ventura * Test macOS only * iOS and macOS tests on ventura * Run "Test on Linux ARM64" * Fix gstreamer on arm * Fix again * Install ninja-build on arm * Run macOS test * Version bumped to 0.8.0 * Unpack app to the same directory to avoid disk pollution * Unpack app into support directory * Enable all CI jobs * Simplify iOS test * Update examples * Update macOS example * Updated readmes * Update CHANGELOG.md
1 parent ba361bc commit f2b9275

File tree

82 files changed

+10794
-9601
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

82 files changed

+10794
-9601
lines changed

‎.appveyor.yml‎

Lines changed: 25 additions & 165 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,19 @@
11
skip_branch_with_pr: true
22

33
environment:
4-
PYTHON_STACK: python 3.10
54
GITHUB_TOKEN:
65
secure: 9SKIwc3VSfYJ5IChvNR74mEv2nb0ZFftUzn3sGRdXipXEfKSxY50DoodChHvlqZduQNhjg0oyLWAAa3n+iwWvVM2yI7Cgb14lFNClijz/kHI/PibnjDMNvLKaAygcfAc
76

87
matrix:
9-
- job_name: Build Python for iOS
10-
job_group: build_python_darwin
11-
APPVEYOR_BUILD_WORKER_IMAGE: macos-monterey
12-
13-
- job_name: Build Python for Android
14-
job_group: build_python_android
15-
APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu2004
16-
178
- job_name: Test on macOS
189
job_group: test_serious_python
1910
job_depends_on: build_python_darwin
20-
APPVEYOR_BUILD_WORKER_IMAGE: macos-monterey
11+
APPVEYOR_BUILD_WORKER_IMAGE: macos-ventura
2112

22-
# - job_name: Test on iOS
23-
# job_group: test_serious_python
24-
# job_depends_on: build_python_darwin
25-
# APPVEYOR_BUILD_WORKER_IMAGE: macos-sonoma
13+
- job_name: Test on iOS
14+
job_group: test_serious_python
15+
job_depends_on: build_python_darwin
16+
APPVEYOR_BUILD_WORKER_IMAGE: macos-ventura
2617

2718
- job_name: Test on Android
2819
job_group: test_serious_python
@@ -39,130 +30,17 @@ environment:
3930

4031
- job_name: Test on Linux ARM64
4132
job_group: test_serious_python
42-
APPVEYOR_BUILD_WORKER_IMAGE: ubuntu2004-arm
33+
APPVEYOR_BUILD_WORKER_IMAGE: ubuntu2204-arm
4334

4435
- job_name: Publish serious_python package to pub.dev
4536
job_group: publish_package
4637
job_depends_on: build_python, test_serious_python
4738
APPVEYOR_BUILD_WORKER_IMAGE: Ubuntu2004
4839

4940
stack:
50-
- $PYTHON_STACK
41+
- python 3.12
5142

5243
for:
53-
# ======================================
54-
# Build Python for iOS
55-
# ======================================
56-
57-
- matrix:
58-
only:
59-
- job_name: Build Python for iOS
60-
61-
install:
62-
# update build version
63-
- ps: |
64-
if ($env:APPVEYOR_REPO_TAG_NAME) {
65-
$v = $env:APPVEYOR_REPO_TAG_NAME.replace("v", "")
66-
} else {
67-
$cv = [version](git describe --abbrev=0).substring(1)
68-
$v = "$($cv.major).$($cv.minor+1).0+$($env:APPVEYOR_BUILD_NUMBER)"
69-
}
70-
Update-AppveyorBuild -Version $v
71-
72-
# install Kivy toolchain
73-
- pip3 list
74-
- pip3 install git+https://github.com/flet-dev/python-for-ios.git
75-
- HOMEBREW_NO_AUTO_UPDATE=1 brew install autoconf automake libtool pkg-config
76-
- brew link libtool
77-
78-
build_script:
79-
# build Python 3
80-
- toolchain build python3
81-
- ls dist
82-
83-
# package dist
84-
- DIST_FILE_NAME=dist/python-ios-dist-v$APPVEYOR_BUILD_VERSION.tar.gz
85-
- tar -czvf $DIST_FILE_NAME dist/*
86-
- appveyor PushArtifact $DIST_FILE_NAME -DeploymentName python-dist-macos
87-
88-
deploy:
89-
provider: GitHub
90-
auth_token: $(GITHUB_TOKEN)
91-
release: $(APPVEYOR_REPO_TAG_NAME)
92-
artifact: python-dist-macos
93-
on:
94-
APPVEYOR_REPO_TAG: true
95-
96-
test: off
97-
98-
# ======================================
99-
# Build Python for Android
100-
# ======================================
101-
102-
- matrix:
103-
only:
104-
- job_name: Build Python for Android
105-
106-
install:
107-
# update build version
108-
- ps: |
109-
if ($env:APPVEYOR_REPO_TAG_NAME) {
110-
$v = $env:APPVEYOR_REPO_TAG_NAME.replace("v", "")
111-
} else {
112-
$cv = [version](git describe --abbrev=0).substring(1)
113-
$v = "$($cv.major).$($cv.minor+1).0+$($env:APPVEYOR_BUILD_NUMBER)"
114-
}
115-
Update-AppveyorBuild -Version $v
116-
117-
# install NDK
118-
- export ANDROID_SDK_ROOT="/usr/lib/android-sdk"
119-
- export NDK_VERSION=25.2.9519653
120-
- export SDK_VERSION=android-33
121-
- echo "y" | sdkmanager --install "ndk;$NDK_VERSION" --channel=3 > /dev/null
122-
- echo "y" | sdkmanager --install "platforms;$SDK_VERSION" > /dev/null
123-
124-
# install Kivy for Android
125-
- pip3 install git+https://github.com/flet-dev/[email protected]
126-
- pip3 install --upgrade cython
127-
- p4a --help
128-
- p4a create --requirements python3 --arch arm64-v8a --arch armeabi-v7a --arch x86_64 --sdk-dir $ANDROID_SDK_ROOT --ndk-dir $ANDROID_SDK_ROOT/ndk/$NDK_VERSION --dist-name serious_python
129-
130-
# package
131-
- BUNDLE_NAME=libpythonbundle.so
132-
133-
# arm64-v8a
134-
- cd ~/.local/share/python-for-android/dists/serious_python/_python_bundle__arm64-v8a/_python_bundle
135-
- zip -r $BUNDLE_NAME .
136-
- mv $BUNDLE_NAME ../../libs/arm64-v8a
137-
- cd $APPVEYOR_BUILD_FOLDER
138-
139-
# armeabi-v7a
140-
- cd ~/.local/share/python-for-android/dists/serious_python/_python_bundle__armeabi-v7a/_python_bundle
141-
- zip -r $BUNDLE_NAME .
142-
- mv $BUNDLE_NAME ../../libs/armeabi-v7a
143-
- cd $APPVEYOR_BUILD_FOLDER
144-
145-
# armeabi-v7a
146-
- cd ~/.local/share/python-for-android/dists/serious_python/_python_bundle__x86_64/_python_bundle
147-
- zip -r $BUNDLE_NAME .
148-
- mv $BUNDLE_NAME ../../libs/x86_64
149-
- cd $APPVEYOR_BUILD_FOLDER
150-
151-
# package all .so files
152-
- DIST_FILE_NAME=python-android-dist-v$APPVEYOR_BUILD_VERSION.tar.gz
153-
- cd ~/.local/share/python-for-android/dists/serious_python/libs
154-
- tar -czvf $DIST_FILE_NAME *
155-
- appveyor PushArtifact $DIST_FILE_NAME -DeploymentName python-dist-android
156-
- cd $APPVEYOR_BUILD_FOLDER
157-
158-
deploy:
159-
provider: GitHub
160-
auth_token: $(GITHUB_TOKEN)
161-
release: $(APPVEYOR_REPO_TAG_NAME)
162-
artifact: python-dist-android
163-
on:
164-
APPVEYOR_REPO_TAG: true
165-
16644
# ======================================
16745
# Test on macOS
16846
# ======================================
@@ -177,18 +55,11 @@ for:
17755
- flutter config --enable-macos-desktop
17856
- flutter doctor
17957

180-
# download dist for non-releases
181-
- |
182-
if [[ "$APPVEYOR_REPO_TAG_NAME" == "" ]]; then
183-
python3 ci/download_artifact.py "Build Python for iOS" "python-ios-dist-v{version}.tar.gz"
184-
export SERIOUS_PYTHON_IOS_DIST=$APPVEYOR_BUILD_FOLDER/python_dist/dist
185-
fi
186-
18758
build: off
18859

18960
test_script:
19061
- cd src/serious_python/example/flet_example
191-
- dart run serious_python:main package app/src --dep-mappings "flet=flet-embed" --req-deps "flet-embed"
62+
- dart run serious_python:main package app/src -p Darwin -r flet,--pre
19263
- flutter test integration_test -d macos
19364

19465
# ======================================
@@ -203,28 +74,23 @@ for:
20374
- HOMEBREW_NO_AUTO_UPDATE=1 brew install cocoapods
20475
- flutter upgrade
20576
- flutter config --enable-macos-desktop
206-
- xcrun simctl list runtimes
207-
- xcrun simctl create "e2e test" "iPhone 12" "com.apple.CoreSimulator.SimRuntime.iOS-17-2"
208-
- xcrun xctrace list devices
209-
- |
210-
UDID=$(xcrun xctrace list devices | grep "^e2e test Simulator (17.2)" | awk '{gsub(/[()]/,""); print $NF}')
211-
echo $UDID
212-
xcrun simctl boot "${UDID:?No Simulator with this name found}"
77+
# - xcrun simctl list runtimes
78+
# - xcrun simctl create "e2e test" "iPhone 12" "com.apple.CoreSimulator.SimRuntime.iOS-17-2"
79+
# - xcrun xctrace list devices
80+
# - |
81+
# UDID=$(xcrun xctrace list devices | grep "^e2e test Simulator (17.2)" | awk '{gsub(/[()]/,""); print $NF}')
82+
# echo $UDID
83+
# xcrun simctl boot "${UDID:?No Simulator with this name found}"
21384
#- flutter doctor -v
21485

215-
# download dist for non-releases
216-
- |
217-
if [[ "$APPVEYOR_REPO_TAG_NAME" == "" ]]; then
218-
python3 ci/download_artifact.py "Build Python for iOS" "python-ios-dist-v{version}.tar.gz"
219-
export SERIOUS_PYTHON_IOS_DIST=$APPVEYOR_BUILD_FOLDER/python_dist/dist
220-
fi
221-
22286
build: off
22387

22488
test_script:
89+
- export SERIOUS_PYTHON_SITE_PACKAGES=$APPVEYOR_BUILD_FOLDER/site-packages
22590
- cd src/serious_python/example/flet_example
226-
- dart run serious_python:main package app/src --mobile --dep-mappings "flet=flet-embed" --req-deps "flet-embed"
227-
- flutter drive --driver=test_driver/integration_test.dart --target=integration_test/app_test.dart
91+
- dart run serious_python:main package app/src -p iOS -r flet,--pre
92+
- flutter build ios --no-codesign
93+
# - flutter drive --driver=test_driver/integration_test.dart --target=integration_test/app_test.dart
22894

22995
# ======================================
23096
# Test on Android
@@ -254,18 +120,12 @@ for:
254120
- flutter upgrade --force
255121
- flutter doctor -v
256122

257-
# download dist for non-releases
258-
- |
259-
if [[ "$APPVEYOR_REPO_TAG_NAME" == "" ]]; then
260-
python3 ci/download_artifact.py "Build Python for Android" "python-android-dist-v{version}.tar.gz"
261-
export SERIOUS_PYTHON_BUILD_DIST=$APPVEYOR_BUILD_FOLDER/python_dist
262-
fi
263-
264123
build: off
265124

266125
test_script:
126+
- export SERIOUS_PYTHON_SITE_PACKAGES=$APPVEYOR_BUILD_FOLDER/site-packages
267127
- cd src/serious_python/example/flet_example
268-
- dart run serious_python:main package app/src --mobile --dep-mappings "flet=flet-embed" --req-deps "flet-embed"
128+
- dart run serious_python:main package app/src -p Android -r flet,--pre
269129
- flutter test integration_test -d emulator-5554
270130

271131

@@ -287,7 +147,7 @@ for:
287147

288148
test_script:
289149
- cd src/serious_python/example/flet_example
290-
- dart run serious_python:main package app/src --dep-mappings "flet=flet-embed" --req-deps "flet-embed"
150+
- dart run serious_python:main package app/src -p Windows -r flet,--pre
291151
- flutter test integration_test -d windows
292152

293153
# ======================================
@@ -307,7 +167,7 @@ for:
307167

308168
test_script:
309169
- cd src/serious_python/example/flet_example
310-
- dart run serious_python:main package app/src --dep-mappings "flet=flet-embed" --req-deps "flet-embed"
170+
- dart run serious_python:main package app/src -p Linux -r flet,--pre
311171
- xvfb-run flutter test integration_test -d linux
312172

313173
# ======================================
@@ -321,7 +181,7 @@ for:
321181
install:
322182
# Flutter SDK
323183
- sudo apt update --allow-releaseinfo-change
324-
- sudo apt install -y clang xvfb libgtk-3-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio
184+
- sudo apt install -y clang ninja-build xvfb libgtk-3-dev gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav
325185
- git clone https://github.com/flutter/flutter.git -b stable "$HOME/flutter"
326186
- export PATH="$PATH:$HOME/flutter/bin"
327187
- flutter upgrade
@@ -331,7 +191,7 @@ for:
331191

332192
test_script:
333193
- cd src/serious_python/example/flet_example
334-
- dart run serious_python:main package app/src --dep-mappings "flet=flet-embed" --req-deps "flet-embed"
194+
- dart run serious_python:main package app/src -p Linux -r flet,--pre
335195
- xvfb-run flutter test integration_test -d linux
336196

337197
# =========================================

‎docs/python-ios.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# Python for iOS notes
2+
3+
Source: https://github.com/python/cpython/blob/main/Doc/library/importlib.rst
4+
5+
A specialization of :class:`importlib.machinery.ExtensionFileLoader` that
6+
is able to load extension modules in Framework format.
7+
8+
For compatibility with the iOS App Store, *all* binary modules in an iOS app
9+
must be dynamic libraries, contained in a framework with appropriate
10+
metadata, stored in the ``Frameworks`` folder of the packaged app. There can
11+
be only a single binary per framework, and there can be no executable binary
12+
material outside the Frameworks folder.
13+
14+
To accomodate this requirement, when running on iOS, extension module
15+
binaries are *not* packaged as ``.so`` files on ``sys.path``, but as
16+
individual standalone frameworks. To discover those frameworks, this loader
17+
is be registered against the ``.fwork`` file extension, with a ``.fwork``
18+
file acting as a placeholder in the original location of the binary on
19+
``sys.path``. The ``.fwork`` file contains the path of the actual binary in
20+
the ``Frameworks`` folder, relative to the app bundle. To allow for
21+
resolving a framework-packaged binary back to the original location, the
22+
framework is expected to contain a ``.origin`` file that contains the
23+
location of the ``.fwork`` file, relative to the app bundle.
24+
25+
For example, consider the case of an import ``from foo.bar import _whiz``,
26+
where ``_whiz`` is implemented with the binary module
27+
``sources/foo/bar/_whiz.abi3.so``, with ``sources`` being the location
28+
registered on ``sys.path``, relative to the application bundle. This module
29+
*must* be distributed as
30+
``Frameworks/foo.bar._whiz.framework/foo.bar._whiz`` (creating the framework
31+
name from the full import path of the module), with an ``Info.plist`` file
32+
in the ``.framework`` directory identifying the binary as a framework. The
33+
``foo.bar._whiz`` module would be represented in the original location with
34+
a ``sources/foo/bar/_whiz.abi3.fwork`` marker file, containing the path
35+
``Frameworks/foo.bar._whiz/foo.bar._whiz``. The framework would also contain
36+
``Frameworks/foo.bar._whiz.framework/foo.bar._whiz.origin``, containing the
37+
path to the ``.fwork`` file.
38+
39+
When a module is loaded with this loader, the ``__file__`` for the module
40+
will report as the location of the ``.fwork`` file. This allows code to use
41+
the ``__file__`` of a module as an anchor for file system traveral.
42+
However, the spec origin will reference the location of the *actual* binary
43+
in the ``.framework`` folder.
44+
45+
The Xcode project building the app is responsible for converting any ``.so``
46+
files from wherever they exist in the ``PYTHONPATH`` into frameworks in the
47+
``Frameworks`` folder (including stripping extensions from the module file,
48+
the addition of framework metadata, and signing the resulting framework),
49+
and creating the ``.fwork`` and ``.origin`` files. This will usually be done
50+
with a build step in the Xcode project; see the iOS documentation for
51+
details on how to construct this build step.

‎src/serious_python/CHANGELOG.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,17 @@
1+
## 0.8.0
2+
3+
* Not based on Kivy!
4+
* Fast packaging uses pre-built Python binary packages hosted on https://pypi.flet.dev and https://pypi.org. If a binary package for specific platform/arch is not found the packaging process does not make an attempt to compile it, but just exits with a meaningful error.
5+
* To package for iOS and Android developer should set `SERIOUS_PYTHON_SITE_PACKAGES` environment variable with a path to a temp directory for installed app packages. The contents of that directory is embedded into app bundle during app compilation. For macOS, Linux and Windows app packages are installed into `__pypackages__` inside app package asset zip.
6+
* Packaging command is not looking for `requirements.txt` or `pyproject.toml` anymore, but all requirements should be passed explicitly via `--requirements` option. The value of `--requirements` option is passed "as is" to `pip` command. For example, `--requirements flet,numpy==2.1.1` install two requirements directly, or `--requirements -r,requirements.txt` installs deps from a file.
7+
* MacOS packaging includes Python binaries for both `arm64` and `x86_64` architectures. Can limit to only one architecture with `--arch` option.
8+
* New options to enable compilation and cleanup of app and packages .py files: `--compile-app`, `--compile-packages` and `--cleanup`.
9+
* Packaging for `web` is no longer relied on a HTML document with links, but spawns its own PyPI-compatible server with links to Pyodide packages.
10+
* Build python distributive is cached in Flutter's `build` directory, not temp, to avoid re-downloading on consequent re-packages.
11+
* Web builds updated to Pyodide 0.26.2.
12+
* Packages for iOS and Android are built with [Mobile Forge](https://github.com/flet-dev/mobile-forge).
13+
* Python for all platforms is built with [flet-dev/python-build](https://github.com/flet-dev/python-build/) and [this CI job](https://ci.appveyor.com/project/flet-dev/python-build). Python distros for Dart and Mobile Forge uploaded to [releases](https://github.com/flet-dev/python-build/releases).
14+
115
## 0.7.1
216

317
* Added `namespace` definition to Android Gradle build.

0 commit comments

Comments
 (0)