Add android.py package command

Author: mhsmith

Android/README.md

@@ -1,19 +1,22 @@
 # Python for Android
 
-These instructions are only needed if you're planning to compile Python for
-Android yourself. Most users should *not* need to do this. Instead, use one of
-the tools listed in `Doc/using/android.rst`, which will provide a much easier
-experience.
+If you obtained this README as part of a release package, then the only
+applicable sections are "Prerequisites", "Testing", and "Using in your own app".
 
+If you obtained this README as part of the CPython source tree, then you can
+also follow the other sections to compile Python for Android yourself.
+
+However, most app developers should not need to do any of these things manually.
+Instead, use one of the tools listed
+[here](https://docs.python.org/3/using/android.html), which will provide a much
+easier experience.
 
-## Prerequisites
 
-First, make sure you have all the usual tools and libraries needed to build
-Python for your development machine.
+## Prerequisites
 
-Second, you'll need an Android SDK. If you already have the SDK installed,
-export the `ANDROID_HOME` environment variable to point at its location.
-Otherwise, here's how to install it:
+If you already have an Android SDK installed, export the `ANDROID_HOME`
+environment variable to point at its location. Otherwise, here's how to install
+it:
 
 * Download the "Command line tools" from <https://developer.android.com/studio>.
 * Create a directory `android-sdk/cmdline-tools`, and unzip the command line
@@ -33,9 +36,11 @@ The `android.py` script also requires the following commands to be on the `PATH`
 ## Building
 
 Python can be built for Android on any POSIX platform supported by the Android
-development tools, which currently means Linux or macOS. This involves doing a
-cross-build where you use a "build" Python (for your development machine) to
-help produce a "host" Python for Android.
+development tools, which currently means Linux or macOS.
+
+First we'll make a "build" Python (for your development machine), then use it to
+help produce a "host" Python for Android. So make sure you have all the usual
+tools and libraries needed to build Python for your development machine.
 
 The easiest way to do a build is to use the `android.py` script. You can either
 have it perform the entire build process from start to finish in one step, or
@@ -73,9 +78,24 @@ call. For example, if you want a pydebug build that also caches the results from
 ```
 
 
+## Packaging
+
+After building an architecture as described in the section above, you can
+package it for release with this command:
+
+```sh
+./android.py package HOST
+```
+
+`HOST` is defined in the section above.
+
+This will generate a tarball in the `dist` directory, whose structure is similar
+to the `Android` directory of the CPython source tree.
+
+
 ## Testing
 
-The test suite can be run on Linux, macOS, or Windows:
+The Python test suite can be run on Linux, macOS, or Windows:
 
 * On Linux, the emulator needs access to the KVM virtualization interface, and
   a DISPLAY environment variable pointing at an X server.
@@ -90,7 +110,7 @@ and find `hw.ramSize` in both config.ini and hardware-qemu.ini. Either set these
 manually to the same value, or use the Android Studio Device Manager, which will
 update both files.
 
-Before running the test suite, follow the instructions in the previous section
+Before running the test suite, follow the instructions in the section above
 to build the architecture you want to test. Then run the test script in one of
 the following modes:
 
@@ -133,4 +153,4 @@ until you re-run `android.py make-host` or `build`.
 
 ## Using in your own app
 
-See `Doc/using/android.rst`.
+See https://docs.python.org/3/using/android.html.

Android/android.py

@@ -2,7 +2,7 @@
 
 import asyncio
 import argparse
-from glob import glob
+import json
 import os
 import re
 import shlex
@@ -13,15 +13,17 @@
 import sysconfig
 from asyncio import wait_for
 from contextlib import asynccontextmanager
+from datetime import datetime, timezone
+from glob import glob
 from os.path import basename, relpath
 from pathlib import Path
 from subprocess import CalledProcessError
 from tempfile import TemporaryDirectory
 
 
 SCRIPT_NAME = Path(__file__).name
-CHECKOUT = Path(__file__).resolve().parent.parent
-ANDROID_DIR = CHECKOUT / "Android"
+ANDROID_DIR = Path(__file__).resolve().parent
+CHECKOUT = ANDROID_DIR.parent
 BUILD_DIR = ANDROID_DIR / "build"
 DIST_DIR = ANDROID_DIR / "dist"
 PREFIX_DIR = ANDROID_DIR / "prefix"
@@ -544,6 +546,67 @@ async def run_testbed(context):
         raise e.exceptions[0]
 
 
+def package_version(prefix_subdir):
+    vars_glob = f"{prefix_subdir}/lib/python*/_sysconfig_vars__android_*.json"
+    vars_paths = glob(vars_glob)
+    if len(vars_paths) != 1:
+        sys.exit(f"{vars_glob} matched {len(vars_paths)} paths.")
+    with open(vars_paths[0]) as vars_file:
+        version = json.load(vars_file)["py_version"]
+
+    # If not building against a tagged commit, add a timestamp to the version.
+    # Follow the PyPA version number rules, as this will make it easier to
+    # process with other tools.
+    if version.endswith("+"):
+        version += datetime.now(timezone.utc).strftime("%Y%m%d.%H%M%S")
+
+    return version
+
+
+def package(context):
+    prefix_subdir = subdir(PREFIX_DIR, context.host)
+    version = package_version(prefix_subdir)
+
+    with TemporaryDirectory(prefix=SCRIPT_NAME) as temp_dir:
+        temp_dir = Path(temp_dir)
+
+        # All tracked files in the Android directory.
+        for line in run(
+            ["git", "ls-files"],
+            cwd=ANDROID_DIR, capture_output=True, text=True, log=False,
+        ).stdout.splitlines():
+            src = ANDROID_DIR / line
+            dst = temp_dir / line
+            dst.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy2(src, dst, follow_symlinks=False)
+
+        # Anything in the prefix directory which could be useful either for
+        # apps embedding Python, or packages built against it.
+        for rel_dir, patterns in [
+            ("include", ["openssl*", "python*", "sqlite*"]),
+            ("lib", ["engines-3", "libcrypto*.so", "libpython*", "libsqlite*",
+                     "libssl*.so", "ossl-modules", "python*"]),
+            ("lib/pkgconfig", ["*crypto*", "*ssl*", "*python*", "*sqlite*"]),
+        ]:
+            for pattern in patterns:
+                for src in glob(f"{prefix_subdir}/{rel_dir}/{pattern}"):
+                    dst = temp_dir / relpath(src, ANDROID_DIR)
+                    dst.parent.mkdir(parents=True, exist_ok=True)
+                    if Path(src).is_dir():
+                        shutil.copytree(
+                            src, dst, symlinks=True,
+                            ignore=lambda *args: ["__pycache__"]
+                        )
+                    else:
+                        shutil.copy2(src, dst, follow_symlinks=False)
+
+        DIST_DIR.mkdir(exist_ok=True)
+        package_path = shutil.make_archive(
+            f"{DIST_DIR}/python-{version}-{context.host}", "gztar", temp_dir
+        )
+        print(f"Wrote {package_path}")
+
+
 # Handle SIGTERM the same way as SIGINT. This ensures that if we're terminated
 # by the buildbot worker, we'll make an attempt to clean up our subprocesses.
 def install_signal_handler():
@@ -556,6 +619,8 @@ def signal_handler(*args):
 def parse_args():
     parser = argparse.ArgumentParser()
     subcommands = parser.add_subparsers(dest="subcommand")
+
+    # Subcommands
     build = subcommands.add_parser("build", help="Build everything")
     configure_build = subcommands.add_parser("configure-build",
                                              help="Run `configure` for the "
@@ -567,25 +632,27 @@ def parse_args():
     make_host = subcommands.add_parser("make-host",
                                        help="Run `make` for Android")
     subcommands.add_parser(
-        "clean", help="Delete the cross-build directory")
+        "clean", help="Delete all build and prefix directories")
+    subcommands.add_parser(
+        "build-testbed", help="Build the testbed app")
+    test = subcommands.add_parser(
+        "test", help="Run the test suite")
+    package = subcommands.add_parser("package", help="Make a release package")
 
+    # Common arguments
     for subcommand in build, configure_build, configure_host:
         subcommand.add_argument(
             "--clean", action="store_true", default=False, dest="clean",
-            help="Delete any relevant directories before building")
-    for subcommand in build, configure_host, make_host:
+            help="Delete the relevant build and prefix directories first")
+    for subcommand in [build, configure_host, make_host, package]:
         subcommand.add_argument(
-            "host", metavar="HOST",
-            choices=["aarch64-linux-android", "x86_64-linux-android"],
+            "host", metavar="HOST", choices=HOSTS,
             help="Host triplet: choices=[%(choices)s]")
     for subcommand in build, configure_build, configure_host:
         subcommand.add_argument("args", nargs="*",
                                 help="Extra arguments to pass to `configure`")
 
-    subcommands.add_parser(
-        "build-testbed", help="Build the testbed app")
-    test = subcommands.add_parser(
-        "test", help="Run the test suite")
+    # Test arguments
     test.add_argument(
         "-v", "--verbose", action="count", default=0,
         help="Show Gradle output, and non-Python logcat messages. "
@@ -614,14 +681,17 @@ def main():
         stream.reconfigure(line_buffering=True)
 
     context = parse_args()
-    dispatch = {"configure-build": configure_build_python,
-                "make-build": make_build_python,
-                "configure-host": configure_host_python,
-                "make-host": make_host_python,
-                "build": build_all,
-                "clean": clean_all,
-                "build-testbed": build_testbed,
-                "test": run_testbed}
+    dispatch = {
+        "configure-build": configure_build_python,
+        "make-build": make_build_python,
+        "configure-host": configure_host_python,
+        "make-host": make_host_python,
+        "build": build_all,
+        "clean": clean_all,
+        "build-testbed": build_testbed,
+        "test": run_testbed,
+        "package": package,
+    }
 
     try:
         result = dispatch[context.subcommand](context)

Android/testbed/app/build.gradle.kts

@@ -6,8 +6,7 @@ plugins {
     id("org.jetbrains.kotlin.android")
 }
 
-val PYTHON_DIR = file("../../..").canonicalPath
-val ANDROID_DIR = "$PYTHON_DIR/Android"
+val ANDROID_DIR = file("../..").canonicalPath
 val PREFIX_DIR = "$ANDROID_DIR/prefix"
 
 val ABIS = mapOf(
@@ -147,8 +146,14 @@ androidComponents.onVariants { variant ->
             }
 
             into("lib/$pyPlusVer") {
-                // To aid debugging, the source directory takes priority.
-                from("$PYTHON_DIR/Lib")
+                // To aid debugging, the source directory takes priority when
+                // running inside a Python source tree.
+                if (
+                    file(ANDROID_DIR).name == "Android"
+                    && file("$ANDROID_DIR/../pyconfig.h.in").exists()
+                ) {
+                    from("$ANDROID_DIR/../Lib")
+                }
 
                 // The predix directory provides ABI-specific files such as
                 // sysconfigdata.

Doc/using/android.rst

@@ -27,9 +27,8 @@ details.
 Adding Python to an Android app
 -------------------------------
 
-These instructions are only needed if you're planning to compile Python for
-Android yourself. Most users should *not* need to do this. Instead, use one of
-the following tools, which will provide a much easier experience:
+Most app developers should use one of the following tools, which will provide a
+much easier experience:
 
 * `Briefcase <https://briefcase.readthedocs.io>`__, from the BeeWare project
 * `Buildozer <https://buildozer.readthedocs.io>`__, from the Kivy project
@@ -41,11 +40,12 @@ If you're sure you want to do all of this manually, read on. You can use the
 :source:`testbed app <Android/testbed>` as a guide; each step below contains a
 link to the relevant file.
 
-* Build Python by following the instructions in :source:`Android/README.md`.
+* Build Python by following the instructions in :source:`Android/README.md`,
+  or download a pre-built package from python.org.
 
 * Add code to your :source:`build.gradle <Android/testbed/app/build.gradle.kts>`
   file to copy the following items into your project. All except your own Python
-  code can be copied from ``cross-build/HOST/prefix/lib``:
+  code can be copied from ``prefix/HOST/lib``:
 
   * In your JNI libraries: