write "Depends" into .deb and build for Python 3.10 on Ubuntu 22.04

Summary:
Although D39042765 (a04fc2e) appeared to successfully produce `.deb` files, it turns
out that they would only work if you already had the requisite package
dependencies installed. For example, running `hg --version` on a clean
Ubuntu instance would yield:

```
hg: error while loading shared libraries: libpython3.8.so.1.0: cannot open shared object file: No such file or directory
```

It turned out that our `DEBIAN/control` file was missing a key `Depends:` line
to tell `dpkg` what dependencies to install. Once again, I decided to look and
see how wezterm deals with this, and the solution appears to be `dpkg-shlibdeps`:

https://github.com/wez/wezterm/blob/97eaa58112a4/ci/deploy.sh#L234,L240

`dpkg-shlibdeps` looks at your executables and, based on the symbols, determines
what the appropriate packages are. For our Ubuntu 22.04 `.deb`, this turns out to be:

```
Depends: libc6 (>= 2.34), libgcc-s1 (>= 4.2), libpython3.10 (>= 3.10.0), libssl3 (>= 3.0.0~~alpha1), zlib1g (>= 1:1.1.4)
```

Apparently you cannot have a package in the `Depends:` line that comes from a PPA,
so if you want to make your `.deb` easy for users to install, that means you should
limit your dependencies to "standard" packages for the distro. In our case, that meant
that our Ubuntu 22.04 `.deb` should use Python 3.10 instead of Python 3.8 (which we
previously fetched from `ppa:deadsnakes/ppa`).

In order to support this, this diff makes a number of changes:

- Adds logic to `setup.py` to check the `PY_VERSION` environment variable to decide
  whether to use Python 3.10 or something else.
- Updates `gen_workflows.py` to define different Python deps based on the Ubuntu version.
- Updates `gen_workflows.py` to remove the logic that uses the `ppa:deadsnakes/ppa`.
- Ran `buck2 run //eden/oss/ci:gen_workflows -- eden/oss/.github/workflows/` to update the GitHub Actions
- Moved the logic for the `deb` target in the `Makefile` into a separate shell script because
  it was getting too complex to express directly in Make.
- Introduced separate `deb-ubuntu-20.04` and `deb-ubuntu-22.04` targets in the `Makefile`.
- Updated `bytearrayobject.rs` to support Python 3.10 in addition to Python 3.9.
- Updated `pick_python.py` to consider `python3.10` if `python3.8` is not found before going
  down the rest of the list.

As you can see in the Test Plan, we are *really close* to things "just working," but we also have
to register `git` as a dependency, which is not discovered by `dpkg-shlibdeps`. This will be
addressed in D39156794.

Reviewed By: DurhamG

Differential Revision: D39156794

fbshipit-source-id: ca1e0a73096e0de97230804a97f316114b8bfc2e

Author: bolinfest

ci/gen_workflows.py

@@ -22,15 +22,25 @@
 
 UBUNTU_VERSIONS = ["20.04", "22.04"]
 
+UBUNTU_VERSION_DEPS = {
+    "20.04": [
+        "python3.8",
+        "python3.8-dev",
+        "python3.8-distutils",
+    ],
+    "22.04": [
+        "python3.10",
+        "python3.10-dev",
+        "python3.10-distutils",
+    ],
+}
+
 UBUNTU_DEPS = [
     "nodejs",
-    "python3.8",
-    "python3.8-dev",
     "pkg-config",
     "libssl-dev",
     "cython3",
     "make",
-    "python3.8-distutils",
     "g++",
     "cargo",
     # This is needed for dpkg-name.
@@ -79,16 +89,7 @@ def gen_build_ubuntu_image(self, *, ubuntu_version: str) -> None:
                 + cargo_prefetch_commands
             )
 
-        python38_ppa = ""
-        if ubuntu_version == "22.04":
-            python38_ppa = """\
-RUN apt-get -y install software-properties-common
-RUN add-apt-repository -y ppa:deadsnakes/ppa
-RUN apt-get -y update
-"""
-        else:
-            python38_ppa = ""
-
+        full_deps = UBUNTU_DEPS + UBUNTU_VERSION_DEPS[ubuntu_version]
         dockerfile = f"""\
 FROM {image_name}
 
@@ -104,9 +105,9 @@ def gen_build_ubuntu_image(self, *, ubuntu_version: str) -> None:
 # Use a PPA to ensure a specific version of Node (the default Node on
 # Ubuntu 20.04 is v10, which is too old):
 RUN curl -fsSL https://deb.nodesource.com/setup_16.x | bash -
-{python38_ppa}
+
 # Now we can install the bulk of the packages:
-RUN apt-get -y install {' '.join(UBUNTU_DEPS)}
+RUN apt-get -y install {' '.join(full_deps)}
 
 # Copy the full repo over because `cargo fetch` follows deps within the repo,
 # so assume it needs everything.
@@ -256,7 +257,7 @@ def gen_build_ubuntu_cli_job(self, *, ubuntu_version: str, deb_version_expr: str
                 {
                     "name": "Create .deb",
                     "working-directory": "./eden/scm",
-                    "run": f"${{{{ format('VERSION=0.0-{{0}} make deb', env.{DEB_UPSTREAM_VERISION}) }}}}",
+                    "run": f"${{{{ format('VERSION=0.0-{{0}} make deb-ubuntu-{ubuntu_version}', env.{DEB_UPSTREAM_VERISION}) }}}}",
                 },
                 {
                     "name": "Rename .deb",

eden/scm/Makefile

@@ -107,11 +107,17 @@ build:
 
 deb:
 	DESTDIR=install make install-oss
-	mkdir -p "install/DEBIAN"
-	cp packaging/debian/control install/DEBIAN/control
-	sed -i "s/%VERSION%/$(VERSION)/g" install/DEBIAN/control
-	dpkg-deb --build --root-owner-group install
-	dpkg-name install.deb
+# Define the build script in a separate file so we don't have to escape shell
+# commands to work with Make inline.
+	./packaging/debian/build_deb.sh
+
+deb-ubuntu-20.04:
+	# For Ubuntu 20.04, target Python 3.8.
+	PY_VERSION=38 make deb
+
+deb-ubuntu-22.04:
+	# For Ubuntu 22.04, target Python 3.10.
+	PY_VERSION=310 make deb
 
 rpm:
 	rpmbuild \

eden/scm/contrib/pick_python.py

@@ -23,7 +23,7 @@ def main(args):
     names = args or ["python3"]
     if names == ["python3"]:
         # Try different pythons
-        names = ["python3.8", "python3.7", "python3.6"] + names
+        names = ["python3.8", "python3.10", "python3.7", "python3.6"] + names
     for name in names:
         for dir in dirs:
             path = os.path.join(dir, name)

eden/scm/lib/cpython-ext/src/bytearrayobject.rs

@@ -51,7 +51,7 @@ struct PyByteArrayObject {
 }
 
 #[cfg(feature = "python3")]
-#[cfg(Py_39)]
+#[cfg(any(Py_39, Py_310))]
 #[repr(C)]
 struct PyByteArrayObject {
     pub ob_base: PyVarObject,

eden/scm/packaging/debian/build_deb.sh

@@ -0,0 +1,24 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2.
+
+set -e
+
+# dpkg-shlibdeps requires the file `debian/control` to exist in the folder
+# in which it is run.
+mkdir -p install/debian
+cp packaging/debian/control install/debian
+cd install
+DEB_DEPS=$(dpkg-shlibdeps -O -e usr/local/bin/*)
+cd ..
+
+# In contrast to dpkg-shlibdeps, dpkg-deb requires the file to be named
+# `DEBIAN/control`, so we rename the directory and proceed.
+mv install/debian install/DEBIAN
+echo "$DEB_DEPS" | sed -e 's/shlibs:Depends=/Depends: /' >> install/DEBIAN/control
+sed -i "s/%VERSION%/$VERSION/g" install/DEBIAN/control
+
+dpkg-deb --build --root-owner-group install
+dpkg-name install.deb

eden/scm/packaging/debian/control

@@ -3,3 +3,4 @@ Version: %VERSION%
 Architecture: amd64
 Maintainer: Meta Platforms, Inc.
 Description: A source control client.
+Source: https://github.com/facebookexperimental/eden

eden/scm/setup.py

@@ -43,10 +43,12 @@
 if sys.version_info.major == 2:
     raise RuntimeError("This setup.py is Python 3 only!")
 
-if os.name == "nt":
-    PY_VERSION = "39"
-else:
-    PY_VERSION = "38"
+PY_VERSION = os.environ.get("PY_VERSION")
+if PY_VERSION is None:
+    if os.name == "nt":
+        PY_VERSION = "39"
+    else:
+        PY_VERSION = "38"
 
 ossbuild = bool(os.environ.get("SAPLING_OSS_BUILD"))