unix: rewrite shebangs of Python scripts to reference local interpreter

Before, the shebangs referenced a path in the build environment, which
likely didn't exist in the run-time environment.

After this change, the shebang uses a multiline shell script to
call `exec` on the `python*` binary in the same directory as the
script.

The solution is similar to #61 but the technical approach is a bit
different. We perform the rewrite inside the build environment, so
this should work when building in Docker containers. And we use
binary I/O everywhere, ensuring bytes are preserved. We also always
exec the canonical `python*` binary to avoid symlink overhead.

Author: indygreg

cpython-unix/build-cpython.sh

@@ -933,6 +933,50 @@ if [ -n "${PYTHON_BINARY_SUFFIX}" ]; then
         ${ROOT}/out/python/install/bin/python${PYTHON_MAJMIN_VERSION}
 fi
 
+if [ ! -f ${ROOT}/out/python/install/bin/python3 ]; then
+    echo "python3 executable does not exist"
+    exit 1
+fi
+
+# Fixup shebangs in Python scripts to reference the local python interpreter.
+cat > ${ROOT}/fix_shebangs.py << EOF
+import os
+import sys
+
+ROOT = sys.argv[1]
+
+for f in sorted(os.listdir(ROOT)):
+    full = os.path.join(ROOT, f)
+
+    if os.path.islink(full) or not os.path.isfile(full):
+        continue
+
+    with open(full, "rb") as fh:
+        initial = fh.read(64)
+
+    if not initial.startswith(b"#!"):
+        continue
+
+    print("rewriting shebang in %s" % full)
+
+    lines = []
+
+    with open(full, "rb") as fh:
+        next(fh)
+
+        lines.extend([
+            b"#!/bin/sh\n",
+            b'"exec" "\$(dirname \$0)/python${PYTHON_MAJMIN_VERSION}${PYTHON_BINARY_SUFFIX}" "\$0" "\$@"\n',
+        ])
+
+        lines.extend(fh)
+
+    with open(full, "wb") as fh:
+        fh.write(b"".join(lines))
+EOF
+
+${BUILD_PYTHON} ${ROOT}/fix_shebangs.py ${ROOT}/out/python/install/bin
+
 # Also copy object files so they can be linked in a custom manner by
 # downstream consumers.
 for d in Modules Objects Parser Parser/pegen Programs Python; do

docs/quirks.rst

@@ -4,27 +4,6 @@
 Behavior Quirks
 ===============
 
-.. _quirk_shebangs:
-
-Bad Shebangs in Python Scripts
-==============================
-
-Various Python scripts under ``install/bin/`` (e.g. ``pip``) have
-shebangs looking like ``#!/build/out/python/install/bin/python3``.
-This ``/build/out/`` directory is where the distribution is built
-from. Python is writing out shebangs for Python scripts with
-that absolute path.
-
-To work around this issue, you can mass rewrite the shebangs to
-point the directory where the distribution is extracted/installed
-to. Here is a sample shell one-liner to get you started::
-
-   $ find install/bin/ -type f -exec sed -i '1 s/^#!.*python.*/#!.\/python3/' {} \;
-
-Alternatively, you can sometimes execute ``python3 -m <module>``
-to get equivalent functionality to what the installed script would
-do. e.g. to run pip, ``python3 -m pip ...``.
-
 .. _quirk_backspace_key:
 
 Backscape Key Doesn't work in Python REPL