pythonbuild: support reading environment variables from a file

This enables you to define persisted environment variables between
builds. I'm now using this to make use of an S3 bucket for sccache
storage. This speeds up my local builds dramatically.

While I was here, I also documented how to configure sccache to
speed up builds because it is a game changer.

Author: indygreg

docs/building.rst

@@ -109,3 +109,40 @@ If building CPython 3.8+, you will need to specify the path to a
 
 To build a 32-bit x86 binary, simply use an ``x86 Native Tools
 Command Prompt`` instead of ``x64``.
+
+Using sccache to Speed up Builds
+================================
+
+Builds can take a long time. The bulk of the CPU time is bootstrapping a modern
+Clang toolchain, which requires building a modern GCC then potentially doing
+a 3 stage bootstrap of Clang.
+
+python-build-standalone can automatically detect and use the
+`sccache <https://github.com/mozilla/sccache>`_ compiler cache to speed
+up subsequent builds on UNIX-like platforms. ``sccache`` can shave dozens
+of minutes from fresh builds, even on a 16 core CPU!
+
+If there is an executable ``sccache`` in the source directory, it will
+automatically be copied into the build environment and used. For non-container
+builds, an ``sccache`` executable is also searched for on ``PATH``.
+
+The ``~/.python-build-standalone-env`` file is read if it exists (the format is
+``key=value`` pairs) and variables are added to the build environment.
+
+In addition, environment variables ``AWS_ACCESS_KEY_ID``,
+``AWS_SECRET_ACCESS_KEY``, and any variable beginning with ``SCCACHE_`` are
+automatically added to the build environment.
+
+The environment variable support enables you to define remote build caches
+(such as S3 buckets) to provide a persistent, shared cache across builds and
+machines.
+
+Keep in mind that when performing builds in containers in Linux (the default
+behavior), the local filesystem is local to the container and does not survive
+the build of a single package. So sccache is practically meaningless unless
+configured to use an external store (such as S3).
+
+When using remote stores (such as S3), ``sccache`` can be constrained on
+network I/O. We recommend having at least a 100mbps network connection to
+a remote store and employing a network store with as little latency as possible
+for best results.

pythonbuild/utils.py

@@ -391,13 +391,29 @@ def add_env_common(env):
     if "CI" in os.environ:
         env["CI"] = "1"
 
-    for k in (
-        # Proxy variables used for sccache remote cache.
-        "AWS_ACCESS_KEY_ID",
-        "AWS_SECRET_ACCESS_KEY",
-        "SCCACHE_BUCKET",
-        "SCCACHE_S3_USE_SSL",
-    ):
+    env_path = os.path.expanduser("~/.python-build-standalone-env")
+    try:
+        with open(env_path, "r") as fh:
+            for line in fh:
+                line = line.strip()
+                if line.startswith("#"):
+                    continue
+
+                key, value = line.split("=", 1)
+
+                print("adding %s from %s" % (key, env_path))
+                env[key] = value
+    except FileNotFoundError:
+        pass
+
+    # Proxy sccache settings.
+    for k, v in os.environ.items():
+        if k.startswith("SCCACHE_"):
+            env[k] = v
+
+    # Proxy cloud provider credentials variables to enable sccache to
+    # use stores in those providers.
+    for k in ("AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"):
         if k in os.environ:
             env[k] = os.environ[k]