feat(py_venv): Replace untennable copying with symlinks (#644)

As reported by customers, the naive but correct strategy of using copies
in `py_venv_*` can lead to laughable disk usage. Some clients are
reporting order 10min slowdowns and order 100GiB disk usage wasted
copying inputs into binaries. We need a more scalable strategy such as
symlinking.

Thankfully we can generate symlinks from tools driven by Bazel into a
TreeArtifact so long as the symlinks aren't dangling. By carefully
crafting relative symlinks we're able to produce a tree of links which
is valid both at and after action time. When relocating a `.runfiles`
tree containing such links (for instance into a OCI later tar) these
links must be dereferenced but that Just Works.

While I'm at it, refactor the venv machinery to operate in terms of
strategies and combinators on strategies so that it's simpler to talk
about the production-grade behavior we want which is:

* `site-packages` trees in 1stparty code get relocated/linked into the
venv
* `bin` sibling trees in 1stparty code get relocated/patched into the
venv
* General trees in 1stparty code are referred to by `.pth` file entries
* General trees in 3rdparty code get relocated/linked into the venv
* `bin` sibling trees in 3rdparty code get relocated/patched into the
venv

This makes the venv builder significantly more flexible, allows for
better error reporting and opens the door to more flexible error
handling.

Incorporates an implementation of #606, but testing is required.
Should include an implementation of #635, but testing is required.

### Changes are visible to end-users: yes

- Searched for relevant documentation and updated as needed: yes
- Breaking change (forces users to change their own code or config): no
- Suggested release notes appear below: yes

`py_venv_*` now use symlinks rather than hard file copies which
radically reduce disk usage while improving venv building performance.

### Test plan

- Covered by existing test cases
- New test cases added
- Manual testing; please provide instructions so we can reproduce:
TODO.

### Remaining work
- [x] Strip debug prints
- [x] Improve collision handling
- [x] Rework the command interpreter to implement the last-wins
semantics
- [x] Mitigate spooky dangling symlink issues 
- [x] Fix a regression which can cause a `site-packages/__init__.py`
file to be linked
- [x] Add sha256-sum based collision ignoring
- [ ] Add a test covering that a `site-packages/__init__.py` file will
not be linked
- [ ] Add a test covering bin shebang patching
- [ ] Integrate the test case from #635
- [ ] Manually test that linked venvs still work; should just be fine

---------

Co-authored-by: Alexander Payne <[email protected]>

Author: arrdem

py/private/py_venv/py_venv.bzl

@@ -128,7 +128,7 @@ def _py_venv_base_impl(ctx):
             "--bin-dir=" + ctx.bin_dir.path,
             "--collision-strategy=" + ctx.attr.package_collisions,
             "--venv-name=" + venv_name,
-            "--mode=static-copy",
+            "--mode=" + ctx.attr.mode,
             "--version={}.{}".format(
                 py_toolchain.interpreter_version_info.major,
                 py_toolchain.interpreter_version_info.minor,
@@ -276,6 +276,15 @@ A collision can occur when multiple packages providing the same file are install
         default = "error",
         values = ["error", "warning", "ignore"],
     ),
+    "mode": attr.string(
+        doc = """The venv assembly mode.
+
+* "static-pth": Efficient. Just use a .pth file. Ignore binaries.
+* "static-symlink": Efficient. Use .pth entries for firstparty and symlinks for 3rdparty. Copies and patches binaries.
+        """,
+        default = "static-symlink",
+        values = ["static-pth", "static-symlink"],
+    ),
     "interpreter_options": attr.string_list(
         doc = "Additional options to pass to the Python interpreter.",
         default = [],

py/tests/py_venv_conflict/test_import_roots.py

@@ -48,13 +48,13 @@ def tree(dir_path: Path, prefix: str=''):
 print(sys.prefix)
 
 import conflict
-print(conflict.__file__)
+print("conflict.__file__", conflict.__file__)
 assert conflict.__file__.startswith(sys.prefix)
 
 import noconflict
-print(noconflict.__file__)
+print("noconflict.__file__", noconflict.__file__)
 assert noconflict.__file__.startswith(sys.prefix)
 
 import py_venv_conflict.lib as srclib
-print(srclib.__file__)
+print("srclib.__file__", srclib.__file__)
 assert not srclib.__file__.startswith(sys.prefix)

py/tests/py_venv_image_layer/my_app_amd64_layers_listing.yaml

@@ -2442,7 +2442,7 @@ files:
 layer: 1
 files:
   - drwxr-xr-x  0 0      0           0 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/
-  - -rwxr-xr-x  0 0      0         328 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/_aspect.pth
+  - -rwxr-xr-x  0 0      0         280 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/_aspect.pth
   - -rwxr-xr-x  0 0      0          19 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/_virtualenv.pth
   - -rwxr-xr-x  0 0      0        4342 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/_virtualenv.py
   - drwxr-xr-x  0 0      0           0 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/colorama-0.4.6.dist-info/

py/tests/py_venv_image_layer/my_app_arm64_layers_listing.yaml

@@ -2423,7 +2423,7 @@ files:
 layer: 1
 files:
   - drwxr-xr-x  0 0      0           0 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/
-  - -rwxr-xr-x  0 0      0         328 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/_aspect.pth
+  - -rwxr-xr-x  0 0      0         280 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/_aspect.pth
   - -rwxr-xr-x  0 0      0          19 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/_virtualenv.pth
   - -rwxr-xr-x  0 0      0        4342 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/_virtualenv.py
   - drwxr-xr-x  0 0      0           0 Jan  1  2023 ./py/tests/py_venv_image_layer/my_app_bin.runfiles/aspect_rules_py/py/tests/py_venv_image_layer/.my_app_bin/lib/python3.9/site-packages/colorama-0.4.6.dist-info/