Skip to content

[Idea]: Symlink tree minimization for py_venv_* #646

@arrdem

Description

@arrdem

Brainfiring while working on #644.

Today the py_venv_* strategy is to create a link for every input file. This works nicely in general because it requires no real work planning and handles merging directory structures naturally.

Consider the inputs

@@foo_bar//site-packages/foo:__init__.py
@@foo_bar//site-packages/foo:bar.py
@@foo_baz//site-packages/foo:__init__.py
@@foo_baz//site-packages/foo:baz.py

In this case we have to generate a directory containing symlinks because we have files in the same destination directory which come from more than one source. But the more common case is stuff like

@@requests//site-packages/requests:__init__.py
@@clap//site-packages/clap:__init__.py

that is each external dependency provides a unique subtree and only rarely do they need to merge. In this second case we don't need to generate a link for each input file, we can instead create one link to the directory which contains only input files among which no merging is required.

This allows us to trade some compute (which is cheap) for I/O (which is not) and reduce the number of links we create in venv trees substantially.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions