fix: use graphlib in build-parallel core

[tiran]

Our custom dependency tracker for build-parallel code has a bug. It does not track and handle installation requirements of build requirements correctly. This causes build bugs when the installation requirement tree of a build dependency is deep.

This PR completely rewrites and replaces the core logic of build-parallel command with stdlib's graphlib module. The TopologicalSorter detects cycles and tracks when a new node becomes ready. Nodes with a dependency on an exclusive build node now become built-ready earlier.

I also changed the code to use a single instance of ThreadPoolExecutor. It's a tiny bit more efficient and saves one level of indention.

See: #755

[dhellmann]

This looks much easier to follow. I have one question about the difference in ordering inline.

[tiran]

I'm going to rewrite this PR on top of #763 once the PR is approved.

[dhellmann]

This needs to include the build requirements of the edge.destination_node, too.

• A needs B to build.
• B needs C to build.
• B needs D to install.

To build A you need B, C, and D.

[tiran]

That's the beauty of graphlib and my iter_build_requirements approach -- I can handle each package in the package set independently. There is no need to track build requirements of build requirements here.

• iter_build_requirements() give me a set of packages that must be installed in the build environment of a package.
• then I feed all packages and their build requirements into graphlib. TopologicalSorter.prepare freezes the graph and checks for cycles.
• each iteration over the graph returns nodes that have all their build requirements satisfied. The first round are self-bootstrapping packages like setuptools and flit. The second round everything that only needs setuptools or flit, and so on.

In your example:

• A.iter_build_requirements() returns B (direct build requirement) and D (via traverse_install_requirements(B.children)).
• B.iter_build_requirements() returns C (direct build requirement)
• C.iter_build_requirements() is empty
• D.iter_build_requirements() is empty

>>> topo = graphlib.TopologicalSorter()
>>> topo.add("A", "B", "D")
>>> topo.add("B", "C")
>>> topo.add("C")
>>> topo.add("D")
>>> topo.prepare()
>>> while topo.is_active():
...     nodes = topo.get_ready()
...     print(nodes)
...     topo.done(*nodes)
...     
('D', 'C')
('B',)
('A',)

[dhellmann]

OK, I don't understand how this code ensures that both build and installation requirements are included, though.

I see on line 141 that build requirements are skipped when iterating over installation requirements and I see on line 118 that installation requirements are skipped when iterating over build requirements. So as far as I can tell, only nodes where the edge has a type of build are included. That's the bug we have today, we aren't building everything we need in the right order.

This is the same issue we had in the bootstrap code where D needs to be treated as a build requirement of A. There we had the why chain to look at to realize that D was a build dependency, even though the edge type was install. Here we don't have that why chain.

Rather than being very specific with the algorithm, we can just build all of the dependencies of A, regardless of their type, before we build A. That will fix the issue AND give us simpler code, which is a big benefit in this logic since it's been generally buggy.

[tiran]

I have added additional comments and test examples. The line numbers do not match the code any more.

I think you are misunderstanding the code of iter_build_requirements. The method has two loops and two yields. The outer loop is a shallow iteration over the direct build requirements of the package. The inner loop is recursive, depth first algorithm that yields the installation requirements for each build requirement.

1. all direct build dependencies (packages in [build-system].requires)
2. the children of packages in (1) that are installation requirements
3. recursive dependencies of (2), depth first
4. ...

iter_build_requirements() returns the entire build dependency set -- all packages that have to be installed in the build environment of a package.

The DependencyGraph.get_build_topology method feeds all nodes + all their entire build dependency set into the topological sorter. On every loop, the topological sorter returns all nodes that have no unsatisfied dependencies. In other words, the new algorithm immediately starts to build a package when its entire build dependency set is satisfied.

The algorithm may build a package although one of its installation requirements is not satisfied, yet. That's okay, because we don't have to install packages immediately. The algorithm ensures that all packages are processes, because all packages are in the graph.

[tiran]

Closing in favor of #796