--depth option for building only direct deps of foundational packages

[lf-]

When making certain kinds of low-level change, it's desirable to build the subset of the package set which is most likely to be affected by a given change, namely the direct dependencies. Specific example: I want to test a change to pytest's runtime behaviour, which only affects direct deps.

matrix discussion: https://matrix.to/#/!lymvtcwDJ7ZA9Npq:lix.systems/$I7aXJMSoLSrQPcrM1c_LwAnnCF5QZCyVjic0GIy5CQE?via=lix.systems&via=matrix.org&via=catgirl.cloud

cc @emilazy

[SuperSandro2000]

How should that work when fetching the changed packages from the GitHub action?
For local eval this could probably be somehow hacked together but I do not have a concrete plan for that from top of my head.

[pbsds]

I've toyed with the idea of inspecting and filtering all derivations to see if the drvPath or outPath of the packages mentioned in the commit titles appear in their list of inputs. This can be repeated depth number of times. The major cost to this is evaluating/realizing all the .drv files. Alternatively it could be implemented as a nix function, or in c using nix as a library, or using a tool like https://github.com/tweag/nixtract.

[emilazy]

I have a simple and elegant design for this feature which this margin is too small to contain :)

If anyone is interested enough in this feature to try implementing it, given such a design, then I can try to allocate time to explain it on this issue.

[wolfgangwalther]

If anyone is interested enough in this feature to try implementing it, given such a design, then I can try to allocate time to explain it on this issue.

Certainly, eventually. My take on it: It's all about finding the entrypoint. Aka "directly changed packages". Once you know these, you can find packages that directly depend on these etc.

Finding the entrypoint needs to happen by matching up "touched files" in a PR with packages. I have experimented with generating vastly improved meta.position data, but that's still heavily WIP. I think it should be possible, though. Better position data with static analysis of the code to include imported files. This would have many benefits, from much more accurate reviewer requests, to allow more merges via nixpkgs-merge-bot - and also issues like these, to only build a subset of changed packages.

What's your idea about?

[emilazy]

My idea only looks at .drv files, which I think is better because it is simpler and is the way nixpkgs-review only works. The relevant .drvs to consider are those that are not present in the “before” evaluation – which is already how nixpkgs-review computes changed packages. Let’s call this the changed set. Then we have two possible definitions of which ones of those are at --depth=1: ones that do not reference any .drvs in the total changed set, or ones that have a change other than switching to reference any .drv from the changed set.

The difference between these two approaches is that the latter will also catch the case where a .drv depends on a directly‐changed package but also had a direct change itself. The other difference is that the latter requires correlating .drvs with previous versions. This is probably easy in practice (filter by name etc.), but I’d be tempted to go with the former definition for simplicity. This is also where you can pull in the source files if you want to, because you can match up .drvs from the same attrpath. (Still, package renames are not uncommon and break these heuristics, so I’d personally lean towards doing the simple thing.)

Then for each step you just modulo out the .drv paths from the previous depth and do the same thing, until eventually the limit --depth=inf (the default) covers the total changed set.

In practice the useful targets are “build everything I directly touched”, “build everything that directly depends on anything I directly touched” (e.g. for libraries that are widely indirectly used), “build everything that directly depends on anything that directly depends on anything I directly touched” (e.g. changing things that result in changes to the development outputs of libraries, like tools to create pkg-config files or rewrite dynamic libraries), and “build everything”. Those correspond roughly to --depth=1, --depth=2, --depth=3, and --depth=inf, but if you don’t do the .drv correlation stuff then you may need to bump up the depth or have a way to explicitly specify packages to “pull in” to a previous depth.

[wolfgangwalther]

My idea only looks at .drv files

The problem with that approach of looking only at .drv files without narrowing down the search space with an explicit entrypoint beforehand: These .drv files are not available after default Eval. You would need to run a full Eval run that also creates all the .drv files. That is going to be painfully slow, though. A full Eval run with nix-eval-jobs on 32 cores, 128 GB RAM... will take ~20 minutes for that, IIRC. Just because there's a lot of IO going on.

We have discussed this a lot when trying to figure out how to implement the fod-oracle into Nixpkgs CI.

Although... we did not try to put the nix store for that evaluation on a tmpfs, I think. I'm not sure how much memory it would need, though. It's probably too much for GHA.

That being said.. even when knowing an entrypoint, it will probably only work with a patched Nix - one that can selectively write .drv files to disk when they are direct dependencies of that entrypoint.

[emilazy]

You only have to materialize the ones whose hashes differ from the base, i.e. the ones that default nixpkgs-review would end up building anyway.

[Mic92]

Guix also actually implements this feature for their q&a tool. It should be fairly straight-forward to implement this in nix without having to write any derivations. I have been thinking of a read-only mode in nix-eval-jobs that works more like nix-env - there are a lot of use-cases for a tool like this. However some better heuristics are required to make nix-eval-jobs not consume all available the memory by having smarter defaults.