Skip to content

[WIP] Multiple by-name directory support #180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mdaniels5757 wants to merge 21 commits into
base: main
Choose a base branch
from
Draft

[WIP] Multiple by-name directory support #180

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
2b5ceae
Tests pass, but that's all that works currently.
mdaniels5757 Oct 17, 2025
54d5acc
Some changes, also run fmt
mdaniels5757 Oct 18, 2025
4054e34
now broken in a different way
mdaniels5757 Oct 18, 2025
50655ee
tests are passing again now
mdaniels5757 Oct 19, 2025
d3c3cb9
clippy + fmt
mdaniels5757 Oct 19, 2025
0efc5bf
use fixtures crate to put all tests into cargo output
mdaniels5757 Oct 19, 2025
1c66e08
use env var for config
mdaniels5757 Oct 20, 2025
d228d28
Works?
mdaniels5757 Oct 21, 2025
ce852a1
move concurrency to each by-name-dir check
mdaniels5757 Oct 23, 2025
572e8b7
doc config
mdaniels5757 Oct 23, 2025
7757cc2
combine multiple evaluations into one (well, two)
mdaniels5757 Oct 23, 2025
945f246
misc work, display only the correct by-name directory when erroring
mdaniels5757 Oct 24, 2025
8cc9474
fix: use a wildcard that doesn't match nested attrs
mdaniels5757 Oct 25, 2025
c9a9a35
tests now pass again, including my new ones!
mdaniels5757 Oct 25, 2025
0450bb5
don't perform extra logging
mdaniels5757 Oct 25, 2025
7e0fa3f
treefmt
mdaniels5757 Oct 31, 2025
b131c34
enable python by-name dir
mdaniels5757 Oct 31, 2025
bba094b
seperate test dirs by attrset
mdaniels5757 Oct 31, 2025
54891e7
fix CI build
mdaniels5757 Oct 31, 2025
3438417
clippy
mdaniels5757 Nov 1, 2025
0d147d5
Copied all tests and modified them to work with tcl. All pass!
mdaniels5757 Nov 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 4 additions & 2 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,5 +50,7 @@ jobs:
- uses: serokell/xrefcheck-action@v1
with: # Invalid symlinks for testing purposes.
xrefcheck-args: >
--ignore tests/symlink-invalid/main/pkgs/by-name/fo/foo/foo
--ignore tests/multiple-failures/main/pkgs/by-name/A/fo@/foo
--ignore tests/top-level/symlink-invalid/main/pkgs/by-name/fo/foo/foo
--ignore tests/top-level/multiple-failures/main/pkgs/by-name/A/fo@/foo
--ignore tests/tcl/multiple-failures/main/pkgs/development/tcl-modules/by-name/A/fo@/foo
--ignore tests/tcl/symlink-invalid/main/pkgs/development/tcl-modules/by-name/fo/foo/foo
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
target
.direnv
result*
by-name-config-generated.json
168 changes: 168 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ textwrap = "0.16.2"
derive-enum-from-into = "0.2.1"
derive-new = "0.7.0"
derive_more = { version = "2.0.1", features = ["display"] }
readonly = "0.2.13"

[dev-dependencies]
fixtures = "2.4.0"
pretty_assertions = "1.4.1"
temp-env = "0.3.6"
37 changes: 37 additions & 0 deletions by-name-config.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Configure by-name directories in this file.
# Then build with `nix-build -A build` to automatically generate by-name-config-generated.json
# (Or, to generate manually, `nix-instantiate --eval --json --strict by-name-config.nix > by-name-config-generated.json`)
# In the attrsets that make up by_name_dirs:
# * The aliases_path field is optional.
# * The ID field must be short and unique.
# * Exactly one attr_path_regex must be a wildcard ("^[^\\.]*$").
# * All non-wildcard attr_path_regexes must be mutually exclusive.
# * Exactly one unversioned_attr_prefix must be the empty string.
# * All non-wildcard unversioned_attr_prefixes must be mutually exclusive.
{
by_name_dirs = [
{
id = "py";
path = "pkgs/development/python-modules/by-name";
attr_path_regex = "^(python3\\d*Packages|python3\\d*.pkgs)\\..*$";
Copy link
Member

@infinisil infinisil Nov 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The biggest issue with how it's done currently closely relates to this here. As far as I understand, the new eval.nix/eval.rs does a recursive Nixpkgs evaluation, lists all resulting attribute paths and then checks which ones match the regex defined here.

This complicates the eval.nix because it needs to handle all the random failing evaluations in Nixpkgs, making it very brittle and slow.

Instead, please change this configuration options to be like suggested in #31:

attr_path = [ "python3Packages" ];

Notably we don't need to check all the different version sets like python311Packages, etc., because the only reason we need to evaluate anything at all is for the eval-time checks and ratchet checks. Since all the python3*Packages sets only differ in very minor ways, checking one of them is good enough. This will be much more efficient and simplify the code a lot.

On the implementation side, the code then only needs to evaluate the attributes directly under that attribute path, just like the eval.nix in the main branch (I'm afraid a lot of your eval.nix changes won't be needed 😅) . I also recommend changing what eval.nix outputs to be more suited to the new functionality, e.g. something like

{
  py = [ [ name value ] ... ];
  main = [ [ name value ] ... ];
  tcl = [ [ name value ] ... ];
}

This should make the diff to what it was fairly small.

unversioned_attr_prefix = "python3Packages";
all_packages_path = "/pkgs/top-level/python-packages.nix";
aliases_path = "/pkgs/top-level/python-aliases.nix";
}
{
id = "tcl";
path = "pkgs/development/tcl-modules/by-name";
attr_path_regex = "^(tcl\\d*Packages)\\..*$";
unversioned_attr_prefix = "tclPackages";
all_packages_path = "/pkgs/top-level/tcl-packages.nix";
}
{
id = "main";
path = "pkgs/by-name";
attr_path_regex = "^[^\\.]*$";
unversioned_attr_prefix = "";
all_packages_path = "/pkgs/top-level/all-packages.nix";
aliases_path = "/pkgs/top-level/aliases.nix";
}
];
}
11 changes: 10 additions & 1 deletion nixpkgs-check.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
runCommand,
nixpkgs-vet,
initNix,
writeText,
nixpkgs,
nix,
nixVersions,
Expand All @@ -20,6 +21,10 @@ let
eval.success && lib.isDerivation eval.value
) attrset;

configFile = writeText "by-name-config-generated.json" (
builtins.toJSON (import ./by-name-config.nix)
);

mkNixpkgsCheck =
name: nix:
runCommand "test-nixpkgs-vet-with-${nix.name}"
Expand All @@ -29,7 +34,10 @@ let
nix
];

env.NIXPKGS_VET_NIX_PACKAGE = lib.getBin nix;
env = {
NIXPKGS_VET_NIX_PACKAGE = lib.getBin nix;
NIXPKGS_VET_CONFIG_FILE = configFile;
};

passthru = {
# Allow running against all other Nix versions.
Expand All @@ -46,6 +54,7 @@ let
${initNix}
# This is what nixpkgs-vet uses
export NIXPKGS_VET_NIX_PACKAGE=${lib.getBin nix}
export NIXPKGS_VET_CONFIG_FILE=${configFile}
${nixpkgs-vet}/bin/.nixpkgs-vet-wrapped --base "${nixpkgs}" "${nixpkgs}"
touch $out
'';
Expand Down
Loading
Loading