[RFC 0197] package sets by-name #197
+254
−0
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,258 @@ | ||
| --- | ||
| feature: nixpkgs package set location | ||
| start-date: 2026-01-24 | ||
| author: quantenzitrone | ||
| co-authors: (find a buddy later to help out with the RFC) | ||
| shepherd-team: (names, to be nominated and accepted by RFC steering committee) | ||
| shepherd-leader: (name to be appointed by RFC steering committee) | ||
| related-issues: (will contain links to implementation PRs) | ||
| --- | ||
|
|
||
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| Different ideas on how to handle package sets in nixpkgs. | ||
|
|
||
| # Motivation | ||
| [motivation]: #motivation | ||
|
|
||
| - get rid of the package categories as directories. | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - make packages in package sets take advantage of the by-name greatness | ||
| - auto updates with r-ryantm + merge bot maintainer merging | ||
|
||
| - *your goal here* | ||
This comment was marked as off-topic.
Sorry, something went wrong. There was a problem hiding this comment. Updates across package sets are very different. I don't think this should be a goal of this RFC, but instead something that can be developed separately as an |
||
|
|
||
| <details> | ||
| <summary> | ||
|
|
||
| # Idea 1: new directory for package sets with by-name structure | ||
|
|
||
| </summary> | ||
|
|
||
| Proof-Of-Concept implementation in https://github.com/NixOS/nixpkgs/pull/482538 and https://github.com/NixOS/nixpkgs/pull/482543 | ||
|
|
||
| ## Detailed design | ||
|
|
||
| - Create a new directory under `pkgs/`, e.g. `pkgs/sets-by-name` that contains package sets. | ||
| - For example `pkgs/sets-by-name/fishPlugins`, `pkgs/sets-by-name/python3Packages`. | ||
| - Each packageset has a `packageset.nix`, that functions as an entrypoint to the package set. | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - The `packageset.nix` has some code like `top-level/by-name-overlay.nix` that autocalls packages | ||
| in the folder. | ||
| - with sharding for large package sets like `python3Packages` | ||
| - without sharding for small package sets like `fishPlugins` | ||
| - threshold for sharding is to be discussed | ||
| - 1000 (including extra files like `README.md` and `packageset.nix`) should the the absolute | ||
| maximum because of GitHubs UI | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - the `packageset.nix` can include aliases, functions like `fishPlugins.buildFishPlugin` and | ||
| overrides. | ||
| - The `packageset.nix` files are autocalled by some `pkgs/top-level/by-name-overlay.nix` like file. | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - Versioned package sets like `python316Packages` are done in `all-packages.nix` by calling the | ||
|
||
| `packageset.nix` again with different arguments. | ||
|
|
||
| ``` | ||
| pkgs | ||
| ├── by-name | ||
| │ └── ... | ||
| └── sets-by-name | ||
| ├── fishPlugins <- small package set so no sharding | ||
| │ ├── async-prompt | ||
| │ │ └── package.nix | ||
| │ ├── autopair | ||
| │ │ └── package.nix | ||
| │ ... | ||
| │ ├── z | ||
| │ │ └── package.nix | ||
| │ └── packageset.nix <- entrypoint to package set | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| │ | ||
| ├── python3Packages <- large package set with sharding | ||
| │ ├── a2 | ||
| │ │ └── a2wsgi | ||
| │ │ └── package.nix | ||
| │ ├── aa | ||
| │ ... | ||
| │ ├── zx | ||
| │ │ ├── zxcvbn | ||
| │ │ │ └── package.nix | ||
| │ │ ├── zxcvbn-rs-py | ||
| │ │ │ └── package.nix | ||
| │ │ └── zxing-cpp | ||
| │ │ └── package.nix | ||
| │ └── packageset.nix <- entrypoint to package set | ||
| ... | ||
| ``` | ||
|
|
||
| ## Advantages | ||
|
|
||
| - accessibility through github ui navigation (better than idea 2 and 3) | ||
| - clear package separation | ||
| - reuse of RFC 140 concepts | ||
| - making maintainer merging work for this is probably relatively simple | ||
| - implement tooling to work for a non-sharded by-name structure | ||
| - enable tooling on all subdirectories of `pkgs/sets-by-name` | ||
|
|
||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ## Drawbacks | ||
|
|
||
| - autocalling logic has to be duplicated for each package set | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - *your drawback here* | ||
|
|
||
| ## Unresolved Questions | ||
|
|
||
| - *your question here* | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| </details> | ||
|
|
||
| <details> | ||
| <summary> | ||
|
|
||
| # Idea 2: nested by-name structure | ||
|
|
||
| </summary> | ||
|
|
||
| Proof-Of-Concept implementation in https://github.com/NixOS/nixpkgs/pull/483432 | ||
|
|
||
| ## Detailed design | ||
|
|
||
| - Package sets are done in a nested `by-name` structure under `pkgs/by-name`, e.g. | ||
| `fishPlugins.puffer` would be in `by-name/fi/fishPlugins/pu/puffer`. | ||
| - To distinguish packages from packagesets we have to mark packagesets somehow, for example with a | ||
| file `by-name/fi/fishPlugins/.packageset`, if that file exists, it's called as a packageset, | ||
| if not the `package.nix` must exist and is called as a package. | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ``` | ||
| pkgs | ||
| └── by-name | ||
| ├── 0_ | ||
| ... | ||
| ├── fi | ||
| │ ├── fiano | ||
| │ ├── fiche | ||
| │ ├── ... | ||
| │ ├── fishnet | ||
| │ ├── fishPlugins | ||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| │ │ ├── .packageset | ||
| │ │ ├── as | ||
| │ │ │ └── async-prompt | ||
| │ │ ├── au | ||
| │ │ ... | ||
| │ │ └── z_ | ||
| │ │ └── z | ||
| │ ├── fishy | ||
| │ ... | ||
| ... | ||
| ``` | ||
|
|
||
| ## Advantages | ||
|
|
||
| - clear package separation | ||
| - reuse of RFC 140 concepts | ||
| - making maintainer merging work for this is probably relatively simple | ||
|
|
||
| ## Drawbacks | ||
|
|
||
| - unresolved questions | ||
| - *your drawback here* | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Unresolved Questions | ||
|
|
||
| - How do we handle functions like `fishPlugins.buildFishPlugin`? | ||
| - How do we handle aliases? | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - How do we handle versioned package sets? | ||
| - How do we move large package sets? | ||
|
|
||
| </details> | ||
|
|
||
| <details> | ||
| <summary> | ||
|
|
||
| # Idea 3: package sets in `pkgs/by-name` (probably worse than Idea 2 in every way) | ||
|
|
||
| </summary> | ||
|
|
||
| Proof-Of-Concept implementation in https://github.com/NixOS/nixpkgs/pull/483128 | ||
|
|
||
| ## Detailed design | ||
|
|
||
| - Instead of `by-name/<shard>/<pname>` we have `by-name/<shard>/<attrpath>`, so e.g. | ||
| `fishPlugins.puffer` would go in `by-name/fi/fishPlugins.puffer`. | ||
| - The `top-level/by-name-overlay.nix` will call all folders in a `<shard>` that contain a dot as a | ||
| package set. | ||
|
|
||
| ``` | ||
| pkgs | ||
| └── by-name | ||
| ├── 0_ | ||
| ... | ||
| ├── fi | ||
| │ ├── fiano | ||
| │ ├── fiche | ||
| │ ├── ... | ||
| │ ├── fishnet | ||
| │ ├── fishPlugins.async-prompt | ||
| │ ├── fishPlugins.autopair | ||
| │ ├── fishPlugins.z | ||
| │ ├── fishy | ||
| │ ... | ||
| ... | ||
| ``` | ||
|
|
||
| ## Advantages | ||
|
|
||
| - clear package separation | ||
| - reuse of RFC 140 concepts | ||
| - making maintainer merging work for this is probably easy | ||
|
|
||
| ## Drawbacks | ||
|
|
||
| - huge shards due to package sets | ||
| - currently only few shards like `li` are too large for GitHubs UI, but with this idea a lot of | ||
| shards like `py` (`python3Packages`), `ha` (`haskellPackages`) or `vi` (`vimPlugins`) will be | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| huge as well. | ||
| - directory names as attrpaths is sketchy | ||
| - unresolved questions | ||
|
|
||
| ## Unresolved Questions | ||
|
|
||
| - How do we handle functions like `fishPlugins.buildFishPlugin`? | ||
|
There was a problem hiding this comment. (Answering because this applies to idea 1) I never thought it made sense for functions and derivation builders to be inside package sets. I think our Go infrastructure has the right idea: move them to the top-level, and version them in the attribute name.
Member
There was a problem hiding this comment. This proposal is awkward with Python where you have CPython and PyPy and then language version-dialects within them, and even worse for Common Lisp where you have same base language but multiple compatible implementations with their completely own versioning.
Author
There was a problem hiding this comment. i also think keeping functions in package sets is fine |
||
| - How do we handle aliases? | ||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - How do we handle versioned package sets? | ||
|
There was a problem hiding this comment. (Answering because this applies to idea 1) I generally think it's a good idea to think of Nixpkgs as having three types of package infrastructure systems:
Author
There was a problem hiding this comment.
Author
There was a problem hiding this comment. i want handle both 2 and 3 with this RFC |
||
| - How do we move large package sets? | ||
|
There was a problem hiding this comment. (Answering because this applies to idea 1) Some of them are easy enough. If we resolve the aforementioned issues with the python edge cases, it should be a simple matter of moving and renaming files. OCaml should also be pretty simple. The generated sets, like Haskell, Node, Lisp and R, will be impossible, though. Those packages don't actually exist as separate files in Nixpkgs, and are instead imported from some other upstream. This is something that again, we'll have to get the input of package set maintainers on. Moving everything over will end up being a case-by-case basis. |
||
|
|
||
| </details> | ||
|
|
||
| # Detailed design | ||
| [design]: #detailed-design | ||
|
|
||
| TODO: decide for one of the above ideas | ||
quantenzitrone marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| # Examples and Interactions | ||
| [examples-and-interactions]: #examples-and-interactions | ||
|
|
||
| TODO | ||
|
|
||
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
|
|
||
| TODO | ||
|
|
||
| # Alternatives | ||
| [alternatives]: #alternatives | ||
|
|
||
| TODO: move other designs here once decided | ||
|
|
||
| # Prior art | ||
| [prior-art]: #prior-art | ||
|
|
||
| - `by-name` stucture for `python3Packages` https://github.com/NixOS/nixpkgs/pull/449896 https://github.com/NixOS/nixpkgs-vet/pull/180 | ||
| - https://github.com/NixOS/nixpkgs/issues/482537 | ||
| - https://github.com/NixOS/nixpkgs/issues/432625 | ||
| - `tclPackages` has their own `by-name` structure https://github.com/NixOS/nixpkgs/pull/344716 | ||
| - attempt to move `nushellPlugins` to `by-name` https://github.com/NixOS/nixpkgs/pull/482961 | ||
|
|
||
| # Unresolved questions | ||
| [unresolved]: #unresolved-questions | ||
|
|
||
| What parts of the design are still TBD or unknowns? | ||
|
|
||
| # Future work | ||
| [future]: #future-work | ||
|
|
||
| What future work, if any, would be implied or impacted by this feature without being directly part of the work? | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is a package set? There are some that do not have a direct top-level name. I think in general some places of the text is written for a package set that comes with its set of expressions, while there are cases where meaningfully sets share a large fraction of the expressions.