fix: flakes locking

[Malix-Labs]

Currently

flakes aren't pin locked

There is also no periodic batch flakes checking,
Even though flakes are imported periodically (currently hourly) with potentially changing sources, without pinning locking flakes between checks, nor preventing the individual failing ones from being updated (see #1027)

Solution

Pin locking flakes

• enhance(flakes): switch from toml to flake.nix #1145

Related

• fix(ci): Flakes Check - Decoupling Patches and Batch #1027

[raboof]

What would be the advantage of pinning the flakes? We want the documentation to describe the 'latest' information on flakes, wouldn't pinning cause require additional work/automation to make that work? I'm not sure the advantages weigh up to the costs.

[Malix-Labs]

Pinning gives us reproducibility in CI as if one flake breaks, we would be able to not omit the update of the specific broken flake in the lockfile and so make their latest valid version still searchable

Also, it would then enable us to add a state information like "✔️ Evaluable" / "❌ Unevaluable" or similar along with the ref and date of the latest proved evaluable version, and if unevaluable, the ref and date of the latest check

We could even then uncomment the currently broken flakes so that they would still be checked periodically (or on demand with workflow_dispatch) in case they are working again

[raboof]

OTOH, dropping broken flakes might motivate people to fix them 😄. And sometimes no information is better than outdated information...

I'm not entirely against the idea of pinning, but it seems like it'd either require work to update the pins (and we already have plenty of work) or additional automation (which would have to be developed, secured and maintained) - I still don't see the advantages weigh up to that cost.

[Malix-Labs]

What I started working on is a refactor of flake-info with the exact same input and output as the official nix flake unified cli

If we drop TOML, rely on the nix flake unified cli (including update), and dropping some now not-needed-anymore failbacks for ES, it could potentially (moonshot, i can absolutely not be sure yet, but possible) be lower surface

Since this refactor encompasses both #1144 and #1145 (since it will be nix-input-only and drop toml and json input support (but keep json output support)), I will talk about it more in the draft PR i will open in a decade or so

[raboof]

I will talk about it more in the draft PR i will open in a decade or so

😆

I mostly worry about: if we start pinning, who/how/where are these pins updated?

[Malix-Labs]

who/how/where are these pins updated?

In CI (GitHub Action), a job for each nix flake input : nix flake update <FLAKE_INPUT> + flake-info flake <FLAKE_INPUT>; if success: pipe to ES; else: pipe the failure infos to ES + revert nix flake update

[raboof]

who/how/where are these pins updated?

In CI (GitHub Action), a job for each nix flake input : nix flake update <FLAKE_INPUT> + flake-info flake <FLAKE_INPUT>; if success: pipe to ES; else: pipe the failure infos to ES + revert nix flake update

and then CI would commit those pin updates straight to the main branch? I'm still skeptical of the cost in work and complexity (and security) vs the usefulness of the change

[Malix-Labs]

and then CI would commit those pin updates straight to the main branch?

Yes, like what https://github.com/NixOS/nixos-search/blob/main/.github/workflows/update-flake-lock.yml alraedy does for the top-level flake.nix (but in this case we would not commit the update to the ranges corresponding to the failing flakes)

I'm still skeptical of the cost in work and complexity (and security) vs the usefulness of the change

I don't think it would really be that much more complex

About the security aspect, it would actually be more secure since we would rely on the nix flake unified cli which has made its proof so far to pin the flakes with verifiable SHAs, along with the added reproducibility

In my opinion and how I use nixos-search for flakes in my workflow, this would be quite useful

I'll let you know when the draft PR drops so you can see

[raboof]

and then CI would commit those pin updates straight to the main branch?

Yes, like what https://github.com/NixOS/nixos-search/blob/main/.github/workflows/update-flake-lock.yml alraedy does for the top-level flake.nix (but in this case we would not commit the update to the ranges corresponding to the failing flakes)

That one needs a manual approval every week. That's OK, but I wouldn't want to have to manually approve more often, and I wouldn't want to have updates to 3rd-party flakes delayed by so much...

[Malix-Labs]

We wouldn't benefit from manually approving the ./flakes ones though, since that's the current baseline too