Duplicated values and the purpose of the router system

[ProducerMatt]

Hi there, I've been trying to migrate my NixOS config to den as it seems like it will ultimately make more sense and be easier to use. To avoid creating a bunch of bugs at once, I'm trying to migrate my repo with as few changes as possible, after which I can start refactoring it to make the most sense in den.

I've gotten stuck on a duplicate values bug that I can't seem to duplicate in the test repo, the classic "aspect being invoked multiple times" bug. Here is a commit of my repo at a state with the issue, running nix flake check will induce it. I was able to fix a few occurrences by limiting the aspect containing it to {OS,host}, however my progress was halted once I got to a behavior I wanted to put in aspects.default. I tried using take.exactly but this didn't fix the problem.

A core issue is I don't understand the dependency/routing system or why it needs to be able to output multiple times.

1. I missed the dependency page because I thought it was a section name, until the README linked me there and I realized I could have clicked it. (Since none of the other section headers have text, it might make sense to make it the first page of the section rather than living on the top level.)
2. I don't know any way to figure out why an aspect is being induced twice. A debug option showing what the dependency system is wanting would be extremely valuable.
3. I don't understand why any non-parametric aspect should ever be induced twice on the same system -- won't it only ever cause a duplication error? Or is this actually unintended behavior, and I've encountered a duplication bug?

Thanks for your work, would love to see any advancement in NixOS management as it's ornery at its best.

[vic]

Hey, I have not been able to find time to investigate this, nor look carefully at your repo, if you have made some progress please keep us updated.

[ProducerMatt]

I haven't made any progress yet. 😬

[ProducerMatt]

No change after updating to the latest den, here's the error, and here's the error with --show-trace. Wish I could pull my weight more but I'm still very baffled by the dependency system would induce an aspect twice.

[vic]

Work in progress: #175.

[ProducerMatt]

Thank you again for all your work, especially your kindness for documenting in English. Your explanation helped me understand a little more.

If I understand correctly: my duplicate issues are happening because my aspects are being invoked for both NixOS and the NixOS-managed home-manager, because these are separate contexts. I stumbled on a way to limit it to NixOS with {OS, host} but couldn't do the same for aspects.defaults (I tried using take.exactly but it didn't work).

Here's drawings of where my expectations were wrong:

I understand now why it had to be designed this way, as (for example) this lets you change configuration details depending on whether certain contexts are available.

I think a big source of confusion is the interface. For example, let's say every flake module got called once, with ctx having all available context for one machine in it:

ctx: with (ctx) lib; {
  nixosConfiguration = mkIf (ctx ? host) {
    ...
  };
}

This layout would have been more familiar, as it looks like classic Nix configuration. But this does not solve for multiple per-machine users, or other kinds of configs that would be nested in the same way.

[vic]

Take a look at these tests I'm working on today. https://github.com/vic/den/blob/ctx/templates/ci/modules/features/context/host-propagation.nix

There's an example of how to use take.exactly, and the expected list shows why there are duplicates being generated if you use a lax context function.

[ProducerMatt]

I think my example uses take.exactly in the way shown.

den.default.includes = [
  (den.lib.take.exactly ({OS, host}: ...

But I still get duplication. I notice that the error is from "determinate.enable", which is actually being set by a third-party nixosModule in nixos.imports. Could den be invoking the module twice, even though it isn't part of the aspect system?

[vic]

Hey @ProducerMatt, I borrowed your idea of using diagrams to document how Den works.

Please update to latest Den and see if the problem persists, because I've also fixed some duplicate problems we had -these were regarding den.default owned-configs causing dups-, but feel free to send a failing test under /templates/ci/modules/features/deadbugs/. The documentation is much better now, hopefully it will help understand how to avoid the problems with den.default now that we have much better alternatives to it.

[vic]

Today we solved #216, it was a tough one, not sure if that had anything to do with this other thread.

[ProducerMatt]

Hey Vic!

Thank you so much for the updates to the documentation, they're very high quality and I think I would have understood the system much better the first time. Also hooray for graphs 😁

Your updates so far have fixed two of my duplication bugs! first the one at module level and now the one under den.defaults. with den.lib.take.exactly ({host}: ...

Now I'm facing a duplication bug in users.users.matt.shell which is defined in here: https://github.com/ProducerMatt/flake/blob/5f3b3acbeb52c44cb9e93f0b403f4372559a62fd/modules/matt/default.nix#L17 but I haven't experimented much at all with how the aspect is included so maybe one of the lib.parametric functions or something similar will fix it. Once I've tried more I'll get back to you.

[ProducerMatt]

Sadly, making matt parametric actually adds a third duplicate 😭 So does parametric.atLeast but not parametric.exactly. Just tried wrapping all other modules containing imports with parametric as well, but no dice.

[vic]

oh be sure to update to latest Den. We solved something similar at #216, just in case you haven't updated yet.

[vic]

bogus repos help me a lot, since I don't have enough time to understand all people's code. if you are able to reproduce it, I'd happily see what is going wrong.

[ProducerMatt]

I'll try to recreate the bug in the bogus repo. Is there a way I can test the config without building the VM? I don't have enough ram for it.

EDIT: disregard the above, maybe the VM is no longer necessary (or I confused it with another project)

[vic]

Haha :D we do have a vm but in the default template, not in the bogus one.

[vic]

Hey @ProducerMatt, In main branch #272 Bidirectionality is now opt-in. Previously it was enabled by default causing confusion about why duplicate values might appear, the den.ctx.user description (Read the linked PR) provides insight on what Bidirectional means and why people were seeing duplicates since this was enabled by default.

Also, I want to personally thank you for this discussion, your input has been very valuable and influential to Den, from using diagram on documentation, to den.ctx that was started precisely for solving discussion, and now den.provides.bidirectional being opt-in and properly document why/how duplication can appear. Thanks a lot @ProducerMatt, for contributing to Den.

[ProducerMatt]

Never mind, I just updated the full lock file (previously did "nix flake update den") and this error doesn't appear anymore

[vic]

Yeah, read 0.12.0 release notes: https://github.com/vic/den/releases/tag/v0.12.0

[ProducerMatt]

In my config I have den.aspects.newPortable for my host newPortable, here I set nixos.system.stateVersion homeManager.home.stateVersion. Then I set includes to bring in the config. However I'm hitting

error: The option `home.stateVersion' was accessed but has no value defined. Try setting the option.

Any ideas what's causing this? Do the contents of the current aspect not get set until the others have been completely fulfilled?

[vic]

Oh you are using master branch, the reason for that you have a homeManager module at a host-aspect. It worked before #272 because Bidirectionality was enabled by default. Read that PR for what that means! After talking about bidirectionality with other users at matrix, I decided to make it opt-in, since not all people might want that enabled by default: The PR above mentions in detail why having bidir-enabled causes duplicate values for functions included at host-aspect.includes. Your host-aspect.homeManager module is not being included on your user unless your user now explicitly requests that it wants configuration from the host-aspect, this is done via den.provides.bidirectional, all details about this in the PR linked here. We also have an announcement post in here, be sure to read announcements and release notes when updating den, currently at v0 it is moving fast.

[ProducerMatt]

Ahhh ok. I enabled bidirectionality, unfortunately it enters an infinite loop in evaluating 'nixosConfigurations.newPortable' w/ memory continually growing. It surprised me since Nix usually catches infinite loops.

Latest commit is here. But I'll probably have to rebuild my config from scratch, test at every step, and ping you once I figure out what's causing it.