Need choice of NixOS as platform

[hzeller]

I often run into issues where the somewhat leaky hermeticity of bazel hits hard hermeticity enforcements in NixOS, so things that sorta compile with bazel on 'regular' Linux machines are breaking on NixOS.

This often results in BCR modules being broken on NixOS because they exploit some hidden assumptions of the target system. Problem of course is, that this platform is not available as presubmit platform in BCR -- so many modules are broken silently with an uphill battle to fix them one PR at a time by users of a hermetic system.

It would be a real benefit if it was possible to have a NixOS CI platform to choose from (in particular in the BCR so that it is possible to fix things and be sure that they don't backslide.

I suspect the BCR choices of available presubmit platforms hinges on bazelbuild/continuous-integration, hence I file an issue here.

I see that there was work towards a Nix set-up available (#1703) , but then looks like it got removed again. I don't know all the context, but it would be good if this can be re-activated (and stay stuck). I am happy to help if needed.

This is what I see happened before:

• Add BuildKite Docker image for NixOS #1779 Container with Nix was added
• NixOS container switched to Ubuntu container with Nix installed #1799 changed to use Ubuntu + nix package manager.
• Ubuntu (Nix) container removed #1827 Looks like Ubuntu+Nix got removed again :(

CC @rickvanprim as he did a lot of work here

[hzeller]

I see that I can find some time to work on this.

@rickvanprim : do you have some more context about the previous changes ? Naively, I'd start out with just a nix container, like you did in #1779 but then I see you changed it in #1799
Also, what was triggering removing it in #1827 ?

Just want to know what hurdles I get myself into :)

[rickvanprim]

Hey @hzeller, I don't remember the exact details given how long ago it was, but I think the issue I was hitting was that rules_rust (which is what I was trying to add this for) uses Bazelisk to fetch a variety of prebuilt Bazel versions for their tests and I was hitting issues trying to use a Nix version of Bazel instead. There may also have been something about issues with nix-ld not propagating to Bazel actions running prebuilt binaries. And finally, I'm thinking there wasn't a great way to make a derivative image of the official NixOS one with stuff preinstalled (which may not be needed anyways).

Given it sounds like you want a pure NixOS test hopefully those issues shouldn't be relevant 🙂 Apologies for not having a more concrete answer.