Feedback and questions

[senpro-ingwersenk]

Hello there!

Apologies for opening a Q&A ticket; but I don't have a Discord for my job and my matrix server is currently belly-up (Dendrite update imploded and haven't had time to fix it).

We recently purchased Raspberry Pi 5s to use as off-site monitoring agents at customers. So far, we Have had been using NixOS on the rpi3 and rpi4 - but the rpi5 currently does not have an official NixOS image. While searching for potential workarounds, I came across SkiffOS - and I like it a lot! Granted, I am the only geek in this company, so I need to condense compexity down to a minimum if possible; not everyone here is used to this stuff. But, for now, it'll be fine.

I wanted to leave some feedback of the things that I discovered, perhaps they are useful to you:

• Finding my IP. ...was a nightmare. I used nmap to pick up the Pi after boot, but it certainly wasn't fun. I saw that the recent NixOS installers actually print the IP on the fb console - which is super handy. Perhaps adding something like this to SkiffOS would be possible? Alternatively, why not add a static network interface over the USB controller (the USB Type C port) with a static IP? That way I could get the initial configuration in, and set a static IP for further work.
• The core/nixos config is old; 21.11 is rather old by now. I haven't fully explored the docker-nixos repo to figure out where and how I could upgrade the base image.
• Setting time. My time was basically unset and I had to manually do it. Might be nice to run an ntp by default?
• Docs are excellent. I was within actually building the image very quickly. Simple and to the point.

Now, as for questions:

• Network Management. Our previous NixOS deployments all configure static IPs and firewall settings. It would be great if we could pass network setup into the container. That way, we wouldn't have to have vastly different onboarding steps. In fact, we use a configuration.nix template with most of this stuff prepared that we copy over, edit and adjust, and then nixos-rebuild switch to once done. And update nix-channels, if neccessary.
• Swapping core and root: Our configuration relies on root@ip a lot, so I would like to swap the root login to instead be the container, and core to instead go into SkiffOS itself. Is that possible?
• SSH key management: Is it possible to propagate SSH keys between SkiffOS and the NixOS container? Normally we have a set of five or so keys that we add to our configuration - and sometimes expand that with an additional key. Due to the overlay configuration, /root/.ssh is populated, but NixOS places those into /etc/ssh/authorized_keys.d. Any idea how that could be done? Alternatively I had thought of just running two SSH servers; one from SkiffOS and one from NixOS, so that Nix can manage it's own configuration just fine and using the other one to access the underlying SkiffOS, when needed.

So far, this is a great way to get started on a new device. I had previously worked with OpenADK for ucLibC shenanigans on RISC-V so buildroot does not feel too alien to me. And, the A/B booting is really nice to have in case something goes south.

Thanks for this awesome project and I hope to do much more with it in the future =)

Kind regards!

[paralin]

Thanks for your message! I'll review and reply to your questions today

[paralin]

@senpro-ingwersenk Thanks for trying out SkiffOS and for your comments. Feedback is very helpful to know what to prioritize in future for real-world use cases. Please don't hesitate to post more issues as needed.

Finding IP address: yes - this is a known problem, and definitely the setup experience in general could use improvement. I have long planned to implement a layer within SkiffOS which provides a basic GUI on the HDMI output port which provides some initial information and configuration knobs. For now logging the IP address would be a great start. If you have a proposed solution here I am happy to merge PRs.

As for the static network interface on USB - some of the boards actually have this already - see here:

SkiffOS/configs/usbarmory/mk2/root_overlay/etc/modprobe.d/usbarmory.conf

Line 4 in fba9dad

# serial only:

and

SkiffOS/configs/usbarmory/mk2/root_overlay/etc/network/interfaces

Line 6 in fba9dad

iface usb0 inet static

I believe you can accomplish this within overrides/workspaces/myworkspace/root_overlay adding the two files I put above. It would be possible also to configure this within NetworkManager.

NixOS - this is a quite old prototype and I tried to update it recently but ran into a number of problems with the newer versions. I can try a bit harder now that I know someone is using it. I think @TRPB has done work to update it. The base image is here: https://github.com/paralin/docker-nixos and TRPB updates: https://github.com/TRPB/docker-nixos

Time - this one should be handled automatically, systemctl status ntpd shows ntp running on my pi4 and it's configured here:

SkiffOS/configs-base/pre/buildroot/time

Line 3 in fba9dad

BR2_PACKAGE_NTP_NTPDATE=y

but it doesn't work until you have internet which might be the problem you're running into here.

Docs - glad to hear you like it. It of course doesn't work unless you already have a Linux system to build on, which could be improved. But the feedback is helpful here.

Network management - For this you can actually run it within the container if you want, I do this on some of my devices. What you need to do is blacklist the network interfaces you want to have running in the container from NetworkManager. You can do this with nmcli dev set wlp2s0 managed no or you can disable with a udev rule - https://github.com/skiffos/SkiffOS/blob/fba9dadedf6bea7483b5d0bb4f173abfea72ed64/configs/allwinner/d1/root_overlay/etc/udev/rules.d/90-nm-manage-gadget.rules#L3 or here is an example of a configuration file, which you could name 99-unmanaged-devices.conf, placed in /etc/NetworkManager/conf.d/:

[main]
plugins=keyfile

[keyfile]
unmanaged-devices=interface-name:eth0;interface-name:wlan1

I like the keyfile approach the best here.

Swapping core and root - for this I suggest running a second sshd daemon on a different port. I often run a ssh daemon within my Gentoo containers on the default port 22 and run the SkiffOS sshd on a different port. This way everything works as you expect connecting directly into the container, and if the container breaks for some reason you can ssh to the "management" port to access the host system.

SSH Key Management - skiff-core puts the keys in /mnt/persist/skiff/keys/ and you could symlink that path to /etc/ssh/authorized-keys.d within the container - would that work? One option.

Hopefully this helps! Happy to answer further questions or issues

Best regards - christian

[senpro-ingwersenk]

Thank you so much for this super detailed reply! Let me go through this bit by bit.
For context, I am, for now, targeting this configuration: Raspberry Pi 5, NixOS.

For now logging the IP address would be a great start. If you have a proposed solution here I am happy to merge PRs.

My immediate thought is, why not just take over the fb console? I did something rather similiar on a RasPi4 that I use as a worker to spawn btop - but I run Alpine there. Nevertheless:

## in: /etc/inittab

root@senst-nd-vpnbro /etc# cat inittab
# /etc/inittab

::sysinit:/sbin/openrc sysinit
::sysinit:/sbin/openrc boot
::wait:/sbin/openrc default

# Set up a couple of getty's
tty1::respawn:/sbin/agetty --autologin root --noclear 1152000 tty1 ## <-- right here
#tty1::respawn:/sbin/getty 38400 tty1
tty2::respawn:/sbin/getty 38400 tty2
tty3::respawn:/sbin/getty 38400 tty3
tty4::respawn:/sbin/getty 38400 tty4
tty5::respawn:/sbin/getty 38400 tty5
tty6::respawn:/sbin/getty 38400 tty6

# Put a getty on the serial port
#ttyS0::respawn:/sbin/getty -L 115200 ttyS0 vt100

# Stuff to do for the 3-finger salute
::ctrlaltdel:/sbin/reboot

# Stuff to do before rebooting
::shutdown:/sbin/openrc shutdown

I just use a logon script to check if I am logging in interactively and if it's an SSH session. If the latter fails, it's probably a login shell, so I spawn btop. Not exactly the smartest solution - but this way, I can spawn something there. You could use a similiar method to spawn a very simple program to read out the current network config. Considering that skiff-core is itself written in Go, it could perhaps just gain this feature as it is compiled for the target anyway. Something like...

tty1::respawn:/bin/skiff-core splashscreen

perhaps? That way, you could still use Alt+F{n} to open a normal login shell on the screen, but by default, it would show this splash with the IP and perhaps other useful info.

As for the static network interface on USB - some of the boards actually have this already

Oh, so all I would have to do is create a few overlay files to get the setup done on the RasPi5? As far as I know, it's basically just loading the dwc2 driver and having something like this in the cmdline.txt: modules=sd-mod,usb-storage,ext4,dwc2,g_serial (I only enabled serial on the rpi4 - because I was experimenting with this feature, and it came in handy once already!)

However, this isn't neccessarily just for the kernel commandline - I think it could also be done via /etc/modules - which I guess would also go into the overlay. Alright then. :) I'll experiment with that!

NixOS

Yeah - having looked at the various ways NixOS can be built or even is built got me into quite a rabbit hole... I'll take a look here and see if there is something I can suggest or even offer as a pull-request. It basically boils down to something like nix-build <nixpkgs/nixos> -I ... -A ... - and copying the result. Will try to see what I can do with the official nix docker image; technically, all that one would need is to build the rootfs within a builder stage and then export that. Eh, I'll come back here once I have made myself a little more familiar with that. :)

Time (...) but it doesn't work until you have internet which might be the problem you're running into here.

It turns out that when I did try my initial rebuild, it reset it's time, thus destroyed TLS validation and I needed to reset it manually - then it worked. I suppose that was just a quirk of running inside a container, will keep an eye out for that!

Network management

Ah, another file in the overlay - checks out, it's the defacto way to do edits :) The keyfile seems to be the approach I will use also - very simple and straight forward. I will only need to set eth0 to unmanagd, so I will just largely copy that file. It is also the only interface that will need to be configured from within the container - anything else "does not matter", per-se.

Swapping core and root

Good to hear I am not immediately running into unexplored territory. Let me guess - I set this up by supplying my own sshd_config via the overlay as well? At least that would be my immediate guess.

SSH Key Management

That should be totally doable. I'd just have to check if that folder exists and then transfer the contents into the Nix deriviation. We use a sort-of centralized base config ( https://github.com/senpro-it/nixpkgs-overlay ) so I can just add that check there. Since the path is predictable, that should be easy to accomplish.

Again, thank you a lot for all the details! That'll keep me quite busy for the rest of today, that's for sure. If I run across anything, I'll be sure to post about it!

[senpro-ingwersenk]

Alright, got me some results!

Command used was nix-build --verbose '<nixpkgs/nixos>' -A config.system.build.tarball -I nixos-config=mini-configuration.nix.
It took me forever to realize I had a host architecture set in hardware-configuration.nix - since I copied those verbatim from one of our pis. oops!

But this confirmed my thesis that it would be possible to slim down the docker build process by using the nixos/nix Container as a base. The key component is to have the configuration import <nixpkgs/nixos/modules/virtualisation/docker-image.nix>. This has a few very useful side effects. Details of it are here: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/docker-image.nix

I have a lot of changes to do (it added a kernel, for instance) but I think I am at least getting somewhere on this front now!

[senpro-ingwersenk]

Transplanted this kind of setup by using TRPB's repo as a base.

STEP 6/6: RUN ls result/tarball; du -ah result/tarball
nixos-system-x86_64-linux.tar.xz
242M    result/tarball/nixos-system-x86_64-linux.tar.xz
242M    result/tarball
COMMIT nixos
--> a53a22b249ca
Successfully tagged localhost/nixos:latest

FROM nixos/nix:latest AS builder

WORKDIR /root
ADD *.nix .
RUN nix-channel --update
RUN nix-build  --verbose '<nixpkgs/nixos>' \
    -A config.system.build.tarball \
    -I nixos-config=configuration.nix
RUN ls result/tarball; du -ah result/tarball

Going into the container and unpacking the tarball reveals a nearly complete rootfs - symlinks in /etc are entirely missing through...

That was, untill I saw my little mistake. I should have used -A system. So, changed that over, and edited the Container file quite some and ended up with:

...That SystemD is not exactly happy, yep. But that's because I gave it practically nothing in terms of mounts, devices and alike. However, it DID boot!

So here is where I am at:

FROM nixos/nix:latest AS builder

WORKDIR /root
ADD *.nix .

RUN nix-channel --update
RUN nix-build  --verbose '<nixpkgs/nixos>' \
    -A system \
    -I nixos-config=configuration.nix
RUN touch result/etc/NIXOS
RUN mkdir -p result/etc/nixos/
RUN cp configuration.nix ./result/etc/nixos/

FROM scratch AS final

COPY --from=builder /nix /nix
COPY --from=builder /root/result /

COPY options.nix /options.nix
COPY container-base-config-flake.nix /baseconfig/flake.nix
COPY configuration.nix /baseconfig/container.nix
COPY config /config

STOPSIGNAL SIGRTMIN+3
WORKDIR /
ENV container docker

ENTRYPOINT ["/init"]

And with this configuration.nix - it's tweaked compared to the repo.

{ config, pkgs, lib, ... }:
let
  options = import ./options.nix;
  flake = ''
    {
      description = "Container";

      inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
        container-base-config.url = "path:/baseconfig";
        user-config.url = "${options.flakeUrl}";
      };

      outputs = { nixpkgs, user-config, container-base-config, ... }@inputs: {
          nixosConfigurations.container = user-config.nixosConfigurations.${options.nixosConfiguration}.extendModules {
            modules = [container-base-config.nixosModules.containerConfig];
          };
      };
    }
    '';
in {
  imports = [
    #<nixpkgs/nixos/modules/virtualisation/docker-image.nix>
  ];
  system.stateVersion = "25.05";

  boot.initrd.availableKernelModules = [ ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];

  boot.kernel.enable = false;
  boot.isContainer = true;
  boot.loader = {
    systemd-boot.enable = false;
    efi.canTouchEfiVariables = false;
  };
  boot.postBootCommands = lib.mkForce "";
  # This pulls in nixos-containers which depends on Perl.
  boot.enableContainers = lib.mkDefault false;

  fileSystems."/" =
    { device = "overlay";
      fsType = "overlay";
      noCheck = true;
    };

  fileSystems."/run" =
    { device = "none";
      fsType = "tmpfs";
      options = [ "defaults" "size=2G" "mode=777" ];
    };

  swapDevices = [ ];
  powerManagement.cpuFreqGovernor = lib.mkDefault "ondemand";

  nix.settings.sandbox = false;
  networking.firewall.enable = lib.mkDefault false;
  networking.hostName = lib.mkForce "";
  networking.interfaces.eth0.useDHCP = false;
  networking.nameservers = [ "1.1.1.1" "8.8.8.8" ];
  networking.networkmanager.enable = lib.mkForce false;
  #networking.resolvconf.dnsExtensionMechanism = false;
  networking.useDHCP = false;
  networking.wireless.enable = false;
  nix.distributedBuilds = true;
  security.audit.enable = false;
  security.sudo.enable = true;
  systemd.enableEmergencyMode = false;
  systemd.services.console-getty.enable = lib.mkForce false;
  systemd.services.rescue.enable = false;
  systemd.services.systemd-firstboot.enable = lib.mkForce false;
  systemd.services.systemd-hostnamed.enable = lib.mkForce false;

  # minimal.nix
  documentation.enable = lib.mkDefault false;
  documentation.doc.enable = lib.mkDefault false;
  documentation.info.enable = lib.mkDefault false;
  documentation.man.enable = lib.mkDefault false;
  documentation.nixos.enable = lib.mkDefault false;

  # Perl is a default package.
  environment.defaultPackages = lib.mkDefault [ ];

  environment.stub-ld.enable = false;

  # The lessopen package pulls in Perl.
  programs.less.lessopen = lib.mkDefault null;

  programs.command-not-found.enable = lib.mkDefault false;

  services.logrotate.enable = lib.mkDefault false;

  services.udisks2.enable = lib.mkDefault false;

  xdg.autostart.enable = lib.mkDefault false;
  xdg.icons.enable = lib.mkDefault false;
  xdg.mime.enable = lib.mkDefault false;
  xdg.sounds.enable = lib.mkDefault false;

  systemd.mounts = [{
    where = "/sys/kernel/debug";
    enable = false;
  }];

  system.activationScripts.specialfs = lib.mkForce "";

  # don't set sycstl values in a container
  #systemd.services.systemd-sysctl.restartTriggers = lib.mkDefault [ ];
  environment.etc."sysctl.d/60-nixos.conf" = lib.mkForce { text = "# disabled\n"; };
  environment.etc."sysctl.d/50-default.conf" = lib.mkForce { text = "# diasbled\n"; };
  environment.etc."sysctl.d/50-coredump.conf" = lib.mkForce { text = "# disabled\n"; };

  # Docker makes this read only
  environment.etc."hosts".enable = false;
  boot.kernel.sysctl = lib.mkForce { "kernel.dmesg_restrict" = 0; };

  systemd.services.create-switch-script = {
    enable = true;
    script = ''
      mkdir -p /build
      echo '${flake}' > /build/flake.nix
      cp /options.nix /baseconfig/options.nix
      /run/current-system/sw/bin/nixos-rebuild switch --flake /build#container
    '';
    wantedBy = [ "multi-user.target" ];
  };
}

That said, I think the last thing I'd like to do is use ARG NIXOS_VERSION to define the version of nixos to build. Right now, it just picks unstable:

root@SENST-NB-KEIN ~/w/docker-nixos (main) [127]# docker run --privileged -it --entrypoint "/sw/bin/bash" nixos:latest
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory

[I have no name!@ab4aca621a25:/]# echo $SHELL
/bin/sh

[I have no name!@ab4aca621a25:/]# cat /nixos-version
25.11pre836767.e44b8dc0882d

(yep, that's a very unhappy container right there! But ... mad science, and what not.)

I'll take a break for now, but I am pretty sure I am almost at the point where I'd like to replace the image used in the SkiffOS image. In fact - I'd love to just hand it my Containerfile so it can build on first boot.

Oh and since I mainly use Podman, I use "Containerfile" as the filename instead... has gotten to be a habbit of mine.

[paralin]

Awesome! Great to see the progress! I'll try to replicate your work. If you want to send any PRs to update anything in SkiffOS feel free. Otherwise I'll try to integrate your changes soon.

[paralin]

Hi @senpro-ingwersenk I have done some work on the NixOS front:

• Updated docker-nixos to 25.05, latest everything, using Ubuntu rolling to build
• Added automatic GitHub actions builds which publish to docker pull ghcr.io/skiffos/docker-nixos:main-eb0ef39 as docker pull ghcr.io/skiffos/docker-nixos:main
• Updated the SkiffOS skiff-core-nixos to the latest version

I need to test it still to ensure the container starts up properly w/ the updated approach from @TRPB

[senpro-ingwersenk]

Awesome!

I'll spin up my environment, find the Pi 5 I have lying around here and try to make a build and see how it goes. =)

[senpro-ingwersenk]

Oh yeah - one small question: How, or where exactly, do I place any YAML overrides? I want to experiment with that a little. That one normally lives on the persistent partition. Do I just place those into ./overrides/root_overlay/mnt/persist?

Since my collegues are not very well versed in Linux, I want to try and swap the pseudo usernames:

• ssh root@device should go as root into the OS container
• ssh skiff@device should go to SkiffOS - the "bare metal" system

I do have my SSH keys in the overrides already - and I am going to run a build with the applied updates, but I just wanted to ask in advance while this runs :)

[paralin]

You can place a .yaml override in /opt in the root overlay, at: root_overlay/opt/skiff/coreenv/defconfig.yaml

I just realized this isn't documented anywhere, so I'll go document that soon :)

[paralin]

The tricky thing is ssh is running in the "host system" and we can't override "root" very easily. The best workaround I would recommend is to configure (with sshd_config in root overlay) the "host" ssh to run on a different port, like 1920, and run the ssh in the container on port 22. Then, ssh to root goes to the container, and ssh -p 1920 root goes to the host. Would that work for your use case?

[senpro-ingwersenk]

Heya!

I just ran the image I built on the 18th and:

• Had to learn that dtoverlay=uart-rpi5 was needed to get serial - but all I get is a whole bunch of gibberish.

- Watchinv the screen output, I can see `veth`s being added and coming up...but also going down. Sadly, from the scrolling kernel logs alone, I can't get much info, hence why I hoped to pop in via a TTL cable to get a shell or watch early boot.

Because I can't get a shell, I also can't tell if SSH is working or not - and scanning the whole network for 22/tcp would probably get my collegues on high alert because our firewall might not exactly enjoy that (we have >300 devices in the whole network, that'd be a big scan over a large address range).

So I'll re-flash the microSD and try again, perhaps I can get uart working then. But, this is my current status.