Option --copy-host-keys should allow a custom destination

[viceebun]

Prerequisites

• I have searched existing issues to make sure this isn't a duplicate
• I have checked the documentation to see if this feature already exists

Problem Description

I am using both sops-nix and impermanence, which wipes the root partition on boot, only saving the files in /persist. When enabled, writing the host's ssh keys to /mnt/etc/ssh/ssh_host_* isn't enough, as they are wiped immediately, the directory is bind mounted to /persist/etc/ssh, and then new keys are written.

Proposed Solution

Allow --copy-host-keys to have a <destination> option, defaulting to /etc/ssh, allowing for an override to a persistent directory, such as /persist/etc/ssh.

Alternatives Considered

No response

Feature Type

New command-line option

Use Case

Impermanence / NixOS persistence setups on new machines

Implementation Ideas

No response

Additional Context

No response

[tomwrw]

I have the same problem. Ideally, I could copy my build hosts keys (as they are trusted for sops) during build, and then replace them with the correct host keys post-deployment.

I work around this at the moment by copying the target hosts keys (I generate them and keep them encrypted on portable media) to a tmp folder, and passing that temp folder to nixos-anywhere using the --extra-files arguement. The trick for this is to make sure the folder structure of the tmp folder mirrors your persist location.

# /tmp/extra-files/ is what I'll pass to --extra-files. The descendant folders are the persist location on the target.
mkdir -p /tmp/extra-files/persist/etc/ssh

# Copy the pre-generated keys for the target host. These have already been authorised to use sops (I already ran ssh-to-age and imported the age public key to .sops.yaml).
cp ./secrets/hosts/spectre/ssh_host_ed25519_key /tmp/extra-files/persist/etc/ssh/
cp ./secrets/hosts/spectre/ssh_host_ed25519_key.pub /tmp/extra-files/persist/etc/ssh/
# Set the permissions on the tmp copied file.
chmod 600 /tmp/extra-files/persist/etc/ssh/ssh_host_ed25519_key

# Deploy with --extra-files as below.
nix --extra-experimental-features 'nix-command flakes' run github:nix-community/nixos-anywhere -- --disko-mode disko --extra-files /tmp/extra-files --flake .#spectre --target-host nixos@spectre

I guess one good thing about this approach is that I don't need to replace the keys after deployment since I already used to proper pre-generated keys.

[landure]

To complete the answer, for disko LUKS partitions.luks.content.passwordFile option behing '/tmp/luks_passwordfile', and secrets managed by sops-nix inpired by nix-community/disko#641 (comment),
this wrapper decrypt SOPS secrets before running nixos-anywhere.

Note that a better method might be using flake parts with agenix and agenix-shell.

  exec = ''
        set -e

        clear_luks_passwordfile="$(mktemp --quiet)"
        extra_files="$(mktemp --quiet --directory)"
        mkdir --parent "''${extra_files}/etc/ssh"

        delete-secrets() {
          echo "Removing decrypted SOPS secrets"
          [[ -e "''${clear_luks_passwordfile}" ]] && rm "''${clear_luks_passwordfile}" || true
          [[ -e "''${extra_files}" ]] && rm "''${extra_files}" || true
        }

        trap 'delete-secrets' EXIT

        hostname="''${1}"
        target_host="''${2}"

        secret_file="''${DEVENV_ROOT:-.}/secrets/hosts/''${hostname}.yaml"

        # SOPS_AGE_SSH_PRIVATE_KEY_FILE="''${HOME}/.ssh/id_ed25519"
        
        # See https://github.com/nix-community/disko/issues/641#issuecomment-2142627125
        sops --enable-local-keyservice --decrypt --extract "['luks_password']" \
            "''${secret_file}" > "''${clear_luks_passwordfile}"
        # See https://github.com/nix-community/nixos-anywhere/issues/604#issuecomment-3642062243
        sops --enable-local-keyservice --decrypt --extract "['openssh']['private_key']" \
            "''${secret_file}" > "''${extra_files}/etc/ssh/ssh_host_ed25519_key"
        sops --enable-local-keyservice --decrypt --extract "['openssh']['public_key']" \
            "''${secret_file}" > "''${extra_files}/etc/ssh/ssh_host_ed25519_key.pub"
        
        chmod 600 "''${extra_files}/etc/ssh/ssh_host_ed25519_key"

        nix run 'github:nix-community/nixos-anywhere' -- \
          --disk-encryption-keys '/tmp/luks_passwordfile' "''${clear_luks_passwordfile}" \
          --extra-files "''${extra_files}" \
          --flake ".#''${hostname}" \
          --target-host "''${target_host}"
  '';

[viceebun]

What? I'm not trying to strip the secrets. I'm not having troubles with disk encryption. I would simply like to have the copy keys option be able to have a custom destination without having to go the --extra-files route, which as tomwrw suggested, works (thanks by the way!). I've gone the route of generating new keys, echoing an aged key to the terminal, rekeying, then deploying.

[landure]

sorry, I know i'm "hors sujet" but since doc on how to integrate secret management with nixos-anywhere is sparse, i've put my script here. to complement tomwrw reply for those interested. I aim for my hosts keys to be predictable, so I store them in secrets.

[viceebun]

Sorry for the rude response. Neat idea to store the luks password in sops as well, wasn't initially sure if this meant you were stripping all the secrets from the install. Definitely a lot going into this! But probably overengineered for my purposes. If nothing else, I might try to define my own sops secret file for the password and somehow pass that through extra-files in the future