Feature Request: Optional Build Proxy Server

[l4b4r4b4b4]

Prerequisites

• I have searched existing issues to make sure this isn't a duplicate
• I have checked the documentation to see if this feature already exists

Problem Description

Currently, nixos-anywhere offers two build modes:

1. Building locally on the machine where nixos-anywhere is run
2. Building directly on the remote target using --build-on auto|remote|local

Both approaches have limitations:

• Local builds require powerful hardware on the management machine
• Remote builds burden the target machine during installation
• Neither approach effectively caches builds across multiple similar deployments
• Deployment from resource-constrained management machines is challenging
• Pushing custom builds with slower internet connections takes long

Proposed Solution

I propose adding a third option: using a dedicated build proxy server that sits between the management machine and target systems.

Key Features

• Offload system builds to a machine with matching architecture as the target
• Cache built systems for faster deployment to multiple similar targets
• Store and manage flake configurations, disko settings, hardware information, and required files
• Automatic initialization of the proxy server on first use

Implementation Details

Add a new command line option:

--build-proxy [host] --build-proxy-ssh-key [key-path] [--build-proxy-flake [flake-uri]]

Where:

• host is the SSH connection string to the build proxy (e.g., user@proxy-server)
• build-proxy-ssh-key is the path to the SSH key for authenticating with the proxy
• build-proxy-flake is an optional reference to a flake for setting up the proxy server

On first use with a new proxy server, nixos-anywhere would:

1. Detect that the proxy isn't yet running NixOS or lacks required components
2. Deploy a minimal NixOS configuration to the proxy server using nixos-anywhere
3. Configure it with necessary build tools and caching capabilities
4. Set up SSH keys and access control for secure communication
5. Continue with the main deployment process

The build proxy would enable users to opt into either of the following independently:

1. Store a synchronized copy of the flake, configurations, hardware information (from facter), and extra files
2. Build the NixOS system generation(s) for the target machine(s)
3. Cache built generations and packages
4. Transfer only the needed artifacts to the target machine

Technical Requirements

1. The build proxy must have the same architecture as the target machine(s) it builds for
2. It should support SSH-based authentication for both control and target communications
3. It should maintain a synchronized state of configurations
4. It should provide caching capabilities to speed up similar deployments
5. It should integrate with existing nixos-anywhere workflow and options
6. It should be deployable to any machine with SSH access and kexec support
7. It should support running multiple proxy servers for different target architectures
8. It should automatically initialize the proxy server on first use if needed

Benefits

• Unlike local builds, doesn't require powerful local hardware
• Unlike --build-on remote, doesn't burden the target with building during installation
• Provides caching across multiple targets
• Ensures consistent builds from a controlled environment
• Speeds up deployments through parallelization and caching
• Allows for heterogeneous deployments (managing targets of different architectures)
• Can be deployed on-demand as needed and potentially destroyed after use
• Simplifies workflow by handling initialization automatically
• Serves as a persistent build cache for an organization's deployment pipeline

Backward Compatibility

This feature would be implemented as an additional option that doesn't affect existing workflows, maintaining backward compatibility with current usage patterns.

Alternatives Considered

No response

Feature Type

New command-line option

Use Case

Cluster Deployment

When deploying multiple similar nodes in a cluster, build once on the proxy and deploy to all nodes. This dramatically speeds up the deployment of homogeneous clusters by caching common components.

Low-resource Management Machine

Allow deployment from a lightweight management machine (like a laptop or small VM) by offloading build tasks to a more powerful build server. This enables administrators to manage deployments from resource-constrained environments.

Automated Deployments

Enable CI/CD pipelines to deploy to bare metal without needing to build the entire system locally in the CI environment. The build proxy can be a persistent component in the deployment infrastructure, supporting automatic testing and deployment workflows.

Heterogeneous Fleet Management

Maintain multiple build proxies for different target architectures, allowing a single management system to efficiently deploy to a diverse fleet of machines (x86_64, aarch64, etc.) without needing to perform architecture-specific builds locally.

Implementation Ideas

No response

Additional Context

No response

[Enzime]

Setting up a binary cache and using #554 would solve most of the usecases you mentioned

[l4b4r4b4b4]

@Enzime could you elaborate?!