Ghaf Infra

This repository declaratively defines the NixOS configuration for the Ghaf CI/CD infrastructure. All host configurations (including secrets) are version-controlled here.

Overview

The infrastructure includes:

• Jenkins CI environments (prod, dev, release) hosted at Hetzner
• Multi-architecture remote builders for x86_64 and aarch64
• On-prem test agents with connected hardware devices
• Supporting services: monitoring, logging, authentication, Nebula overlay network, NetHSM hardware signing, and an OCI container registry
• Secrets management via sops-nix (see architecture overview)

Getting Started

This document assumes you have nix with flakes support.

Clone this repository:

❯ git clone https://github.com/tiiuae/ghaf-infra.git
❯ cd ghaf-infra

Bootstrap nix shell with the required dependencies:

❯ nix develop

All commands referenced in the documentation are executed inside the nix-shell.

Development

The dev shell includes pre-commit hooks that run automatically on git commit. See nix/git-hooks.nix for the full list. To run them manually against all files:

nix fmt

To evaluate all Nix expressions and validate NixOS configurations without building derivations (catches syntax errors, type mismatches, and missing attributes):

nix flake check --no-build

To run the full check suite including builds:

nix flake check

Directory Structure

ghaf-infra
├── docs/               # Documentation (see Documentation section below)
├── hosts/              # NixOS host configurations
│   ├── builders/       # Remote builder machines
│   ├── hetzci/         # Jenkins CI environments (see hetzci/README.md)
│   ├── testagent/      # On-prem test agents
│   ├── ghaf-*/         # Supporting services (monitoring, auth, registry, etc.)
│   └── machines.nix    # Host inventory (IPs, keys, Nebula addresses)
├── nix/                # Flake plumbing (deployments, apps, git-hooks)
├── scripts/            # Operational scripts
├── services/           # Shared NixOS service modules
├── users/              # Admin user configurations
└── tasks.py            # Invoke tasks (see docs/tasks.md)

Documentation

• Architecture overview — how all the pieces fit together
• Adding a new host — step-by-step runbook for onboarding a host
• Deployment tasks — install, reboot, and other operational tasks
• Deploying with deploy-rs — how to deploy configuration changes
• Monitoring — Grafana and Prometheus setup
• Nebula overlay network — network connectivity between hosts
• NetHSM hardware signing — hardware-backed signing
• Jenkins authentication — Jenkins auth setup
• Jenkins test agents — on-prem test agents
• Jenkins CI development — developing the CI environment

Common Tasks

• Deploy configuration changes — deploy-rs
• Add a new host — adding a host
• Add a remote builder user — add their SSH key to developers.nix, then deploy
• Onboard a new admin — add their user to users/, optionally add their age key to .sops.yaml and run inv update-sops-files, then deploy
• Manage secrets — secrets management
• Install, reboot, and other operational tasks — tasks

Note: Hosts may be reinstalled at any time. Do not store important data outside the configurations in this repository — including in /home directories on the hosts.

License

This project is REUSE-compliant. See LICENSES/ and the SPDX headers in each file.