Generate declarative jobsets for Hydra for a GitHub Project

This tool uses Hydra's declarative jobsets to automatically create Hydra jobsets for your GitHub project.

Configuring declarative jobsets use several points of indirection, and this repository's goal is to simplify and streamline the setup process.

Testing

In this example, the tool will generate a jobset for fc-nixos.

First, fetch all open PRs from the API and transform it into the format Hydra expects:

curl "https://api.github.com/repos/flyingcircusio/fc-nixos/pulls?per_page=100" | jq 'map({(.number|tostring): .})|add' > prs.json

This assumes checkouts of nixpkgs and fc-nixos in ../. Replace with the path to the checkouts on your machine. Generate the declarative jobset JSON with the following command:

nix-build jobset/generate.nix \
    --arg nixpkgs ../nixpkgs \
    --arg generator ./. \
    --arg pull_requests ./data.json \
    --arg generator_config ../fc-nixos \
    --arg ignore-prs-from-forks true

For faster iteration times, it makes sense to work with cargo run:

nix-shell
cd jobset-generator
cargo run -- ../data.json ../../fc-nixos/.hydra/config.json  --ignore-prs-from-forks|jq 'length'

How Declarative Jobsets Work

The goal of Declarative Jobsets is to have a Hydra Project whose Jobsets are entirely defined outside of Hydra's interface. This means Declarative Jobsets is configured at the Project level.

Briefly:

• The Project's configuration points to a JSON document configuring the Hydra jobset .jobsets.
• The .jobsets jobset has a single Hydra job called jobsets.
• The jobsets job should produce a JSON document at $out with a map of jobset names and jobset configurations.

Some implementation details:

• Before the .jobsets jobset is evaluated, the Project's configuration is updated. This means updates to the data pointed to by the Project is reflected.
• The jobsets job does not and should not include the configuration for the .jobsets job.

How to use this tool

This example is assuming you will build PRs for a project at github.com/YOURORGNAME/YOURREPONAME.

Configuring your repository: .jobsets

In your project's repository, create a file at .hydra/project.json. This contains the configuration for your Hydra's .jobsets jobset. It should look like this:

{
    "enabled": 1,
    "hidden": false,
    "description": "GitHub Pull Request Jobset Generator",
    "nixexprinput": "generator",
    "nixexprpath": "jobset/generate.nix",
    "checkinterval": 300,
    "schedulingshares": 100,
    "enableemail": false,
    "emailoverride": "",
    "keepnr": 3,
    "inputs": {
        "generator_config": {
            "type": "git",
            "value": "https://github.com/YOURORGNAME/YOURREPONAME.git main",
            "emailresponsible": false
        },
        "generator": {
            "type": "git",
            "value": "https://github.com/DeterminateSystems/hydra-github-jobsets-generator.git main",
            "emailresponsible": false
        },
        "nixpkgs": {
            "type": "git",
            "value": "https://github.com/NixOS/nixpkgs.git nixos-unstable-small",
            "emailresponsible": false
        },
        "pull_requests": {
            "type": "githubpulls",
            "value": "YOURORGNAME YOURREPONAME",
            "emailresponsible": false
        }
    }
}

Take care to replace both YOURORGNAME and YOURREPONAME with your GitHub organization and repo's names.

Configuring your repository

Then, create a file at .hydra/config.json, and include all of the other Hydra inputs you want to use.

For example, if your project depends on Nixpkgs, write the following to .hydra/config.json:

{
    "inputs": {
        "nixpkgs": {
            "type": "git",
            "value": "git://github.com/NixOS/nixpkgs.git nixos-unstable-small",
            "emailresponsible": false
        }
    }
}

If your project has no other inputs, write {} to the file.

Configuring your Hydra

Create a project on your Hydra which uses a declarative input of type git with the file pointing to .hydra/project.json.

If you use the Terraform Hydra provider, it would look like this:

resource "hydra_project" "pr-example" {
  name         = "pr-example"
  display_name = "PR Example"
  description  = "An example of building a GitHub repository's PRs."
  owner        = "alice"
  enabled      = true
  visible      = true

  declarative {
      file = ".hydra/project.json"
      type = "git"
      value = "https://github.com/YOURORGNAME/YOURREPONAME.git main"
  }
}

Recap

When the .jobsets jobset evaluates and builds you should now see a jobset for every open PR. This jobset will re-evaluate every 300 seconds (5 minutes).