A test that runs a Cirrus-CI task

[0xB10C]

A good test for this repository could be a test similar to the basic.nix test that tries to run a Cirrus CI task. Currently, the basic.nix test only starts the runner-VM and checks that it's online, but doesn't check that it could actually run a Cirrus-CI task.

A few toughs:

• needs a Cirrus-CI worker pool for this project
• the test uses a hardcoded token to join the pool
• each PR or push to master should create a new Cirrus-CI job. We'd need a .cirrus.yml in this repository with a basic task definition e.g.:
  • check that the expected cache directories are available (and possibly try to write to them)
  • try to do a (trivial) docker build
  • try to run a basic script inside docker
• We won't be able to run the test offline, as wouldn't ever spawn a Cirrus-CI task. Figure out a way to skip it offline..
• Might run into problems with the Cirrus-CI scheduling weirdness discussed here: add --ephemeral mode for persistent worker cirruslabs/cirrus-cli#813 (comment)

[0xB10C]

Basically, we use a VM in a GitHub Actions CI run to run a Cirrus-CI task.

[s373nZ]

I've been digging on this one and want to share an update. Last weekend, I managed to get a simple NixOS test to complete a Cirrus CI task when run locally from my development machine.

However, things seem way more complicated when trying to run under Github Actions. After getting lost down quite a few rabbit holes (especially with regard to resource allocation), I've got a combination of prototype changes which manage to actually run the cirrus-worker service as well as start up rootless Docker. Presently, though, I haven't figured out how to fix the networking for the cirrus-cli worker to resolve its API endpoint in order to fetch jobs.

IIUC, the nested virtualization tree we are working with in this scenario is something like:

1. GitHub Actions hosted runner (a VM itself?)
  |- 2. QEMU VM started by the `nix flake check` for the (new) `cirrus-task` test
      |- 3. QEMU VM started by the `cirrus-ephemeral-runner.service` for the defined workers (only one in the current prototype)
          |- 4. Rootless Docker container started by `cirrus-cli`

Still working on the configuration and time outs, but the complexity of the nested environments is presently challenging my ability to conceive of the topology, as well as analyze and debug the networking.

In terms of VM resourcing, practical experimentation has shown that allocating more than 2 cores¹ to the service VM results in a lot of CPU stalling and kernel panics. The cirrus runner service VM is rather sluggish to boot in all cases. DHCP and rootless Docker often have a difficult time starting up. I've had to remove/reduce timeouts for a few services.

Current state of the (messy, mixed) experimentation branch is here, and these two CI runs are rather indicative of the present challenges with DNS resolution inside the container:

• Run 13544425840
  • Lastest change was adding br_netfilter kernel module to the cirrus-ephemeral-runner.service VM.
• Run 13544036877
  • Without that kernel module, docker info reports warnings WARNING: bridge-nf-call-ip(6)tables is disabled

Will keep thinking about this and attempting new adjustments, but now seems like a good time to share progress, review the feasibility constraints of the nested VM tree, and see if another set of eyes has any suggestions.

Footnotes

1. The hosted GitHub runner has 4 CPUs. ↩

[0xB10C]

Just had the chance to give this a closer look - thanks for looking into it! I appreciate the time you spent on it.

Still working on the configuration and time outs, but the complexity of the nested environments is presently challenging my ability to conceive of the topology, as well as analyze and debug the networking.

Yeah, I think complexity is a general theme with this setup...

Current state of the (messy, mixed) experimentation branch is here, and these two CI runs are rather indicative of the present challenges with DNS resolution inside the container:

I think you meant to write VM here (cirrus-runner doesn't run inside the docker container). Though, I'm having some problems with DNS resolution inside the docker container as well 0xB10C/bitcoin-core-cirrus-runner-infra#3.

---

As discussed elsewhere, were going to pause working on this for now.