Run Spegel directly on host

[phillebaba]

Describe the problem to be solved

Running Spegel in a Pod is easy, as the Helm chart facilitates applying all the required manifests. Some users may however want to run Spegel on the host machine directly even if it may add more complexity.

Proposed solution to the problem

Spegel should start releasing a binary along with a Systemd configuration file and documentation for how to run Spegel directly on the host using Systemd.

There will be two considerations when implementing this feature. The first is that a new bootstrap method needs to be added as the Kubernetes DNS service is no longer available. The other is that network traffic is no longer done through the CNI which means that host to host network connectivity needs to be validated and firewall rules have to be documented.

[phillebaba]

@jipperinbham I created an issue to track the progress of supporting running Spegel with Systemd.

[Giaco9NN]

Hi! I'm really interested in this feature. Do you have an initial documentation to start testing it? I could act as a sort of beta tester and report suggestions, errors... 🙏🏻

[phillebaba]

Running Spegel as a binary is not that different from running it a container. The main difference is the bootstrapper. Spegel instances need to be able to find each other and agree on a set of nodes to act as bootstrap nodes. Currently when deploying in Kubernetes we use a headless service to do this discovery.

There is already someone else running Spegel as a binary and the way they solved this was to create their own DNS record outside of Kubernetes that Spegel can resolve. We cant use CoreDNS in the cluster as host processes do not use it as a DNS server.

@Giaco9NN could you share a bit more about the environment you are running in and the infrastructure that you have available when bootstrapping Spegel?

[Giaco9NN]

Hi :) The infrastructure is pretty standard and straightforward. I have a K8S cluster running on AWS with EKS made up of around 50 nodes, the Amazon VPC CNI and nothing more. The number of pods starting and terminating is very high because all the nodes are EC2 spot instances, which continuously start and terminate.
Running Spegel natively would ensure it is already running when the first pod is scheduled on the node. I know it is possible to achieve the same goal with taints, but I also want to investigate the native approach because it looks clearer and has fewer moving parts.
I hope this makes sense to you.

Thank you!

[phillebaba]

Thought about this for a while longer now. Kubernetes bootstrapping has its problems but has a major benefit that it is standardized method that works for all cloud providers. Running Spegel on a host outside of Spegel means we cant rely on Kubernetes for this work. I can't really build a bootstrap method for every cloud provider as it would take too much time to build and maintain in the future.

I am happy to setup so that out of tree bootstrap methods are possible if you want to build an maintain your own that works for AWS. So far the only company that I know of running Spegel on the host has built a custom bootstrapper for using their internal tooling. Alternatively if you do not have the resources to do it, this could be something that could be built and maintained as part of the Spegel enterprise offering.

[mickymiek]

We've implemented Spegel as a systemd unit on Flatcar Linux (bare-metal with containerd):

[Unit]
Description=Spegel - P2P OCI Registry Mirror
After=network-online.target containerd.service
Requires=containerd.service

[Service]
Type=simple
ExecStart=/opt/bin/spegel registry \
  --containerd-sock=/run/containerd/containerd.sock \
  --containerd-namespace=k8s.io \
  --containerd-content-path=/var/lib/containerd/io.containerd.content.v1.content \
  --registry-addr=:5000 \
  --router-addr=:5001 \
  --metrics-addr=:5090 \
  --bootstrap-kind=http \
  --http-bootstrap-addr=:5002 \
  --http-bootstrap-peer=http://<OTHER_NODE_IP>:5002 \
  --data-dir=/var/lib/spegel \
  --log-level=INFO
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Depending on your environment, you may need iptables rules to allow traffic between nodes on ports 5000 (registry), 5001 tcp+udp (DHT router), and 5002 (http bootstrap).

Run once after Spegel is healthy (can run as a systemd unit as well):

spegel configuration \
  --containerd-registry-config-path=/etc/containerd/certs.d \
  --mirrored-registries=https://your-registry.example.com \
  --mirror-targets=http://127.0.0.1:5000 \
  --resolve-tags=true

For bootstrap peer discovery, we generate --http-bootstrap-peer at provisioning time by pointing each node to another node's IP in the cluster.

This is not tested in production yet, but initial testing shows it's working as intended. Hope this helps :)