Hybrid clusters running on Cloud & Dedicated (Robot) Servers

[Aubermean]

Since the issue discussing this is burried and hard to find, I thought I should open up a discussion here.

4 months ago Hetzner merged the features from the Syself fork into hcloud-cloud-controller-manager (HCCM), that allows management of dedicated/Robot servers alongside cloud ones. They released it in v1.19.0.

Hetzner say that many customers use clusters on hybrid cloud/dedicated, and it was their reason behind adding the feature. But this project does not currently have support. I don't have the depth of knowledge required to implement it, but I imagine that many seasoned k3s/hcloud people have, and maybe we can discuss the steps or work involved to implement the feature.

[mysticaltech]

@Aubermean We would love to support this if implemented neatly. There have been others that were able to get something to work even before Hetzner merging in the new functionality. Please don't hesitate to tag them in, I'm sure they will appreciate the initiative.

[maaft]

yes, but it required a custom wirgeguard overlay network (or any other VPN technology) for the control plane since Hetzners vSwitch (that connects bare metal servers to private networks) is too slow according some comments on the internet.

Ultimately, I had a fork of this repo that is non-compatible with upstream which I more and more began to dislike.

Although, I'd really love to use bare metal for cheap base load, I came to the conclusion that I'd rather pay more money for cloud nodes that are tightly integrated with this repo to maintain ability to upgrade.

additionally, by using bare-metal, you loose all the automated stuff and e.g. recreating a new cluster from scratch becomes tedious.

[mysticaltech]

But @maaft, recently Hetzner added the possility to connect metal nodes to cloud private networks directly, so I guess without vswitch.

[maaft]

oh really? That's indeed highly interesting, but the other points still stand I guess.

Although it would be probably also possible to integrate provisioning of those also in this repo, right?

anyway, thanks for the heads up :)

[mysticaltech]

If it's not too convoluted, would love to support in this repo, at the very least have an example in the examples folder.

[BuddhiAbeyratne]

@maaft you are right about it being too tedious for stateless app workloads

However if you ever want to make a stateful db workload for eg a crunchy postgres deployment with anything more than a tb of storage you are kind of screwed with the vps instances. so I still think its valuable to have a standard approach even if its outside of this projects scope. ie: have a standard way to add workers manually.

Also with regards to the speed. It's determined by the network interface you purchase so you have to do your own duedeligenoe which I think its understandable as its a more advanced use case.

[dmorn]

I wanted to post a new discussion about this but maybe here is more appropriate. I have a working cluster that uses cilium. Now I'd like to join the servers equipped with GPUs to the cluster... vswitch? Wireguard? What options are there? Is using cilium with this requirements a good idea or am I shooting myself in the feet?

[mysticaltech]

@BuddhiAbeyratne I love cloudflare tunnels myself, but never used them on kube directly, but the thought had crossed my mind more than once. When you get this setup, it would be super cool if you could PR in a guide to the examples section and folder 🙏

[BuddhiAbeyratne]

@BuddhiAbeyratne I love cloudflare tunnels myself, but never used them on kube directly, but the thought had crossed my mind more than once. When you get this setup, it would be super cool if you could PR in a guide to the examples section and folder 🙏

http://reclaim-the-stack.com/

Far better write up than mine would ever be
Don't agree with every decision here like not using terraform for spinning up instances (Id argue thats a bit tedious to write though) but they have outlined thought process which is better than most other opinionated solutions. They have a set of charts that outline how it works.

[WebSpider]

@dmorn

this would be great but unfortunately my servers are not Robot ones, they are in a colocation not managed by Hetzner!

You may want to have a look at liqo

[aamir-s18]

@dmorn have you managed to make it work? I am trying to archive exactly the same thing.

[dmorn]

Hi @aamir-s18, yes it is a production grade cluster now. Unfortunately I did everything from scratch so I did not use this project to achieve the setup, but the approach is very similar in terms of configuration with cloud-config, terraform and some scripting.

• Tailscale for node interconnection. Authentication with auth_keys
• control-plane behind an Haproxy server, also within Tailscale, which you have to deploy first. You'll use that address as control plane endpoint
• kubeadm with containerd, be careful in setting up addresses properly. In particular, you just want to use tailscale IPs (this is where I struggled with in k3s, but I was a kube noob)
• Flannel CNI. Again, mind the address ranges

Drawbacks: you'll not be able to use the Hetzner Cloud Controller as it expects (or expected) the whole cluster to be hetzner hosted. It kept on complaining otherwise. If I reming correctly, setting node selectors did not help but again, I did not know.

---

Would I do it again? No. I'd make two separate clusters. Your nodes that are using the GPUs are going to export some sort of public API anyway, so let them live in a cluster (maybe even not HA, I'd use k8s just for the sake of kubectl-ling it) by themselves and instead create and manage the cluster on Hetzner separately, for example with this great project. If you really want them to be part of the same cluster, I'd check Cilium cluster mesh instead. My golden rule now is to avoid hybrid clusters as you loose a lot of features that cloud controllers can provide. Hope it helps!

[olexiyb]

Take a look in my pull #1405

[mcosti]

Hijacking this discussion: I have a few very strong machines for free in Oracle Cloud and it would be great if I could reuse their compute power in my hetzner cluster.

I saw the tutorial from k3s but I'm still quite new to this, is anyone willing to write some more detailed steps on how to achieve this (on an existing cluster, ideally, but willing to make a new one).

Thank you!

[olexiyb]

It's possible, but will be very tricky see

[mcosti]

Yes, I saw that page, but I am still unsure how to achieve this in this terraformed context

[dmorn]

I want to post an update to let you know how I achieved this setup (in production right now, works like a charm).

Disclaimer: as soon as I had to understand the whole setup, I did not use the module of this project for setting things up, so feel free to remove my comment if you think it does not belong here. On the other side, I think it might be helpful for those that know k3s and this module better than me to consider integration.

This is the list of things that you need to understand:

• each node needs to start the tailscale tunnel. For authentication, use authkeys. Enable magic DNS. turn the --accept-dns option on.
• cloud controller cannot be used in the hybrid setup, CSI can if you provide proper node affinities. The good news is you don't really need it: use terraform to create an HW load-balancer that points to the nodes of the cluster (you can use labels for that). When you want to expose a service, use NodePort and create a new target in the load balancer.
• when starting kubeadm/k3s, ensure the cluster only knows and uses VPN's ips (this was bitter for me with k3s). This is an idea of the kubeadm configuration you need

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
networking:
  podSubnet: "10.25.0.0/16"
  serviceSubnet: "10.26.0.0/16"
apiServer:
  certSANs:
    - "$node_vpn_ip"
controlPlaneEndpoint: "$control_endpoint:6443"
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  criSocket: "unix://var/run/containerd/containerd.sock"
  kubeletExtraArgs:
    node-ip: "$node_vpn_ip"
localAPIEndpoint:
  advertiseAddress: "$node_vpn_ip"

One note of the $control_endpoint: in my case that's an Haproxy in the tailnet that points to the control nodes for HA.

• I selected flannel as CNI as Cilium was breaking in so many weird ways that I stopped bothering. Ensure you set the correct subnets AND you make flannel use tailscale's interface. One thing that will save you some headaches: sometimes tailscale does a systemctl network restart equivalent apparently, which will leave your node w/o a flannel.1 interface, which means you'll loose intra node pod connectivity. Add a liveness probe to the kube-flannel container for the rescue

        livenessProbe:
          exec:
            command:
            - ifconfig
            - flannel.1
          initialDelaySeconds: 60
          periodSeconds: 5

That's basically it. The most important thing to get right is IP assignment. You want your nodes to have only the internal-ip set, and that must be the tailscale one. The setup can be fully automated with terraform and cloud-init.

Last thing: before connecting the cluster, ensure Tailscale is able to make direct connections between nodes or you'll shoot yourself in the feet 😇.

[pat-s]

After reading through all recent topics related to adding dedicated server support and the new featured related to it in hccm v1.19, I'd like to share my working setup (k3s 1.30.6 + AX42 node). It includes a few steps that must be taken which can't be done through this module but overall the module should be able to greatly assist such efforts with a few changes.

---

In hetznercloud/hcloud-cloud-controller-manager#561 support for automatically adding appropriate metadata to robot nodes in a cluster was added. While this simplifies a lot, more is needed to be able to connect robot servers to a cluster:

• robot and cloud servers must be able to communicate over a private network -> I don't think there's a way a round using a vSwitch atm. Yes, I've read that there are self-baked wireguard solutions but I haven't looked into these in more detail. In the end, there must be some way to ensure robot and cloud nodes can communicate with each other through a private network
• One must have a "Webservice User" in the robot account settings to be able to list all robot servers of a user (hccm leverages https://169.254.169.254/hetzner/v1/metadata/instance-id to get information about a users robot servers (using the webservice user))
• On the robot server one must manually configure a network interface and routes to be able to route the internal ip requests (10.x) through the vSwitch gateway. While there are detailed instructions for Ubuntu, RHEL-based ones are missing (provided below)
• In the current form of this module, one must deploy hccm via helm using specific settings (see below)

HCCM settings

The following values result in a successful auto-detection and processing of HCCM of new robot nodes joining the cluster.

Important:

• One must update the existing k8s secret hcloud with the robot-password and robot-user values.
• networking.enabled: true must be set (I couldn't get it working without)
• the value of cluster-cidr might be different for you, it should be the subnet value of the pods running in the cluster
• You need to deploy version >=1.19
• As the helm-chart does some things behind the scenes when setting networking.enabled: true and robot.enabled: true, replicating the same behavior using the existing template of this module is hard and likely not worth it.

networking:
  enabled: true
robot:
  enabled: true

args:
  allocate-node-cidrs: "true"
  cluster-cidr: "10.42.0.0/16"

env:
  HCLOUD_LOAD_BALANCERS_ENABLED:
    value: "true"
  HCLOUD_LOAD_BALANCERS_LOCATION:
    value: "fsn1"
  HCLOUD_LOAD_BALANCERS_USE_PRIVATE_IP:
    value: "true"
  HCLOUD_LOAD_BALANCERS_DISABLE_PRIVATE_INGRESS:
    value: "true"
  HCLOUD_NETWORK_ROUTES_ENABLED:
    value: "false"

  HCLOUD_TOKEN:
    valueFrom:
      secretKeyRef:
        name: hcloud
        key: token

  ROBOT_USER:
    valueFrom:
      secretKeyRef:
        name: hcloud
        key: robot-user
        optional: true
  ROBOT_PASSWORD:
    valueFrom:
      secretKeyRef:
        name: hcloud
        key: robot-password
        optional: true

With this, the cluster should continue working normally without any robot node having joined yet.

Next, install the k3s-agent on the robot node with the same config of the other existing nodes:

1. Create /etc/rancher/k3s/config.yaml. In the below config I've reserved 6GB memory for "system-reserved" which means that k3s won't use this memory for the node (optional and you can also use all existing memory). You might need to change the flannel-iface setting and set it to the main interface device name of your robot node (-> ifconfig).

"flannel-iface": "enp6s0"
"kubelet-arg":
- "cloud-provider=external"
- "volume-plugin-dir=/var/lib/kubelet/volumeplugins"
- "kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi"
- "system-reserved=cpu=250m,memory=6000Mi"
"node-label":
- "k3s_upgrade=true"
"node-taint": []
"selinux": true
"server": "https://$IP:6443"
"token": "$TOKEN"

Before starting/installing the k3s-agent, ensure that you can ping other nodes in the cluster (including the API head nodes) using their internal IP, e.g. ping 10.255.0.101 (random node ip). Without a successful response, the above won't work and you need to check on your vSwitch/routing config.

Last, here's an interface+routing config via nmcli which works on redhat-based OS like Almalinux. This assumes that

• the vSwitch subnet was created with 10.1.0.0/24
• the vSwitch ID is 4022
• the main network interface is named enp6s0

Details

nmcli connection add type vlan con-name vlan4022 ifname vlan4022 vlan.parent enp6s0 vlan.id 4022

nmcli connection modify vlan4022 802-3-ethernet.mtu 1400
nmcli connection modify vlan4022 ipv4.addresses '10.1.0.2/24'
nmcli connection modify vlan4022 ipv4.gateway '10.1.0.1'
nmcli connection modify vlan4022 ipv4.method manual
# this allows routing all 10.x ips through the vswitch gateway
nmcli connection modify vlan4022 +ipv4.routes "10.0.0.0/8 10.1.0.1"

# required to apply config
nmcli connection down vlan4022
nmcli connection up vlan4022

Notes

• Hetzner cloud volumes cannot be used as the csi driver won't work on robot servers, so you either need to use longhorn volumes or other external storage. This also means existing pods which use cloud volumes cannot be scheduled on robot nodes.
• Depending on the amount and size of other nodes in your cluster, you want to restrict which pods can be placed on it and think about whether you really want to use it for longhorn or just as a compute node. In this case you can set taints and node labels so that the node is not instantly filled up by pods and local replicas.
• For longhorn: open-iscsi must be installed on the robot server (including a systemctl start iscsid) otherwise the manager pods fail
• To prevent the hetzner csi pods to be scheduled on the node, one needs to apply the instance.hetzner.cloud/provided-by=robot label
• Important: while this setups works in my current cluster (including two hetzner LB), I don't know if it covers all network-related edge cases, private networks, non-wireguard clusters or other CNI libraries

[mysticaltech]

@pat-s k3s already supports helm charts automatically via rancher helm. See readme, and don't hesitate to try it. https://docs.k3s.io/helm

[pat-s]

It is not about helm per se in k3s but about integration into this module (as opposed to using kustomize).

[mysticaltech]

@pat-s a helm chart distribution of your solution would be ideal.

[pat-s]

Update here: I've been running with an AX42 happily for several weeks now. Even the metrics server is meanwhile working (I think after the HCCM 1.21 release).
Would be great to have cloud volume support at some point but with Longhorn it is doable with a bit of planning.

@mysticaltech I thought about putting more work in but I am somewhat afraid with all the custom glue between the kustomize and other resources in this module. #1575 was the starting point to allow helm HCCM deployments but it already got complicated there and caused issues (#1596). Given that you have more experience there, maybe you can take it from there? I have a working solution for myself but of course it would be great if others could also (easily) apply it some day.

[mysticaltech]

@pat-s Please don't hesitate to shoot a PR again, I will test it properly before merging.

[garry-t]

I went through discussion and still not clear , do we have working solution to join BR sever to cluster or no?

[pat-s]

sure, just follow #1792

[garry-t]

@pat-s just followed instruction you added in latest PR. The robot node wasn't added after start k3s agent. I use cilium network, can I same config as in doc or I need to change something or better use Flannel ? Thanks

[aamir-s18]

@garry-t We ended up using claudie.io (Its OSS) and our main cluster is on Hetzner Cloud but we have dedicated nodes from different providers in our kubernetes setup.

[pat-s]

@garry-t Can't help from the distance with that sparse information. I didn't use Cilium but I also can't say that this is the reason.

You need to inspect/debug the individual connections/components yourself and see where the issue resides. There's no way around doing a little bit of a deep dive to get this working.

[mysticaltech]

@pat-s Thanks for the help.

Folks, please see https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/docs/add-robot-server.md