Passing cloud-init data to node pools

[rombert]

Hi,

I'd like to execute some per-instance provisioning to enroll nodes in a Wireguard network. I know that:

• MicroOS supports cloud-init ( https://en.opensuse.org/Portal:MicroOS/cloud-init#openSUSE_Kubic_Kubeadm_Node )
• Hetzner supports cloud-init ( https://community.hetzner.com/tutorials/basic-cloud-config )
• cloud-init supports Wireguard ( https://cloudinit.readthedocs.io/en/latest/reference/modules.html#wireguard )

However, it seems cloud-init it not exposed for extensions in this module - https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/e79cfc775fc1fb6752b31c299487aa568f48390a/modules/host/templates/cloudinit.yaml.tpl .

Is this something that you would be willing to take as a feature request? I am also open to other solutions but all I could find was #483 and that did not seem to reach a conclusion.

Thanks!

[mysticaltech]

@M4t7e Maybe you could answer that one better! 🙏

@rombert Basically, anything that do not add too much complexity on top of the current code, and that is reasonably backward compatible, we are willing to consider. PRs always welcome.

[rombert]

Thanks for the reply @mysticaltech . This is very new territory to me so I'm not sure when a contribution might be ready. But at any rate - do you think it makes sense to contribute a specific enhancement for configuring wireguard via cloud-init? It would mirror what is available via https://cloudinit.readthedocs.io/en/latest/reference/modules.html#wireguard.

[mysticaltech]

@rombert You can test it if that's something you need. But not sure MicroOS actually support this cloud-init module.

[rombert]

Good point @mysticaltech - I'll try that first.

[rombert]

I manually created a new node from the same snapshot used for the MicroOS k3s nodes. I added the wireguard cloud-init definition to the respective text box. I don't see it being acted on in the instance though. Looking in the debug log I see

2023-08-24 16:39:54,775 - url_helper.py[DEBUG]: [0/61] open 'http://169.254.169.254/hetzner/v1/userdata' with {'url': 'http://169.254.169.254/hetzner/v1/userdata', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 2.0, 'headers': {'User-Agent': 'Cloud-Init/23.1.2-1.1'}} configuration
2023-08-24 16:39:54,777 - url_helper.py[DEBUG]: Read from http://169.254.169.254/hetzner/v1/userdata (200, 0b) after 1 attempts

but when I try and invoke it manually I get my own script (output truncated)

#cloud-config
debug: True
wireguard:
  interfaces:
    - name: wg0
      config_path: /etc/wireguard/wg0.conf
      content: |
        [Interface]

@mysticaltech - have one you seen something like this or do you maybe have an idea about how to debug?

[rombert]

Ugh, no idea how I missed that

Supported distros: ubuntu

The second option would be to enhance cloudinit_write_files_common and cloudinit_runcmd_common to also include user-supplied entries. These would then be used to

• write /etc/wireguard/wg0.conf
• run systemctl enable --now [email protected]

Testing this out

[rombert]

Edit: that worked just fine. @mysticaltech - let me know if you think the above enhancement makes sense and I will transfer it to an issue and (at some point) try and create a contribution if no one beats me to it.

[mysticaltech]

@rombert Really happy to see you were able to make it work. Now I would love to understand the use case behind this, for now it's not clear to me personally. Definitely, if it's something more people can use, it would be super nice to have a PR for this. @kube-hetzner/core FYI.

[rombert]

@mysticaltech - there are two scenarios I want to cover

Use cases

1. Secure remote access to kubernetes clusters

It's great that the terraform setup includes a firewall, that alleviates most of my concerns regarding network access. At the same time, I can't use an allow list for the control plane Kube API and for the SSH ports because I don't have stable egress IPs.

My solution is to block access completely and then connect using a VPN.

2. Integration with external services

There are integration points to the clusters I manage, e.g. LDAP or MySQL. There are not available over the public internet but only via a Wireguard VPN.

Alternatives

1. Manually configured 'jump host'

I could set up a trusted jump box in Hetzner with a fixed IP, enroll it in the VPN via cloud-init/Saltstack/whatever and configure its IP in the firewall.

That seems a bit more work :-) and also I am not sure how easy/secure a HTTPs proxy for the API server is. Also, increases cost.

2. Configuring wireguard on a per-pod basis

I can also allow access to wireguard to individual containers. Again, that is a bit more work and complicates key management - scaling up a deployment means needing more keys and I don't have the infrastructure set up for dynamic key management. Also, handing out different keys ( via secrets ) to instances of the same Deployment/ReplicaSet is fragile.

Hope that clears things up. Also, feel free to brainstorm more alternatives if you're so inclined, I'm just looking for an idea that makes the most sense for my use cases.

Thanks!