Nodes with public IPv6 only

[laurentpellegrino]

I am new to kube-hetzner and this looks like a fantastic project.

While checking my first configuration file (kube.tf), I was thinking about the following.

Hetzner charges extra for a public IPv4 but nothing for a public IPv6. In the case we do not need a public IPv4 for nodes, I was wondering if there is a way to define in kube.tf that nodes in nodepools, or a nodepool should allocate nodes with no public IPv4, thus allowing to minimize pricing?

[mysticaltech]

@lpellegr It's a good idea, and would solve one long standing issue with some of the IPv4 being wrongfully blacklisted and unable to pull containers from gcr.

We already have partial IPv6 support for outside nodes, now if we can turn off IPv4 for the outside node IPs (not the private network) it would be ideal and would solve the aforementioned issue. Please don't hesitate to submit a PR if interested.

[laurentpellegrino]

@mysticaltech Thanks for your answer.

Looking at Terraform configuration files (that's my first time reading Terraform syntax!) it looks like the file at the following line requires changes:

https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/modules/host/main.tf#L32

Terraform hetznercloud provider docs describes how to select whether to assign an IPv4, an IPv6, or both upon server creation:
https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/server

We could add 2 new variables like without_ipv4 and without_ipv6 (e.g. following hcloud cli option names). In that case, they must be mutually exclusive. Their default value would be false.

I guess variable definitions would be there:
https://github.com/kube-hetzner/terraform-hcloud-kube-hetzner/blob/master/modules/host/variables.tf

This change would require extensive testing. Especially with without_ipv4=true.

I will see for a PR when I have more time.

[aleksasiriski]

I would argue that the option should be called public_network_stack and have 3 options: ipv4, ipv6 and dual. I'm interested in this as well, but remember that Hetzner doesn't have any kind of CGNAT or IPv4 over IPv6 in place, which results in dockerhub and ghcr.io images not being able to be pulled (since they still don't have support for IPv6). Maybe that could be mitigated using Cilium egress (if I'm not mistaken to how it works) or by just using gcr.io and caching required images to it. The best solution would be to have local Kubernetes registry cache that does have access to IPv4 (over a dedicated IP from Hetzner or perhaps an online proxy / VPN)

[mysticaltech]

Very interesting, thanks for the research @lpellegr.

It would need extended testing indeed. I agree with the naming proposed by @aleksasiriski, it's simpler.

If you feel inspired, please don't hesitate to give it a shot, I would love to review the PR.

[mysticaltech]

I am not sure if disabling IPv6 is needed. By default it's on and working well. So we could maybe have one boolean variable ipv6_only or public_ipv6_only. What do you folks think?

[mysticaltech]

Also, it seems that the only changes required would be all the SSH connections during setup. As for the rest, @aleksasiriski it turns out that docker.io supports IPv6 but setting k3s_registries is needed as such:

  registries:
    mirrors:
      docker.io:
        endpoints:
          - https://registry.ipv6.docker.com

[mysticaltech]

As for ghcr.io, it's not clear, we'll have to test. But definitely using the registry proxy as above would work even if it's not supported out of the box.

[LuigiClemente]

Implementing a Nebula Network could bridge limiting factors with Hetzner IPv6 only VMs

Implement a Nebula network with Terraform to address certain limitations and improve network connectivity within a Hetzner environment.

Addressing Limitations

IPv6-Only Environment

Hetzner provides IPv6-only virtual machines (VMs), which can pose challenges when dealing with IPv4-only resources or external network access. Nebula can help bridge the gap between IPv4 and IPv6 networks.

Limited External Connectivity

In an IPv6-only environment, issues may arise when trying to access resources that do not support IPv6, such as DockerHub or GHCR.io. Nebula can provide a private overlay network, potentially allowing servers to access external resources through a proxy or caching mechanism.

Implementing Nebula with Terraform

To implement Nebula with Terraform, follow these steps:

1. Nebula Configuration:

   • Define your Nebula network configurations, including certificates and keys. These configurations will be used to secure communication between nodes.

2. Terraform Resources:

   • Create Terraform resources for Nebula nodes. Define the necessary infrastructure components, including virtual machines, security groups, and network configurations.

3. Node Configuration:

   • Configure each server to use Nebula for networking. Ensure that the Nebula binaries and configurations are set up on your servers.

4. Network Isolation:

   • Use Terraform to define network isolation rules, allowing you to control which servers can communicate with each other over the Nebula network.

5. Testing and Validation:

   • Thoroughly test the Nebula network to ensure secure and reliable communication among your servers.

Benefits of Using Nebula

Implementing Nebula with Terraform offers several benefits:

1. IPv4/IPv6 Bridging:

   • Nebula allows servers with IPv4-only, IPv6-only, or dual-stack configurations to communicate seamlessly over a private network.

2. Security:

   • Nebula provides secure communication between nodes, helping protect your data and traffic, especially in untrusted environments.

3. Routing Control:

   • Nebula gives you control over routing and network isolation, allowing you to define complex network topologies tailored to your specific requirements.

4. Mitigating External Connectivity Issues:

   • Nebula can help mitigate challenges related to external network access in an IPv6-only environment. It enables private communication with external resources through proxy or caching mechanisms.

graph TD

subgraph "Addressing Limitations"
    subgraph "IPv6-Only Environment"
        A((Hetzner IPv6-only VMs))
        B(Nebula Bridge)
        C((IPv4-Only Resources))
        A -->|IPv6| B
        B -->|Nebula Overlay| C
    end

    subgraph "Limited External Connectivity"
        D((IPv6-Only Environment))
        E(Nebula Overlay)
        F((External Resources))
        D -->|Nebula Overlay| E
        E -->|Proxy/Caching| F
    end
end

subgraph "Implementing Nebula with Terraform"
    G["Nebula Configuration"]
    H["Terraform Resources"]
    I["Node Configuration"]
    J["Network Isolation"]
    K["Testing and Validation"]
    G --> H
    H --> I
    I --> J
    J --> K
end

subgraph "Benefits of Using Nebula"
    L["IPv4/IPv6 Bridging"]
    M["Security"]
    N["Routing Control"]
    O["Mitigating External Connectivity Issues"]
    L --> M
    M --> N
    N --> O
end

Loading

[mysticaltech]

Now supported, see docs.