[Feature Request]: Reserve some count public IPv4 / IPv6 for whitelisting

[CroutonDigital]

Description

On GKE in our company we use next case:

We reserve 10-20 IPv4 but cluster use only 4 IPv4, but when autoscaling add new cluster node it take from reserve IPv4.
That need when service communicate with external service where need add IPv4 to whitelist. And need to be add all IPv4 nodes. And when k8s autoscaling up or down we don't have issue with access to external services.

Also we have stateful service where run on each k8s node for balance requests to the external services, count stateful services depends with count running pods in cluster.

I think that case happened often on other projects where services communicate with externals SaaS platforms.

Thank you!

[CroutonDigital]

We deployed k8s on hetzner and move services from GKE but we need access to Google secrets.
And some cluster nodes got IP from iran. We discuss with Hetzner support and they saying can't verify own IP database.

I add additional 2 nodes use terraform and remove 2 nodes with iran IPs. But now I have problem with terraform update.

  # module.kube-hetzner.module.agents["1-5-agent-large"].random_string.server will be destroyed
  # (because module.kube-hetzner.module.agents["1-5-agent-large"] is not in configuration)
  - resource "random_string" "server" {
      - id          = "kqm" -> null
      - keepers     = {
          - "name" = "h-k3s-staging-agent-large"
        } -> null
      - length      = 3 -> null
      - lower       = true -> null
      - min_lower   = 0 -> null
      - min_numeric = 0 -> null
      - min_special = 0 -> null
      - min_upper   = 0 -> null
      - number      = false -> null
      - numeric     = false -> null
      - result      = "kqm" -> null
      - special     = false -> null
      - upper       = false -> null
    }

Plan: 6 to add, 0 to change, 14 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

Better need solution for manages IPv4 / IPv6 for reservation.

============
Dear Client,

Thank you for your request.

Unfortunately, some GeoIP databases incorrectly locate some of our IPs in Iran. We cannot influence these databases.

If this leads to issues for you, please create a Snapshot of the server with the incorrect IP location. Then create a new server with this Snapshot. You can then delete the server with the incorrect IP location.

Kind regards

Kevin-Damian Gosa

Hetzner Online GmbH
Industriestr. 25
91710 Gunzenhausen / Germany
Tel: +49 9831 505-0
Fax: +49 9831 505-3
www.hetzner.com

========
Dear Client,

thanks for your message.

Unfortunately such a temporary blockade is unfortunately beyond our control, since this is a decision by the respective service provider.

There is a workaround to change the IP address of your Cloud-Server. To do this, create new servers before deleting the old ones. (If you have data on the old server, take a snapshot before deleting it). Each server should have a new IP address. Just keep the server that is not blocked.
Optional:
To restore the snapshot of your old server, click on the new server and select the 'REBUILD' section.
You should find the snapshot at the bottom of the Image drop down list. Select it and click REBUILD.

Should you require any further assistance, please do not hesitate to contact us.

Kind regards

Tim Stich

Dear Client,

thanks for your answer.

Unfortunately, I have to inform you that it is not offered to remove any IPs from the rotation. In principle, these will be reserved for you for 24 hours, after which alternatives will be issued.

Kind regards

Tim Stich

Hetzner Online GmbH
Industriestr. 25
91710 Gunzenhausen / Germany
Tel: +49 9831 505-0

===================

[mysticaltech]

@CroutonDigital Yes, this is a real problem with Hetzner IPs. What you could do is use the cilium egress gateway. If you egress from a node that is clean, normally you won't have any issues. Please try it. See kube.tf.example.

[mysticaltech]

@CroutonDigital There is another trick, it's to use reserved IPs to check. Basically hetzner allows you to reserved IPs, you grab one, and you try to pull a container from google registry (registry.k8s.io, any container will do it does not matter, or maybe just an HTTP request), if it fails, you keep the IP reserved, if it succeeds you free it. What will happen is that the free IPs will be now just for you to use (that's how Hetzner works, I do not know how long it stays for you, but at least a few hours). So you find as much free IPs as you need to deploy your cluster. Then you deploy, and released all iran IPs that you found and kept reserved. Please if you managed to write a script that does just that, PR most welcome!

[CroutonDigital]

Hi,

Some our services in k8s make more requests to external services where need balance requests. service on k8s run on each node and when start get Secrets from Google Secrets.

I make blacklist but not easy workflow.

• I create cluster and check external ips on node;
• Remove cluster node with wrong IPs;
• Make reservation IP use rename IP;
• and make terraform apply for created removed k8s nodes with other IPv4;

but some issue when cluster auto add new k8s nodes, and now I plan setup pod distribution for services who access to Google Secrets run on core k8s node.

Can I use egress with multiply IPv4?

Thank you!

[mysticaltech]

I think cilium egress can work with multiple nodes/ips, of course.

[sharkymcdongles]

Can it? I am trying to solve this myself right now because I had an issue creating a GitLab due to the image pulls issue from GHCR. I actually had to make a PR for Trow to allow enough segments for my use case: Trow-Registry/trow#355

Once this was done I had a floating IP node that had an IP that worked for pulls from GHCR. I then created an CiliumEgressGatewayPolicy that used the floating IP from the floating IP node for all outbound calls. This was great until the node had an issue randomly and entered NotReady state.

I am now trying to see how I can either failover the floating IP between two nodes or have 2 whitelisted floating IPs configured for the CiliumEgressGatewayPolicy. Unfortunately it seems there's no way to do this:

In case multiple nodes are a match for the given set of labels, the first node in lexical ordering based on their name will be selected.

So now I am thinking the only way to do this is to have some sort of pod that checks if the node is healthy and if not patches the CiliumEgressGatewayPolicy label to the other node with a whitelisted IP. This kind of sucks and is hacky though.

Have you come up with anything or have any ideas? As for Trow, it works great for this, and I plan to add it to the hcloud-kube-hetzner project once the official release is made with my PR.