issue with outbound connections?

[Gilaco]

Hey,
I wonder if any of you encountered issues with outbound networking blocking ports?
I created a simple cluster with few worker nodes - trying to connect to external service mongo-atlas (on port 27017).

I did set a FW rule to allow all outbound TCP connections on any external ports - still, on some of the machines nodes created - the mongo atlas is not reachable... and in others it is....
this issue is very very frustrating as I couldnt get to the root cause why it happens and I dont have a clue how to solve it.

I read online that sometimes on the openSUSE, there is some issues with high ports being blocked but nothing conclusive towards a solution...

really hope that someone here has experience with such incidences and can help...

the problem is on the node-level, only high number ports like the mongo 27017 port...

looking forwards to some feedback,

thanks :-)
Gilad

[Thijmen]

Is this related? #452

[Gilaco]

Exactly the same... only the solution doesn't work for me.

I have a FW rule to allow outbound for all ports and still some (only some - ands that's the frustrating part) nodes don't allow outbound to high ports, specifically mongo 27017

Any help is appreciated

Thanks
Gilad

[Thijmen]

How do you know 100% sure it's Hetzner blocking it? Have you verified that other clients can connect to that node?

[Gilaco]

I dont know… just wanted to learn if you can think of anything that might help resolving this Question: When applying a FW rule, does it applying immediately once deployed or requires a machine restart?

…

On Mon, 9 Jan 2023 at 21:10 Thijmen Stavenuiter ***@***.***> wrote: How do you know 100% sure it's Hetzner blocking it? Have you verified that other clients can connect to that node? — Reply to this email directly, view it on GitHub <https://github.com/orgs/kube-hetzner/discussions/494#discussioncomment-4637051>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO6HSYMFC6C65HN4VCHZDTWRRPCTANCNFSM6AAAAAATUYOWII> . You are receiving this because you authored the thread.Message ID: <kube-hetzner/terraform-hcloud-kube-hetzner/repo-discussions/494/comments/4637051 @github.com>

[Thijmen]

I believe a firewall rule is being applied on the Hetzner network, so no reboot required. Have you validated that your rules are in the Hetzner cloud console?

[mysticaltech]

Yes, the firewall is being applied at the hetzner network level. There are no firewalls at the node level.

[Gilaco]

Yes I did.. check the FW settings in the cloud ui I will check the iptables on the node itself.

…

On Mon, 9 Jan 2023 at 21:17 Thijmen Stavenuiter ***@***.***> wrote: I believe a firewall rule is being applied on the Hetzner network, so no reboot required. Have you validated that your rules are in the Hetzner cloud console? — Reply to this email directly, view it on GitHub <https://github.com/orgs/kube-hetzner/discussions/494#discussioncomment-4637115>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAO6HS6IEHALEFS72UVFKXDWRRP4HANCNFSM6AAAAAATUYOWII> . You are receiving this because you authored the thread.Message ID: <kube-hetzner/terraform-hcloud-kube-hetzner/repo-discussions/494/comments/4637115 @github.com>

[mysticaltech]

There are nothing at the node level, no firewall running there. It's all happening at the hetzner network level.

[mysticaltech]

@Gilaco Please share you kube.tf, stripped of comment and sensitive values please.

[mysticaltech]

@Gilaco Please be more specific in your extra fw rule, for instance:

  {
     direction = "out"
     protocol  = "tcp"
     port      = "27017"
     destination_ips = [
       "0.0.0.0/0"
     ]
   }

And also, move always from calico, just comment that cni_plugin line, so that flannel is used by default. I suspect that is what is causing it, as it might expect you to set network policies.

[Gilaco]

I typically used flannel, I tried now calico to see if different cni might fix... and same behaviour
I'll revert back to flannel and test the explicit in the FW rule.

still - what bugs my mind is why some nodes block this port and others not... specifically on a pool of 1 node, that node is blocked (re-created cluster 3 times and same behavior)

I'll try your suggestion, thanks

[Gilaco]

@Gilaco have you by any chance seen if the IP-addresses are the same for the nodes that share the problem? I have noticed, that hetzner often "reuses" the address if a server is recreated. This could indicate that the ip of the node is the problem as it could occur on some blacklists

will check it!
in any case on mongo-atlas I removed all access restrictions so I can connect from anywhere (tested)

[Gilaco]

@mysticaltech well - tried it on exiting cluster and didnt work.
created a fresh new project in HZ and re-deployed all - all is working (added your suggeted explicit rule + flannel)
I assume that it was a little bit of everything... perhaps the fact that I was destroying and recreating on same project kept some resournces on HZ side... dont know.

on fresh project - so far looking good.

thank you all ! :-)

[mysticaltech]

Good to hear @Gilaco! Definitely, sometimes it's important to start fresh :)

[mysticaltech]

Be more specific in your extra fw rule, for instance:

  {
     direction = "out"
     protocol  = "tcp"
     port      = "27017"
     destination_ips = [
       "0.0.0.0/0"
     ]
   }

And also, move always from calico, just comment that cni_plugin line, so that flannel is used by default. I suspect that is what is causing it, as it might expect you to set network policies.