Allow egress ips on managedseeds

[modzilla99]

What this PR does / why we need it:

With this PR the Controller will automatically allow the egress IPs of the ManagedSeed. This will fix the blackbox exporter check that tries to externally access the shoots API.

Special notes for your reviewer:
The ConfigMap called shoot-info will be present in Seed clusters that are itself shoots. So it will only work on ManagedSeed(-Sets).

[hown3d]

Thanks for the contribution!
I'd like to propose some changes.

[hown3d]

I was just thinking about wether we should add a check to only add the seed CIDR if the blackbox exporter is actually deployed.
This is configurable in the gardenlet configuration if monitoring is enabled:
https://github.com/gardener/gardener/blob/master/pkg/gardenlet/apis/config/v1alpha1/helper/helpers.go#L59-L59

@timebertt WDYT?

[modzilla99]

I was just thinking about wether we should add a check to only add the seed CIDR if the blackbox exporter is actually deployed. This is configurable in the gardenlet configuration if monitoring is enabled: https://github.com/gardener/gardener/blob/master/pkg/gardenlet/apis/config/v1alpha1/helper/helpers.go#L59-L59

@timebertt WDYT?

I would not know how to implement this. How would I get a hold of the gardenlet configuration to check it?

[timebertt]

I'm curious:
I suspect that allowing the egress IP of the seed cluster for the blackbox-exporter is not required for all setups. Typically, in-cluster clients of a LoadBalancer service directly talk to the clusterIP (due to an optimization by kube-proxy). I.e., the ACL should already allow the blackbox-exporter to access API servers on the same seed in most cases.

Can you outline your setup in detail?

• What kube-proxy mode are you using?
• How is the istio-ingressgateway service configured (traffic policy, and load balancer IP mode)

[modzilla99]

We use a MangedSeetSet with Cilium as the CNI and we let cilium replace kube-proxy.

This is the seeds shoot config:

spec:
  kubernetes:
    kubeProxy:
      mode: IPTables
      enabled: false
  networking:
    type: cilium
    providerConfig:
      bpfSocketLBHostnsOnly:
        enabled: true
      hubble:
        enabled: true
      overlay:
        enabled: true
      tunnel: geneve

apiVersion: v1
kind: Service
metadata:
  annotations:
    loadbalancer.openstack.org/load-balancer-address: 1.1.1.1
    loadbalancer.openstack.org/proxy-protocol: "true"
    networking.resources.gardener.cloud/from-all-seed-scrape-targets-allowed-ports: '[{"port":15022,"protocol":"TCP"}]'
    networking.resources.gardener.cloud/from-world-to-ports: '[{"port":8132,"protocol":"TCP"},{"port":8443,"protocol":"TCP"},{"port":9443,"protocol":"TCP"}]'
    networking.resources.gardener.cloud/namespace-selectors: '[{"matchLabels":{"gardener.cloud/role":"extension"}},{"matchLabels":{"gardener.cloud/role":"shoot"}},{"matchLabels":{"kubernetes.io/metadata.name":"garden"}}]'
    networking.resources.gardener.cloud/pod-label-selector-namespace-alias: all-istio-ingresses
    resources.gardener.cloud/description: |-
      DO NOT EDIT - This resource is managed by gardener-resource-manager.
      Any modifications are discarded and the resource is returned to the original state.
    resources.gardener.cloud/origin: istio-system/istio
  labels:
    app: istio-ingressgateway
    app.kubernetes.io/version: 1.27.1
    istio: ingressgateway
    resources.gardener.cloud/managed-by: gardener
  name: istio-ingressgateway
  namespace: istio-ingress
spec:
  allocateLoadBalancerNodePorts: true
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: tcp
    port: 443
    protocol: TCP
    targetPort: 9443
  - name: tls-tunnel
    port: 8132
    protocol: TCP
    targetPort: 8132
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 1.1.1.1
      ipMode: Proxy

[timebertt]

Got it, thanks for outlining your use case. In that case, you do need this particular change in the ACL extension 👍
Can you add some more context to the code comment explaining in which case the egress IP of the ManagedSeed needs to be allowed?

[hown3d]

I see, should we make this configurable per shoot then? If it's only necessary on these special cases I believe we shouldn't punch in another hole by default.

[modzilla99]

It feels a bit hacky, but I could also check whether the cilium-config exists in the kube-system namespace. If the kube-proxy-replacement is set to true, get the egress IPs.

[hown3d]

It feels a bit hacky, but I could also check whether the cilium-config exists in the kube-system namespace. If the kube-proxy-replacement is set to true, get the egress IPs.

We can check all of that in the shoot itself, no need to check for some ConfigMaps.

I think there are some options we can do:

1. Make it configurable in the ACL provider config to add the Seed Egress IP (e.g. allowSeedEgressIP: true)
2. Check if Cilium is used as the network provider and kubeProxy is disabled in the Shoot object.

[modzilla99]

Check if Cilium is used as the network provider and kubeProxy is disabled in the Shoot object.

But this would then only check if the child Shoot is using Cilium and not the Seed cluster, which is what actually matters. Or did I misunderstand you?

[hown3d]

It is possible for extension to get a garden client to read shoots directly from the garden.
https://gardener.cloud/docs/gardener/extensions/garden-api-access/

This way it would be possible to get the configuration of the Seeds Shoot.