Pod cidrs are not configurable in most ClusterStacks including openstack scs one

[Nils98Ar]

/kind bug

What steps did you take and what happened:

Currently the pod cidrs are not configurable in most ClusterStacks (except the Hetzner one as it seems).

The reason is that by default (ipam mode cluster-pool) cilium uses its own ipam settings for pods and does not respect the Cluster objects .spec.clusterNetwork.pods.cidrBlocks described in the docs:

• cluster-stacks/docs/providers/openstack/quickstart.md

  Lines 203 to 205 in c48b6bc

  	pods:
  	cidrBlocks:
  	- ${CS_POD_CIDR}

• cluster-stacks/docs/providers/openstack/configuration.md

  Lines 17 to 19 in c48b6bc

  	pods:
  	cidrBlocks:
  	- 192.168.0.0/16

We should either switch to ipam mode kubernetes and let cilium use the K8s ipam configuration as in the hetzner ClusterStack or we should make .spec.ipam.podCIDRs in the CiliumNode objects configurable via ClusterClass variable. Propably also configurable via cilium helm value ipam.operator.clusterPoolIPv4PodCIDRList in the ClusterAddon but I think the helm template happens before the ClusterClass templating/patching and therefore we cannot patch helm values this way?

We would also need to check how to deal with existing clusters.

[jschoone]

Thanks for the issue.
I guess the best and easiest way to solve this using Cluster Stacks is by setting IPAM mode to kubernetes.

[Nils98Ar]

@jschoone Do you think that upgrading to a new cluster stack version with the “Kubernetes” PAM mode is sufficient for upgrading existing clusters? Or could manual steps be necessary?

[jschoone]

@jschoone Do you think that upgrading to a new cluster stack version with the “Kubernetes” PAM mode is sufficient for upgrading existing clusters? Or could manual steps be necessary?

I don't know. Will test asap.

[jschoone]

@jschoone Do you think that upgrading to a new cluster stack version with the “Kubernetes” PAM mode is sufficient for upgrading existing clusters? Or could manual steps be necessary?

Had no luck yet, but have some things left I can test.
For now I did the following.
I ported the SCS Cluster Stacks to Docker provider for quicker testing. Then I created a Cluster Stack for 1.30 without ipam.mode="kubernetes" and on for 1.31 with it enabled, because the CSO needs a new Kubernetes version to do anything.
But then I think I faced #202, because I saw the value of the IPAM mode was not updated. I checked the content of the Cluster Stacks inside the CSO pod and saw the value was indeed set to ipam.mode="kubernetes" it was just not applied.
So I changed the cluster-addon to make use of Multi Stage Addon to trigger the apply with the BeforeClusterUpgrade hook, see https://github.com/SovereignCloudStack/cluster-stacks/tree/refactor/modernize_scs_cluster_stacks/providers/docker/scs (part of #203
From that I created again two Cluster Stacks:
registry.scs.community/kaas/cluster-stacks:docker-scs-1-30-v0-custom-noipam
registry.scs.community/kaas/cluster-stacks:docker-scs-1-31-v0-custom-ipam
the latter is not visible in branch yet, the differences are only the Kubernetes version and cilium.ipam.mode="kubernetes" in the cni/values.yaml

So I applied the 1.30 with a Cluster resource without any Cluster.spec.clusterNetwork at all. As expected it worked because Cilium just does its own thing.
Then I changed the Cluster resource to update to the 1.31 ClusterClass and use a clusterNetwork config like this

spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - 192.168.20.0/24
    serviceDomain: cluster.local
    services:
      cidrBlocks:
        - 10.96.0.0/12
  topology:
   class: docker-scs-1-31-v0-custom.ipam
   version: v1.31.6

The upgrade worked without any issues, also the Cilium settings were updated. Then I also did this and ended up with Pods in Pending state because they didnt' get an IP address which is likely because the cidr couldn't be update during runtime.

It looks like if Cluster.spec.clusterNetwork is configured from beginning everything works fine. I see the Pods in the specific CIDR.

@Nils98Ar did you already specified the clusterNetwork? Then I don't see any problems here

[Nils98Ar]

Yes, I have specified .spec.clusterNetwork.pods.cidrBlocks but it was simply ignored bause of ciliums ipam.mode.

So changing the ipam.mode to "kubernetes" should not be a problem in this case?

[jschoone]

Yes, I have specified .spec.clusterNetwork.pods.cidrBlocks but it was simply ignored bause of ciliums ipam.mode.

So changing the ipam.mode to "kubernetes" should not be a problem in this case?

Yes with a node rollover or in combination with a Kubernetes upgrade which does the rollover anyway