Configuring ClusterId via Custom Resources: Exploring Options

[aswinayyolath]

Strimzi generates a unique clusterId for each Kafka cluster it manages. This identifier can be retrieved using the kubectl get kafka command, where you'll observe a randomly generated string by the Strimzi operator.

kubectl get kafka my-cluster-2 -n strimzi-2 -o=jsonpath='{.status.clusterId}'

Similarly, you can obtain the clusterId for Kafka node pools with:

kubectl get kafkanodepool my-pool-2 -o=jsonpath='{.status.clusterId}'

Now, let's discuss the possibility of overriding the clusterId with a user-defined value via Strimzi's CR configurations. Is this feature currently supported?

This will be useful for scenarios where you want more control over the cluster id. One such example could be creating stretch clusters where all brokers across different Kubernetes environments must use the same cluster id.

[scholzj]

The cluster ID is generated as unique. So I'm not sure what is the exact problem you are trying to solve.

[aswinayyolath]

Sure, let me elaborate. I was conducting an exploratory spike to delve into the process of creating a logical stretch cluster. Here's what I did:

• Set up 2 Kubernetes clusters.
• Installed Kafka with KRaft and Nodepool enabled.
• Temporarily halted the operator reconciliation using kubectl annotate Kafka <clustername> strimzi.io/pause-reconciliation="true".
• Edited the ConfigMap generated by Strimzi for each broker/controller in both clusters to utilize the same clusterId. This involved adjusting the controller.quorum.voters property in both clusters to include information about Pods running on both Kubernetes clusters.
• Initially, for the spike, we disabled security and made several other alterations to establish a logical cluster.

As I proceeded, I started thinking about a method to enable users to configure the clusterId via Custom Resources, allowing them to specify it in the Kafka Custom Resource.

I would greatly appreciate your insights and perspective on this topic.

[scholzj]

Well, that is fine. But that does not add value to anyone else because you are basically just hacking around. So I do not think it justifies any effort to support this.

Leaving that aside, I'm not sure how would that be configured because it is not really ocnfigurable beyond the initial deployment. So having an option for it that will be ignored all the time apart from the initial deployment would be probably very confusing.

But what ou can try is to edit the status of the Kafka resource when you create it and set your desired cluster Id there.

[aswinayyolath]

Thank you for your thoughtful response and insights. I completely understand your perspective. My intention was to explore the manual adjustments required initially, with the aim of potentially incorporating them into the operator logic later on. I agree that having an option for configuring only during the initial deployment could indeed lead to confusion. Additionally, I'm fully aware that implementing stretch clusters involves considerable effort and considerations. Your guidance is invaluable in shaping our approach moving forward. Thank you for your patience and assistance in this matter.

[siegenthalerroger]

So after running in to an issue with our KRaft cluster, we ended up needing to restore our cluster from a backup. Our B/DR system uses velero to backup our PVCs & PVs and relies on FluxCD to restore the k8s resources to the same point in time as the backup was made. This implies that strimzi's Kafka CR would be deterministic in it's creation of a Kafka resource.

However this is not a given due to the fact that the clusterID is randomly generated. This makes restoring a strimzi Kafka cluster from data disks a painful process with a race-condition after FluxCD applies the Kafka CR (between the user setting the clusterID within the Kafka.status subresource and strimzi's cluster-operator starting the cluster).

I'm not sure what the benefit of having the clusterID be random is tbh and would really like to see the Kafka CR spec expanded to allow manually setting a clusterID value. An alternative could also be to change the operator to not generate a random ID but rather something deterministic based on the name, possibly requiring the configuration of a random-seed strimzi-wide. Is there any reason not to allow manually setting this value, seeing as you currently can do it by directly manipulating Kafka.status which just seems like an absolute hack tbh.

---

Slack conversation about restoring and the clusterID problem: https://cloud-native.slack.com/archives/CMH3Q3SNP/p1720463076090579?thread_ts=1720455627.117929&cid=CMH3Q3SNP

[a1exus]

How does a read-only field help if you set a wrong cluster ID?

Look, I understand the problem you have. But there is a workaround you can use today. And any solution has to be usable and maintainable for everyone. Not just for you.

@scholzj - would you please describe such a workaround? or if you can point into the right direction, that'd also help) thanks in advance!

[siegenthalerroger]

@a1exus check out the linked slack convoy for the full details. You can manually set the clusterId by applying it to the Kafka.status.clusterId field. You'll need to explicitly tell kubectl to apply to the status subresource and temporarily pause any automatic reconciliation you have running (strimzi & any further like fluxcd).

[a1exus]

@a1exus check out the linked slack convoy for the full details.

i tried to access it, however

my email doesn’t have an account on this workspace.

[scholzj]

You can register for the CNCF Slack here: https://slack.cncf.io/

But in general, you can set the cluster ID in the .status section of the Kafka CR. You can do it for example with something like kubectl edit kafka my-cluster --subresource=status.

[jwijgerd]

Hi folks, I am running into this issue now where I want to be able to recreate my dev cluster (this is running on Google Cloud, so to save costs I want to shut it down when the devs are not working) but I want to retain my data. So I have separate Google Compute Disks and I precreate the PVs an PVCs before creating my NodelPool and Kafka CRDs. However since Strimzi creates a new cluster.id my kafka Pods are crash looping. Updating the cluster.id property in the broker config maps makes the cluster operational again. However since this (imho) is a valid usecase (in development environments) it would be great if this was supported in the CRD(s). Now I have to find a way to get my Terraform code to update the cluster.id (either in the generated config maps or as you mention in the cluster status). So here's my +1 in adding this to Strimzi proper.

[cashol]

Hi.
I also run into a problem relating to clusterID, when redeploying kafka cluster (using KRaft) after first deployment:
Exception in thread "main" java.lang.RuntimeException: Invalid cluster.id in: /var/lib/kafka/data-0/kafka-log0/meta.properties. Expected cfuiDEZDSK6r10UaZL3Idw, but read 9FlnsttMTUerIhRLoEiZ4g
I am using Docker Desktop with Kubernetes enabled on Windows 10 Prof.

[cashol]

Workaround:
Remove meta.properties from all kafka related PVCs, e.g. data-0-kafka-cluster-broker-0 etc.

BTW: On Docker Desktop, deployed via docker-compose (confluent images used), this way does NOT help!

[sahmad-d2x]

IMO this is such a glaring hole/blockers in migrating to kraft(when zookeeper is to be deprecated in strimzi 0.46). It is often quite easy to delete deployments from k8s (people fat-fingering commands). The only reliable way to persist information IMO (in kubernetes or without it) is using reliable block storage. I was expecting that deploying (termed as recreating the cluster earlier in the thread) would mean kafka brokers would read their clusterId from the underlying block storage (i.e kubernetes persistent volume/persistent volume claims) but that is not the case with strimzi/kraft. Even after retaining the persistent volume claims, the clusterId changes rendering the data on the disks useless and a broken cluster.

Also I think keeping a critical cluster attribute (i.e. clusterId) inside a kubernetes resource is a bad design. That means I can't reliably restore kafka just from the backup of the drives? I need to save the instantiated kubernetes crd resources too?

[erikdahmen]

Hi, we also experience the Invalid cluster.id issue and it indeed only happens in KRaft based clusters.

We operate several test environments which we stop when they are not needed. Stopping means deleting the Kafka and KafkaNodepool resources and only retaining the PVCs. Starting means deploying the Kafka and KafkaNodepool resources and let the operator spin up the cluster.

In case of Zookeeper based clusters, the operator is able to find the original ClusterID and set it in both Kafka and KafkaNodepool resources.
In case of KRaft based clusters, the operator assigns a new ClusterID, causing the cluster to fail.

We currently observe this with operators 0.44.0 and 0.45.1 and Kafka 3.8.0.

It would be great if the operator was able to obtain an existing CLusterID also for KRaft based clusters.

[scholzj]

In case of Zookeeper based clusters, the operator is able to find the original ClusterID and set it in both Kafka and KafkaNodepool resources.
In case of KRaft based clusters, the operator assigns a new ClusterID, causing the cluster to fail.

Please keep in mind that this is not a Strimzi choice. This is a difference between ZooKeeper and Kraft that comes from Kafka itself.

We operate several test environments which we stop when they are not needed. Stopping means deleting the Kafka and KafkaNodepool resources and only retaining the PVCs. Starting means deploying the Kafka and KafkaNodepool resources and let the operator spin up the cluster.

That is not how you stop or start it. You should do the following:

1. Pause the reconciliation of the Kafka CR using the anotation
2. Once it is paused, delete the Strimzi Pod Sets
3. To restart the cluster, just unpause the reconciliation

Following this procedure should also help you to avoid any Cluster ID issues.

[erikdahmen]

Thanks for the quick response. Do I understand correctly that the operator is able to fetch the original ClusterID from Zookeper but not from the KRaft controllers?

[scholzj]

No, we can fetch it in both cases only from a running KAfka cluster (in case of ZooKeeper, you can also query it in the ZooKeeper database from a running ZooKeeper cluster, but that is not the major issue and we do not use it for anything). The main change is that with ZooKeeper, Kafka creates the cluster ID on its own and keeps it in ZooKeeper. With KRaft, you need to format the storage with the cluster ID first (see for example here: https://kafka.apache.org/documentation/#quickstart_startserver). But what looks simple in the Quickstart with s single node is harder in a distributed environment with multiple nodes as this needs to happen on all nodes and the cluster ID needs to be the same on each of them.

So this forces us to generate the cluster ID from the operator for new clusters. However, if it is an existing cluster that already has the cluster ID stored at its volumes, you cannot generate a new cluster ID in the operator and force it on it. But you cannot easily read the cluster Id from the cluster tht is not running either. And that is pretty much the situation where your procedure for stopping the Kafka cluster leads.

[erikdahmen]

Got it. Thanks for the detailed explanation. We will find a way to work around it.

btw.

kubectl patch kafkanodepools controller --type=merge --subresource status --patch 'status: {clusterId: original-cluster-id}'

did the trick for us. It was even enough to patch only the controller node pool. The broker node pool got updated automatically.

[skupjoe]

I faced a similar issue. If anyone else is having this problem of restoring Zookeeper pods/data to a previous Kafka cluster I can go into more detail, but it is possible to set the .status.clusterId to the old cluster at boot and then pause reconciliation to intercept and connect ZK before the cluster starts and then update the /cluster/id also to the old cluster and then unpause the reconciliation to let kafka initialize fully.

(In my case I had lost the majority of my old ZK cluster data but retained 2:5 PVs (Id's "5" and "6" in /disk/data/myid) and I was still able to use these to restore my old zk cluster- I just used a test pod before anything to connect to the old zk PVs and adjusted the individual zk node ids in /disk/data/myid to from 5 and 6 to 1 and 2 to match for the new instances and then I was able to "re-grow" my cluster in a new 3-node cluster (3rd node fresh))

[koteshwar96]

I’m trying to create a brand new Kafka cluster in KRaft mode using the Strimzi operator and ArgoCD for deployment. I’ve specified a custom clusterId under .status.clusterId in both the Kafka CR and the KafkaNodePool CRs (for broker and controller), but after syncing via ArgoCD, the cluster spins up with a random clusterId instead of the one I provided.

As per the solution suggested in this github issue, I was wondering at what point I should annotate the Kafka CR with strimzi.io/pause-reconciliation="true" if I want the cluster to retain my specified clusterId when i creating my cluster from scratch. From what I gather, reconciliation needs to be paused so that my clusterId isn’t overwritten, but I’m unsure how to align that with ArgoCD’s sync process — it always ends up showing a drift and the cluster uses a new ID.
Would really appreciate any guidance or tips you can share on this. Thanks in advance!

[rivolity]

It will be a good Idea to add the clusterId at the deployments, just like that you can deploy several time without loosing the cluster, since Data are stored in the PV and cluster ID is already mounted in the configMap.
@im-konge

[scholzj]

The cluster ID cannot be configured once it is set. So, no, it cannot be really configurable because we would not be able to change it. (Also, unless you are doing some sort of disaster recovery, you do not need to set a cluster ID. And if you do some disaster recovery, the effort to set it in the status is negligible.)

[aswinayyolath]

The .status.clusterId field in both the Kafka and KNP CRs is managed by the Strimzi Operator. It reflects the actual cluster.id that gets generated and set by Kafka itself when the cluster starts up for the first time. This field is not meant to be user-defined or pre-populated, and any values placed there by users will be ignored or overwritten during reconciliation.

When I originally started this discussion sometime back, it was during an exploratory spike around Stretch Kafka clusters, where brokers and controllers are deployed across multiple Kubernetes clusters. At the time, I was investigating whether manually setting a shared cluster.id could help coordinate a cross-cluster deployment. However, after deeper experimentation and improvements to the operator code (on a fork), it became clear that setting a custom cluster.id via the CR is not needed, even in stretch cluster scenarios, the Operator handles this correctly during initial bootstrapping.

Regarding the use of the annotation strimzi.io/pause-reconciliation="true", This annotation temporarily disables reconciliation by the Operator. While it can be useful in specific low level debugging or other use cases, it is not recommended for normal workflows, especially when working with GitOps tools like ArgoCD. Pausing reconciliation will stop the Operator from managing your cluster, meaning any changes you make (including syncing CRs from ArgoCD) will not take effect until reconciliation is resumed. This often leads to drift and confusion in GitOps workflows.

As maintainers pointed out, the clusterId is automatically generated during cluster creation and can't be configured or overridden. Once set, it remains fixed, which is necessary for consistency and correct operation of the cluster.

[koteshwar96]

Thank you all for sharing your inputs.
I was hoping there might be a way to set a custom clusterId right when the Kafka cluster is created for the first time — this would help avoid the need to modify manifests and toggle pause-reconciliation during cluster recreation.

I do understand the challenges with exposing clusterId as a configurable field, especially since it can’t be changed once the cluster is initialized. Similar to how some cloud Terraform providers handle immutable fields by recreating resources when those fields are updated, I was wondering if a similar approach could be feasible here.

That said, I completely understand that the range of use cases and the effort required to support this might not justify the feature at this time.

[scholzj]

Well, if you really want to do it, it can be done as discussed in the earlier comments and basically as you suggested:

1. Create the Kafka CR in a paused state
2. Set the cluster ID in the .status section
3. Unpause the cluster

I think the issue you have is that to set the Cluster ID in the status, you need to patch the status. You can do it for example with kubectl edit kafka my-cluster --subresource=status as discussed in one of the previus discussions.

The question you need to figure out is how to do it in ArgoCD. I assume that ArgoCD will by default ignore the status subresource and thus not set the cluster ID. Also, the steps and their order in which they are done matters. I have no idea if (and how) ArgoCD can do this. But that is something you will need to check in Argo docs (or discussion forums etc.).

But also as I said ... I'm not aware of any reason why you would need to set the cluster ID other than a disaster recovery from existing disk backups. And in such case, the process is more complicated, so you would typically first manually recover it (e.g. using some script, kubectl commands, and so on) and only once the cluster is recovered and running you would hand it over to ArgoCD.