Strimzi
Replies: 12 comments 30 replies
-
|
I'm not sure I follow the problem. You tal about some cluster named |
Beta Was this translation helpful? Give feedback.
-
|
Hi. Thank you for replying. Its still creating replicasets like crazy for the entity-operator: ^Those are all elsec-feeds-entity-operator-... On a side note: why is it in the 'kafka' NS (where I have zk and kf) and not in the 'strimzi' NS where the cluster operator lives? (I don't know if that is another symptom or not) After over a day the entity-operator is still not created/running - no idea why. As the the rolling restarts of zk and kf, I will restart the cluster operator with debug logging - however note nothing is interacting with this environment yet, I only just created it. So what is the change that is causing the infinite restarts? |
Beta Was this translation helpful? Give feedback.
-
|
So I decided to delete is all and start again (following: https://strimzi.io/docs/operators/latest/quickstart.html). For reference: Then Then the cluster: => So notes on what's happening in between:
Topic: => ~10:50 GMT I was going to say: It seems this time the entity-operator came up ok, its not creating loads of empty/dead replicasets: However before I finished to post this, I looked again and: So back to square 1 on my first issue... For the brief time it was running it did create my test topic: Also after all that, it is still deleting/restarting the kf and zk pods... so my second issue is also here to stay.. I hope with all this you or anyone else has enough information to help - it would be so greatly appreciated. I was capturing the debug cluster operator logs (stopped log capture at 11:04 GMT): In any case, thank you for taking the time to look at this. |
Beta Was this translation helpful? Give feedback.
-
|
It looks like some tooling in your cluster is injecting all kind of additional information and modifying the statefulsets directly: It modifies the SecurityContext or annotations. That is forcing the rolling updates of Kafka and ZooKeeper and also generating the new replica sets for the Deployments. You have two options how to deal with it:
|
Beta Was this translation helpful? Give feedback.
-
|
Thank you for looking at this and for the response. If I understand correctly Strimzi and Autopilot are not getting along? This how I created the cluster: Following: https://cloud.google.com/kubernetes-engine/docs/quickstart#create_cluster - After those two commands I can use kubectl. Then I moved onto the Strimzi getting started - I did nothing more at the cluster create part (yet). So the first option "Disable whatever is doing this from doing it" says to me: You can't use GKE-Autopilot The second option is (TBH) starting to take me out of my depth, I will have to take some time to digest it. I use to run ZK and KF managed via my own Ansible playbook. Having discovered Strimzi I thought I'd give it a go. Should/Could I open an issue for out-of-the-box support of Strimzi on GKE-Autopilot? Will the template route really work if Autopilot is making annotations? Can I really know such in advanced and add them to strimzi definitions? Would it not be better to be able to list things that Strimzi should ignore, by adopting the new value automatically? |
Beta Was this translation helpful? Give feedback.
-
|
TBH, I don't know what Autopilot is and how exactly it works. So I cannot easily comment on it. The In general, it looks like you have multiple components in your cluster which want to manage the same resources and want them differently configured. And that does not work because they would fight over it and changing it in a circular loop. Using the
Well, you use the auto-pilot ... so if you wanna help to make it work out of the box it, we can work with you on it. But we do now know the auto-pilot feature, so not sure just opening an issue which says make it work would be helpful on its own.
You would certainly need to try it out, I do not know it for sure. You are using the auto-pilot tooling, so you should probably know what it does and how it works. You will probably need to understand that even if you want to deploy and manage Kafka your self.
That is not so easy as it sounds. You cannot use Strimzi to declaratively manage things while also asking it to not manage things declaratively. You have a Kafka custom resource which says how the Kafka cluster should look like. So it creates the Kafka cluster to look like it. Also, from my point of view, the tooling should not modify the StatefulSets and Deployments to add the security context to them. It should instead use admission controllers as other similar tools do. But I guess you might not be able to affect that. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your continuing support in this discussion thread. Autopilot is a new mode or type of GKE (Googles K8s platform), that they released in Feb. 2021. The blog/announcement is here: https://cloud.google.com/blog/products/containers-kubernetes/introducing-gke-autopilot. From the overview documentation: https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview -
Here's a comparison between Autopilot and Standard GKE: https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#comparison - Looking at that table I guess its something that Autopilot pre-configures and manages, such as the things listed under Security. Perhaps I have to deploy a standard GKE, but I was really hoping to spare time and effort by having Google and their automation managed the GKE infrastructure, such as nodes - and in turn have Strimzi manage the application platform: Kafka. Leaving me to do other things. Routes I now see, and will investigate:
At the very least I hope this thread proves useful to others who try Strimzi with GKE Autopilot. - The dream of deploying a fresh GKE, Strimzi and Kafka cluster in something like a dozen simple commands. Happy new year! |
Beta Was this translation helpful? Give feedback.
-
|
Hi. So I now have this: This appears to have stopped the delete/restart loop for Zookeeper and Kafka; however the entity operator still wont come up. I tried to grab some log history via the GCP web ui, attached here: |
Beta Was this translation helpful? Give feedback.
-
|
So, what exactly does it mean with does not come up? Is the pod crashlooping? Or does it not show up at all? Also, can you provide the logs in some more readable format? It looks like what you provided is kind of mixing everything together. |
Beta Was this translation helpful? Give feedback.
-
|
Hi. Is this better? I am finding it difficult to get logs that look useful for something that is so short lived; any advice welcome. and and |
Beta Was this translation helpful? Give feedback.
-
|
The logs are good. But they do not show anything suspicious. It just seems something terminates / deletes the pod after what seems like a very short time. Do Kubernetes logs / events say what or why did it? |
Beta Was this translation helpful? Give feedback.
-
|
It appears to be the Strimzi operator: I am trying to find out what is annoying it - if I am able.. |
Beta Was this translation helpful? Give feedback.
-
|
Nothing in this log suggests it is the cluster operator what deletes it. These are just received events which are ignored. But they confirm that someone / something deletes the pod. |
Beta Was this translation helpful? Give feedback.
-
|
Could it be this annotation? |
Beta Was this translation helpful? Give feedback.
-
|
Well, i suspect the probes do not matter. Based on the logs, the pod is deleted before it has the chance to get ready. Maybe you also want to use your command with the Deployment and not directly with the Replica Set as at the end, if you do it with the ReplicaSet, the Deployment will be yet another party to interfere with it. |
Beta Was this translation helpful? Give feedback.
-
It's not Autopilot that's scaling down the Deployment. The division of responsibilities for Autopilot is that the Pods and Deployments are yours (the operators') to manage, while the Nodes are managed by Autopilot. Autopilot will scale Nodes to zero if you don't have any Pods running, but that's not what's happening here. It's possible that from time to time individual Pods may be terminated, for example due to node upgrade events (which is why we'd always recommend using a Deployment or other higher level workload construct), but that's not the issue here. |
Beta Was this translation helpful? Give feedback.
-
I have seen this kind of problem before (with Knative). One difference with Autopilot, is that it is not unusual for it to take 80-90s for a Pod to move from the Pending to ContainerCreating status. What happens in that time is that Autopilot is provisioning the node to serve the Pod. If an Operator expects Pods to move out of the Pending status in a short amount of time, this could be a problem. In that case, the Operator is a bit overzealous. |
Beta Was this translation helpful? Give feedback.
-
Unlikely. This annotation is simply Autopilot recording a mutation that happened. It indicates that the pod resources were modified to fit within Autopilot's constraints. I wouldn't expect this to cause any problems. Autopilot does two things here:
It's possible if a Pod requires more resources than it requests (i.e. relies on bursting) that it may not work well on Autopilot. In which case, the resources need to be increased. This would actually be a user bug on any Kubernetes cluster, but if may not be obvious if there is always spare capacity on the node. |
Beta Was this translation helpful? Give feedback.
-
Strimzi will by default wait for 5 minutes before calling it an error. But even if it calls it an error, it will not delete the pod, it will keep waiting in next reconciliation. Since we do not manage the pod directly. We manage only the deployment. So I do not think this is the operator being overzealous here. |
Beta Was this translation helpful? Give feedback.
-
|
The health checks on the two operators have: However in GKE these are being interpreted as:
I doubt you wanted port 0, but rather localhost:80 no? There does not seem to be a config option to change these to localhost:80/ready (or localhost:80/healthy) |
Beta Was this translation helpful? Give feedback.
-
|
That looks like a UI issue in whatever UI you are using. They refer to the port named port: healthcheckwhich if you check the full deployment YAML refers to the ports 8080 respectively 8081 . |
Beta Was this translation helpful? Give feedback.
-
|
So I forwarded this discussion thread to Google Cloud support to get their assistance with the EntityOperator failing to deploy. This is their response:
Autopilot Container Isolation[https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#container_isolation]
I think we already see this with the template changes I had to make for the zk and kf sets to stop the delete loop. Is the tls-sidecar container running foul of any other this?
I tried to search for 'stunnel NET_PCAP_RAW' and could not find anything conclusive. Other than some patches to allow certain modes of operations without requiring root. Side question: Why is the tls-sidecar needed when (AFAIK) both Zookeeper and Kafka now support TLS natively? (I know that wasn't true some years ago). Linux Capability Limitations[https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#linux_workload_limitations]
Does tls-sidecar require other capabilities? Ignoring GKE Autopilot; It seems there are many security issues and CVEs (e.g. CVE-2020-12401) associated with CAP_NEW_RAW in the container workload space. If Strimzi (tls-sidecar) does in fact require such, that might be something worth reviewing in the interest of security solely. I am aware that in a complex system such as the components that make up strimzi and the workloads it manages, that these two questions may not be immediately answerable. However I wanted to make this post for posterity. I do plan to try and find time (sometime) to try Strimzi on GKE Standard (maybe even toggle CAP_NET_RAW); not as a long term solution but rather just as a validation test. |
Beta Was this translation helpful? Give feedback.
-
|
Strimzi works fine when all capabilities are dropped. So AFAIK it does not care about the I think none of this explains the Pod DELETE events / scaling down of the Deployment. If for any of the reasons mentioned there would the entity operator not work, it would not be deleting the pod but restarting the containers. |
Beta Was this translation helpful? Give feedback.
-
|
I wasn't convinced either, but it greats to have your thoughts on this too. |
Beta Was this translation helpful? Give feedback.
-
|
So I deployed to a new GKE Standard cluster using all the same as above. All worked first time no issue. |
Beta Was this translation helpful? Give feedback.
-
|
Autopilot isn't the one scaling down the Deployment. Autopilot's job here is to add and remove nodes in response to Pods that are added/removed by the user. It can mutate deployments to modify resource requests, or enforce requirements (like no privileged pods), but it doesn't change the replica count. Since it works on GKE Standard, my guess would be that there is some failure (possibly due to an incompatibility or odd interaction with Autopilot) that the operator is reacting to by scaling the deployment. I'd focus the investigating on root-causing why the scale-down is initiated in the first place. |
Beta Was this translation helpful? Give feedback.
-
|
Well, I do not have access to the cluster. But scaling replicas might mean scaling deployment. It might also mean modifying the Deployment and scaling down one replicaset to scale up another replicaset. I guess @ThorbenJ would need to check that. |
Beta Was this translation helpful? Give feedback.
-
|
FYI: https://issuetracker.google.com/issues/214356345 - Created by google in response to this case. |
Beta Was this translation helpful? Give feedback.
-
|
This seems to have been solved. Documented my experience in #6922 |
Beta Was this translation helpful? Give feedback.
-
|
This is specifically about GKE Autopilot, if am not mistaken your last comment in #6922 is that you (like me) reverted to using GKE Standard. So is this really solved in GKE Autopilot? |
Beta Was this translation helpful? Give feedback.
-
|
Nope, it hasn't been solved. Or better said, the crash behaves differently currently than it did when I initially tested it ~1 year ago. I think back then the Kafka brokers themselves were crashing and now its the operator |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi.
I followed the getting started guide to deploy Strimzi on a GKE Autopilot cluster (1.21.5-gke.1302); for which I also followed the getting started guide.
I first deployed 0.26.1, then tried to upgrade to 0.27.0, and then deleted everything and started again deploying 0.27.0 - and now I am here asking for help. Note: I deployed strimzi into namespace 'strimzi' and Kafka into 'kafka'.
I see two big issues:
1) The abc-entity-operator deployment is failing and is creating an ever longer list of ReplicaSets:
I assume this is not intentional. When I was deleting everything to start again, I had nearly 8000 of these to delete...
In the GCP webui console the reason I most often see for the error state is
Container tls-sidecar is waiting, and I managed to grab this:2) It keeps deleting (and thus restarting) both Zookeeper and Kafka nodes for no apparent reason.
Looking at Zookeeper and Kafka they seem happy, I can even do basic operations like interact with topics:
Here are the operator logs:
strimzi_operator.log
What am I doing wrong? Please advise what can I do?
Beta Was this translation helpful? Give feedback.
All reactions