Possible to create new certificate when expired #8266

ggrames · 2023-03-20T10:54:35Z

ggrames
Mar 20, 2023

Hi,
i am running strimzi operator in the version 0.27.1
The certs are expired since some days.
I faced the problem that the
*-entity-operator not complete running
*-kafka-0 , *-kafka-... in CrashLoopBackOff states
*-zookeeper-0, *-zookeeper-... in CrashLoopBackOff states

Is there a possibility to create new certificates?
Thank you

Answered by scholzj

Mar 20, 2023

If the certs are already expired, you will need to follow throught this process:

Delete the CA secrets and have the operator recreate them
Delete the ZooKeeper secrets and the ZooKeeper pods
Delete the KAfka secrets and the Kafka pods
etc.

=> that way, you should be able to recover the cluster.

View full answer

scholzj · 2023-03-20T13:42:59Z

scholzj
Mar 20, 2023
Maintainer

If the certs are already expired, you will need to follow throught this process:

Delete the CA secrets and have the operator recreate them
Delete the ZooKeeper secrets and the ZooKeeper pods
Delete the KAfka secrets and the Kafka pods
etc.

=> that way, you should be able to recover the cluster.

0 replies

ggrames · 2023-03-21T05:21:14Z

ggrames
Mar 21, 2023
Author

Thanks,
this sounds easy, but i deleted now in the first step the
mycluster-clients-ca
mycluster-cluster-ca

but it is never recreated. I waited around 5 minutes.
I tried to restart the mycluster-entity-operator-.... Pod
but nothing helps

Logfile of the tls-sidecare container:


Starting Stunnel with configuration:
pid = /tmp/stunnel.pid
foreground = yes
debug = notice
sslVersion = TLSv1.2
[zookeeper-2181]
client = yes
CAfile = /tmp/cluster-ca.crt
cert = /etc/tls-sidecar/eo-certs/entity-operator.crt
key = /etc/tls-sidecar/eo-certs/entity-operator.key
accept = 127.0.0.1:2181
connect = mycluster-zookeeper-client:2181
delay = yes
verify = 2


+ exec /usr/bin/tini -w -e 143 -- /usr/bin/stunnel /tmp/stunnel.conf
2023.03.21 05:15:28 LOG5[ui]: stunnel 5.56 on x86_64-redhat-linux-gnu platform
2023.03.21 05:15:28 LOG5[ui]: Compiled with OpenSSL 1.1.1g FIPS  21 Apr 2020
2023.03.21 05:15:28 LOG5[ui]: Running  with OpenSSL 1.1.1k  FIPS 25 Mar 2021
2023.03.21 05:15:28 LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6 TLS:ENGINE,FIPS,OCSP,PSK,SNI
2023.03.21 05:15:28 LOG5[ui]: Reading configuration from file /tmp/stunnel.conf
2023.03.21 05:15:28 LOG5[ui]: UTF-8 byte order mark not detected
2023.03.21 05:15:28 LOG5[ui]: FIPS mode disabled
2023.03.21 05:15:28 LOG4[ui]: Service [zookeeper-2181] uses "verifyChain" without subject checks
2023.03.21 05:15:28 LOG4[ui]: Use "checkHost" or "checkIP" to restrict trusted certificates
2023.03.21 05:15:28 LOG5[ui]: Configuration successful
2023.03.21 05:15:32 LOG5[0]: Service [zookeeper-2181] accepted connection from 127.0.0.1:40236
2023.03.21 05:15:35 LOG3[0]: s_connect: connect 172.30.114.175:2181: No route to host (113)
2023.03.21 05:15:35 LOG3[0]: No more addresses to connect
...

15 replies

scholzj Mar 21, 2023
Maintainer

The Entity operator does not generate any CA or server certificates. If you follow the steps I shared on the beginning and the secrets were not recreated in step one, you need to check the cluster operator log as I suggested.

ggrames Mar 21, 2023
Author

ok, i restartet the strimzi-cluster-operator now.
It looks better.
Zookeepers (1 to 3) are running
Kafka (1 to 3) are running


2023-03-21 10:01:57 INFO  ClusterOperator:79 - Creating ClusterOperator for namespace shared-datastores-test
2023-03-21 10:01:57 INFO  ClusterOperator:94 - Starting ClusterOperator for namespace shared-datastores-test
2023-03-21 10:01:57 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2023-03-21 10:01:59 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2023-03-21 10:02:00 INFO  StrimziPodSetController:419 - Starting the StrimziPodSet controller
2023-03-21 10:02:00 INFO  StrimziPodSetController:387 - Starting StrimziPodSet controller for namespace shared-datastores-test
2023-03-21 10:02:00 INFO  StrimziPodSetController:389 - Waiting for informers to sync
2023-03-21 10:02:00 INFO  StrimziPodSetController:394 - Informers are in-sync
2023-03-21 10:02:00 INFO  ClusterOperator:108 - Opened watch for Kafka operator
2023-03-21 10:02:00 INFO  OperatorWatcher:38 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Kafka mycluster in namespace shared-datastores-test was ADDED
2023-03-21 10:02:00 INFO  ClusterOperator:108 - Opened watch for KafkaMirrorMaker operator
2023-03-21 10:02:00 INFO  AbstractOperator:226 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Kafka mycluster will be checked for creation or modification
2023-03-21 10:02:00 INFO  ClusterOperator:108 - Opened watch for KafkaConnect operator
2023-03-21 10:02:00 INFO  ClusterOperator:108 - Opened watch for KafkaBridge operator
2023-03-21 10:02:00 INFO  ClusterOperator:108 - Opened watch for KafkaMirrorMaker2 operator
2023-03-21 10:02:00 INFO  ClusterOperator:119 - Setting up periodic reconciliation for namespace shared-datastores-test
2023-03-21 10:02:00 INFO  ClusterOperator:178 - ClusterOperator is now ready (health server listening on 8080)
2023-03-21 10:02:00 INFO  Main:151 - Cluster Operator verticle started in namespace shared-datastores-test without label selector
2023-03-21 10:02:05 WARN  VersionUsageUtils:60 - The client is using resource type 'poddisruptionbudgets' with unstable version 'v1beta1'
2023-03-21 10:02:06 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-2.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:06 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-2 is not a leader
2023-03-21 10:02:06 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-1.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:06 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-1 is not a leader
2023-03-21 10:02:06 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-0.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:06 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-0 is not a leader
2023-03-21 10:02:06 INFO  ZookeeperLeaderFinder:177 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): No leader found for cluster mycluster in namespace shared-datastores-test; backing off for 0ms (cumulative 0ms)
2023-03-21 10:02:06 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-2.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:06 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-2 is not a leader
2023-03-21 10:02:06 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-1.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:06 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-1 is not a leader
2023-03-21 10:02:06 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-0.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:06 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-0 is not a leader
2023-03-21 10:02:06 INFO  ZookeeperLeaderFinder:177 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): No leader found for cluster mycluster in namespace shared-datastores-test; backing off for 5000ms (cumulative 5000ms)
2023-03-21 10:02:11 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-2.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:11 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-2 is not a leader
2023-03-21 10:02:11 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-1.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:11 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-1 is not a leader
2023-03-21 10:02:11 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-0.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:11 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-0 is not a leader
2023-03-21 10:02:11 INFO  ZookeeperLeaderFinder:177 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): No leader found for cluster mycluster in namespace shared-datastores-test; backing off for 10000ms (cumulative 15000ms)
2023-03-21 10:02:21 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-2.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:21 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-2 is not a leader
2023-03-21 10:02:21 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-1.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:21 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-1 is not a leader
2023-03-21 10:02:21 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-0.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:21 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-0 is not a leader
2023-03-21 10:02:21 INFO  ZookeeperLeaderFinder:177 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): No leader found for cluster mycluster in namespace shared-datastores-test; backing off for 20000ms (cumulative 35000ms)
2023-03-21 10:02:41 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-2.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:41 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-2 is not a leader
2023-03-21 10:02:41 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-1.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:41 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-1 is not a leader
2023-03-21 10:02:41 WARN  ZookeeperLeaderFinder:237 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): ZK mycluster-zookeeper-0.mycluster-zookeeper-nodes.shared-datastores-test.svc.cluster.local:2181: failed to connect to zookeeper:
2023-03-21 10:02:41 INFO  ZookeeperLeaderFinder:209 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Pod mycluster-zookeeper-0 is not a leader
2023-03-21 10:02:41 WARN  ZookeeperLeaderFinder:171 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Giving up trying to find the leader of mycluster/shared-datastores-test after 4 attempts taking 35000ms
2023-03-21 10:02:41 INFO  ZooKeeperRoller:133 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Rolling pod mycluster-zookeeper-2 due to [server certificates changed]
2023-03-21 10:02:41 INFO  PodOperator:54 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Rolling pod mycluster-zookeeper-2
2023-03-21 10:03:00 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Reconciliation is in progress
2023-03-21 10:03:14 INFO  ZooKeeperRoller:133 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Rolling pod mycluster-zookeeper-1 due to [server certificates changed]
2023-03-21 10:03:14 INFO  PodOperator:54 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Rolling pod mycluster-zookeeper-1
2023-03-21 10:03:46 INFO  ZooKeeperRoller:133 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Rolling pod mycluster-zookeeper-0 due to [server certificates changed]
2023-03-21 10:03:46 INFO  PodOperator:54 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Rolling pod mycluster-zookeeper-0
2023-03-21 10:04:00 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(shared-datastores-test/mycluster): Reconciliation is in progress
2023-03-21 10:04:00 INFO  ClusterOperator:121 - Triggering periodic reconciliation for namespace shared-datastores-test
2023-03-21 10:04:00 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormakers' with unstable version 'v1beta2'
2023-03-21 10:04:00 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2023-03-21 10:04:00 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2023-03-21 10:04:00 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkabridges' with unstable version 'v1beta2'
2023-03-21 10:04:00 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkarebalances' with unstable version 'v1beta2'

ggrames Mar 21, 2023
Author

all is running. Now. Also the entity-operator
The secrets have now the new certs.

Somehow i was blind. So it is running now completly. Thank you very much for you maintenance.

ggrames Mar 21, 2023
Author

BR from vienna

scholzj Mar 21, 2023
Maintainer

Great, glad it works now 👍

ggrames · 2023-03-21T07:22:03Z

ggrames
Mar 21, 2023
Author

I have following config in the Kafka Instance:


apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  creationTimestamp: '2022-01-07T08:54:28Z'
  generation: 9
  name: mycluster
  namespace: shared-datastores-test
  resourceVersion: '1397725206'
  uid: b91e0e0f-a3f3-4a3e-8388-4fb8cdf349e3
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      log.retention.ms: -1
      compression.type: gzip
      log.retention.minutes: -1
      inter.broker.protocol.version: '2.8'
      transaction.state.log.replication.factor: 3
      min.insync.replicas: 2
      replication.factor: 3
      num.partitions: 1
      log.message.format.version: '2.8'
      transaction.state.log.min.isr: 2
      log.retention.hours: -1
      errors.deadletterqueue.topic.replication.factor: 3
      offsets.topic.replication.factor: 3
      default.replication.factor: 3
    listeners:
      - name: plain
        port: 9092
        tls: false
        type: internal
      - name: tls
        port: 9093
        tls: true
        type: internal
      - name: external
        port: 9094
        tls: false
        type: nodeport
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: metrics-config.yml
          name: kafka-broker-metrics
    replicas: 3
    storage:
      class: thin
      deleteClaim: false
      size: 30Gi
      type: persistent-claim
    template:
      pod:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchExpressions:
                      - key: strimzi.io/cluster
                        operator: In
                        values:
                          - mycluster
                  topologyKey: kubernetes.io/hostname
                weight: 100
        tmpDirSizeLimit: 100Mi
    version: 2.8.0
  kafkaExporter:
    enableSaramaLogging: true
    groupRegex: .*
    logging: debug
    topicRegex: .*
  zookeeper:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: metrics-config.yml
          name: zookeeper-metrics
    replicas: 3
    storage:
      class: thin
      deleteClaim: false
      size: 1Gi
      type: persistent-claim

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Possible to create new certificate when expired #8266

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 15 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

Possible to create new certificate when expired #8266

Uh oh!

ggrames Mar 20, 2023

Replies: 3 comments · 15 replies

Uh oh!

scholzj Mar 20, 2023 Maintainer

Uh oh!

Uh oh!

ggrames Mar 21, 2023 Author

Uh oh!

scholzj Mar 21, 2023 Maintainer

Uh oh!

ggrames Mar 21, 2023 Author

Uh oh!

ggrames Mar 21, 2023 Author

Uh oh!

ggrames Mar 21, 2023 Author

Uh oh!

scholzj Mar 21, 2023 Maintainer

Uh oh!

ggrames Mar 21, 2023 Author

ggrames
Mar 20, 2023

Replies: 3 comments 15 replies

scholzj
Mar 20, 2023
Maintainer

ggrames
Mar 21, 2023
Author

scholzj Mar 21, 2023
Maintainer

ggrames Mar 21, 2023
Author

ggrames Mar 21, 2023
Author

ggrames Mar 21, 2023
Author

scholzj Mar 21, 2023
Maintainer

ggrames
Mar 21, 2023
Author