User/topic operator not starting with network policy #8739

nmichaud2 · 2023-06-27T19:18:49Z

nmichaud2
Jun 27, 2023

We have a network policy that blocks pretty much everything by default:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-dev
  namespace: dev
spec:
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

I was able to target the different pods in my dev namespace with another network policy so that zookeeper/kafka/entity operator can talk to each other but I'm still getting an error in the topic operator which I'm not able to track(Very similar network policies are being used in our staging env and it works) The log of the topic operator if anyone has any idea:


Preparing trust store certificates for internal communication
Adding /etc/tls-sidecar/cluster-ca-certs/ca.crt to truststore /tmp/topic-operator/replication.truststore.p12 with alias ca
Certificate was added to keystore
Preparing trust store certificates for internal communication is completed
Preparing key store certificates for internal communication
Preparing key store certificates for internal communication is completed
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/topic-operator/custom-config/log4j2.properties -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20 -classpath :lib/io.strimzi.topic-operator-0.35.1.jar:lib/io.strimzi.api-0.35.1.jar:lib/io.fabric8.kubernetes-model-networking-6.7.0.jar:lib/io.fabric8.kubernetes-model-common-6.7.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.7.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.14.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.14.2.jar:lib/org.yaml.snakeyaml-2.0.jar:lib/io.fabric8.kubernetes-client-6.7.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.7.0.jar:lib/io.fabric8.kubernetes-client-api-6.7.0.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.7.0.jar:lib/io.fabric8.kubernetes-model-resource-6.7.0.jar:lib/io.fabric8.kubernetes-model-rbac-6.7.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.7.0.jar:lib/io.fabric8.kubernetes-model-apps-6.7.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.7.0.jar:lib/io.fabric8.kubernetes-model-batch-6.7.0.jar:lib/io.fabric8.kubernetes-model-certificates-6.7.0.jar:lib/io.fabric8.kubernetes-model-coordination-6.7.0.jar:lib/io.fabric8.kubernetes-model-discovery-6.7.0.jar:lib/io.fabric8.kubernetes-model-events-6.7.0.jar:lib/io.fabric8.kubernetes-model-extensions-6.7.0.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.7.0.jar:lib/io.fabric8.kubernetes-model-metrics-6.7.0.jar:lib/io.fabric8.kubernetes-model-policy-6.7.0.jar:lib/io.fabric8.kubernetes-model-scheduling-6.7.0.jar:lib/io.fabric8.kubernetes-model-storageclass-6.7.0.jar:lib/io.fabric8.kubernetes-model-node-6.7.0.jar:lib/org.snakeyaml.snakeyaml-engine-2.6.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.15.2.jar:lib/io.fabric8.kubernetes-model-core-6.7.0.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/org.apache.logging.log4j.log4j-jul-2.17.2.jar:lib/org.apache.kafka.kafka-clients-3.4.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-handler-4.1.87.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.87.Final.jar:lib/io.netty.netty-codec-4.1.87.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.87.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.87.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/com.fasterxml.jackson.core.jackson-core-2.14.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.14.2.jar:lib/io.vertx.vertx-core-4.3.8.jar:lib/io.netty.netty-common-4.1.87.Final.jar:lib/io.netty.netty-buffer-4.1.87.Final.jar:lib/io.netty.netty-transport-4.1.87.Final.jar:lib/io.netty.netty-handler-proxy-4.1.87.Final.jar:lib/io.netty.netty-codec-socks-4.1.87.Final.jar:lib/io.netty.netty-codec-http-4.1.87.Final.jar:lib/io.netty.netty-codec-http2-4.1.87.Final.jar:lib/io.netty.netty-resolver-4.1.87.Final.jar:lib/io.netty.netty-resolver-dns-4.1.87.Final.jar:lib/io.netty.netty-codec-dns-4.1.87.Final.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.2.Final.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.2.Final.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/org.apache.kafka.kafka-streams-3.4.0.jar:lib/org.rocksdb.rocksdbjni-7.1.2.jar:lib/io.fabric8.openshift-client-api-6.7.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.7.0.jar:lib/io.fabric8.openshift-model-operator-6.7.0.jar:lib/io.fabric8.openshift-model-machine-6.7.0.jar:lib/io.fabric8.openshift-model-whereabouts-6.7.0.jar:lib/io.fabric8.openshift-model-monitoring-6.7.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.7.0.jar:lib/io.fabric8.openshift-model-tuned-6.7.0.jar:lib/io.fabric8.openshift-model-console-6.7.0.jar:lib/io.fabric8.openshift-model-config-6.7.0.jar:lib/io.fabric8.openshift-model-machineconfig-6.7.0.jar:lib/io.fabric8.openshift-model-miscellaneous-6.7.0.jar:lib/io.fabric8.openshift-model-hive-6.7.0.jar:lib/io.fabric8.openshift-model-installer-6.7.0.jar:lib/io.fabric8.openshift-model-operatorhub-6.7.0.jar:lib/io.strimzi.operator-common-0.35.1.jar:lib/io.strimzi.crd-annotations-0.35.1.jar:lib/io.strimzi.certificate-manager-0.35.1.jar:lib/io.fabric8.openshift-model-6.7.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.8.jar io.strimzi.operator.topic.Main
2023-06-27 18:40:37,97209 INFO  [main] Main:37 - TopicOperator 0.35.1 is starting
2023-06-27 18:40:38,40653 INFO  [main] Session:121 - Using config:
	STRIMZI_TRUSTSTORE_LOCATION: /tmp/topic-operator/replication.truststore.p12
	STRIMZI_RESOURCE_LABELS: strimzi.io/cluster=kafka-etl
	STRIMZI_SASL_MECHANISM: 
	STRIMZI_SASL_USERNAME: 
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	STRIMZI_CLIENT_ID: strimzi-topic-operator-ab8e0bf0-c92f-4273-8a0a-abad1d167333
	STRIMZI_SECURITY_PROTOCOL: SSL
	STRIMZI_STALE_RESULT_TIMEOUT_MS: 5000
	STRIMZI_TOPIC_METADATA_MAX_ATTEMPTS: 6
	STRIMZI_KEYSTORE_LOCATION: /tmp/topic-operator/replication.keystore.p12
	STRIMZI_REASSIGN_THROTTLE: 9223372036854775807
	STRIMZI_USE_ZOOKEEPER_TOPIC_STORE: false
	STRIMZI_SASL_PASSWORD: ********
	STRIMZI_KAFKA_BOOTSTRAP_SERVERS: kafka-etl-kafka-bootstrap:9091
	STRIMZI_SASL_ENABLED: false
	STRIMZI_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS
	STRIMZI_NAMESPACE: dev
	STRIMZI_APPLICATION_ID: __strimzi-topic-operator-kstreams
	STRIMZI_ZOOKEEPER_SESSION_TIMEOUT_MS: 18000
	STRIMZI_TOPICS_PATH: /strimzi/topics
	STRIMZI_ZOOKEEPER_CONNECT: localhost:2181
	STRIMZI_TLS_ENABLED: true
	STRIMZI_KEYSTORE_PASSWORD: ********
	STRIMZI_STORE_NAME: topic-store
	STRIMZI_REASSIGN_VERIFY_INTERVAL_MS: 120000
	TC_ZK_CONNECTION_TIMEOUT_MS: 18000
	STRIMZI_TRUSTSTORE_PASSWORD: ********
	STRIMZI_STORE_TOPIC: __strimzi_store_topic

2023-06-27 18:40:38,41200 INFO  [vert.x-eventloop-thread-0] Session:209 - Starting
2023-06-27 18:40:38,48977 INFO  [vert.x-eventloop-thread-0] AppInfoParser:119 - Kafka version: 3.4.0
2023-06-27 18:40:38,48991 INFO  [vert.x-eventloop-thread-0] AppInfoParser:120 - Kafka commitId: 2e1947d240607d53
2023-06-27 18:40:38,48998 INFO  [vert.x-eventloop-thread-0] AppInfoParser:121 - Kafka startTimeMs: 1687891238488
2023-06-27 18:40:38,56214 INFO  [ZkClient-EventThread-34-localhost:2181] ZkEventThread:65 - Starting ZkClient event thread.
2023-06-27 18:40:38,56470 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT
2023-06-27 18:40:38,56493 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:host.name=kafka-etl-entity-operator-565bc97698-xf865
2023-06-27 18:40:38,56500 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.version=17.0.7
2023-06-27 18:40:38,56507 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.vendor=Red Hat, Inc.
2023-06-27 18:40:38,56513 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.home=/usr/lib/jvm/java-17-openjdk-17.0.7.0.7-3.el8.x86_64
2023-06-27 18:40:38,56521 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.class.path=:lib/io.strimzi.topic-operator-0.35.1.jar:lib/io.strimzi.api-0.35.1.jar:lib/io.fabric8.kubernetes-model-networking-6.7.0.jar:lib/io.fabric8.kubernetes-model-common-6.7.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.7.0.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.14.2.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.14.2.jar:lib/org.yaml.snakeyaml-2.0.jar:lib/io.fabric8.kubernetes-client-6.7.0.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.7.0.jar:lib/io.fabric8.kubernetes-client-api-6.7.0.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.7.0.jar:lib/io.fabric8.kubernetes-model-resource-6.7.0.jar:lib/io.fabric8.kubernetes-model-rbac-6.7.0.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.7.0.jar:lib/io.fabric8.kubernetes-model-apps-6.7.0.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.7.0.jar:lib/io.fabric8.kubernetes-model-batch-6.7.0.jar:lib/io.fabric8.kubernetes-model-certificates-6.7.0.jar:lib/io.fabric8.kubernetes-model-coordination-6.7.0.jar:lib/io.fabric8.kubernetes-model-discovery-6.7.0.jar:lib/io.fabric8.kubernetes-model-events-6.7.0.jar:lib/io.fabric8.kubernetes-model-extensions-6.7.0.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.7.0.jar:lib/io.fabric8.kubernetes-model-metrics-6.7.0.jar:lib/io.fabric8.kubernetes-model-policy-6.7.0.jar:lib/io.fabric8.kubernetes-model-scheduling-6.7.0.jar:lib/io.fabric8.kubernetes-model-storageclass-6.7.0.jar:lib/io.fabric8.kubernetes-model-node-6.7.0.jar:lib/org.snakeyaml.snakeyaml-engine-2.6.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.15.2.jar:lib/io.fabric8.kubernetes-model-core-6.7.0.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/org.apache.logging.log4j.log4j-jul-2.17.2.jar:lib/org.apache.kafka.kafka-clients-3.4.0.jar:lib/com.github.luben.zstd-jni-1.5.2-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.8.4.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/com.101tec.zkclient-0.11.jar:lib/org.apache.zookeeper.zookeeper-3.6.3.jar:lib/org.apache.yetus.audience-annotations-0.5.0.jar:lib/io.netty.netty-handler-4.1.87.Final.jar:lib/io.netty.netty-transport-native-unix-common-4.1.87.Final.jar:lib/io.netty.netty-codec-4.1.87.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.87.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.87.Final.jar:lib/org.apache.zookeeper.zookeeper-jute-3.6.3.jar:lib/com.fasterxml.jackson.core.jackson-core-2.14.2.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.14.2.jar:lib/io.vertx.vertx-core-4.3.8.jar:lib/io.netty.netty-common-4.1.87.Final.jar:lib/io.netty.netty-buffer-4.1.87.Final.jar:lib/io.netty.netty-transport-4.1.87.Final.jar:lib/io.netty.netty-handler-proxy-4.1.87.Final.jar:lib/io.netty.netty-codec-socks-4.1.87.Final.jar:lib/io.netty.netty-codec-http-4.1.87.Final.jar:lib/io.netty.netty-codec-http2-4.1.87.Final.jar:lib/io.netty.netty-resolver-4.1.87.Final.jar:lib/io.netty.netty-resolver-dns-4.1.87.Final.jar:lib/io.netty.netty-codec-dns-4.1.87.Final.jar:lib/io.apicurio.apicurio-registry-utils-streams-1.3.2.Final.jar:lib/jakarta.annotation.jakarta.annotation-api-1.3.5.jar:lib/io.apicurio.apicurio-registry-utils-kafka-1.3.2.Final.jar:lib/org.eclipse.microprofile.config.microprofile-config-api-1.4.jar:lib/org.apache.kafka.kafka-streams-3.4.0.jar:lib/org.rocksdb.rocksdbjni-7.1.2.jar:lib/io.fabric8.openshift-client-api-6.7.0.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.7.0.jar:lib/io.fabric8.openshift-model-operator-6.7.0.jar:lib/io.fabric8.openshift-model-machine-6.7.0.jar:lib/io.fabric8.openshift-model-whereabouts-6.7.0.jar:lib/io.fabric8.openshift-model-monitoring-6.7.0.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.7.0.jar:lib/io.fabric8.openshift-model-tuned-6.7.0.jar:lib/io.fabric8.openshift-model-console-6.7.0.jar:lib/io.fabric8.openshift-model-config-6.7.0.jar:lib/io.fabric8.openshift-model-machineconfig-6.7.0.jar:lib/io.fabric8.openshift-model-miscellaneous-6.7.0.jar:lib/io.fabric8.openshift-model-hive-6.7.0.jar:lib/io.fabric8.openshift-model-installer-6.7.0.jar:lib/io.fabric8.openshift-model-operatorhub-6.7.0.jar:lib/io.strimzi.operator-common-0.35.1.jar:lib/io.strimzi.crd-annotations-0.35.1.jar:lib/io.strimzi.certificate-manager-0.35.1.jar:lib/io.fabric8.openshift-model-6.7.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.3.8.jar
2023-06-27 18:40:38,56538 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2023-06-27 18:40:38,56545 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.io.tmpdir=/tmp
2023-06-27 18:40:38,56551 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:java.compiler=<NA>
2023-06-27 18:40:38,56557 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.name=Linux
2023-06-27 18:40:38,56563 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.arch=amd64
2023-06-27 18:40:38,56570 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.version=5.15.0-75-generic
2023-06-27 18:40:38,56577 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:user.name=strimzi
2023-06-27 18:40:38,56585 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:user.home=/home/strimzi
2023-06-27 18:40:38,56606 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:user.dir=/opt/strimzi
2023-06-27 18:40:38,56632 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.memory.free=111MB
2023-06-27 18:40:38,56640 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.memory.max=12704MB
2023-06-27 18:40:38,56647 INFO  [vert.x-worker-thread-0] ZooKeeper:98 - Client environment:os.memory.total=160MB
2023-06-27 18:40:38,56754 INFO  [vert.x-worker-thread-0] ZooKeeper:1006 - Initiating client connection, connectString=localhost:2181 sessionTimeout=18000 watcher=org.I0Itec.zkclient.ZkClient@e15082c
2023-06-27 18:40:38,56931 INFO  [vert.x-worker-thread-0] X509Util:77 - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2023-06-27 18:40:38,57100 INFO  [vert.x-worker-thread-0] ClientCnxnSocket:239 - jute.maxbuffer value is 1048575 Bytes
2023-06-27 18:40:38,57401 INFO  [vert.x-worker-thread-0] ClientCnxn:1736 - zookeeper.request.timeout value is 0. feature enabled=false
2023-06-27 18:40:38,57434 INFO  [vert.x-worker-thread-0] ZkClient:936 - Waiting for keeper state SyncConnected
2023-06-27 18:40:38,57665 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1181 - Opening socket connection to server localhost/127.0.0.1:2181.
2023-06-27 18:40:38,57691 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1183 - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-06-27 18:40:38,57725 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1013 - Socket connection established, initiating session, client: /127.0.0.1:43700, server: localhost/127.0.0.1:2181
2023-06-27 18:40:38,60432 INFO  [vert.x-worker-thread-0-SendThread(localhost:2181)] ClientCnxn:1448 - Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x10005cdebec0002, negotiated timeout = 18000
2023-06-27 18:40:38,60636 INFO  [vert.x-worker-thread-0-EventThread] ZkClient:713 - zookeeper state changed (SyncConnected)
2023-06-27 18:40:38,64320 INFO  [blocking-startup-ops-0] AppInfoParser:119 - Kafka version: 3.4.0
2023-06-27 18:40:38,64348 INFO  [blocking-startup-ops-0] AppInfoParser:120 - Kafka commitId: 2e1947d240607d53
2023-06-27 18:40:38,64357 INFO  [blocking-startup-ops-0] AppInfoParser:121 - Kafka startTimeMs: 1687891238643
2023-06-27 18:40:38,64395 INFO  [blocking-startup-ops-0] KafkaStreamsTopicStoreService:52 - Starting ...
2023-06-27 18:40:38,80971 INFO  [kafka-admin-client-thread | adminclient-2] KafkaStreamsTopicStoreService:135 - Creating new store topic: __strimzi_store_topic
2023-06-27 18:40:38,94005 INFO  [kafka-admin-client-thread | adminclient-2] KafkaStreamsTopicStoreService:97 - Creating Kafka Streams, store name: topic-store
2023-06-27 18:40:38,99657 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:119 - Kafka version: 3.4.0
2023-06-27 18:40:38,99669 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:120 - Kafka commitId: 2e1947d240607d53
2023-06-27 18:40:38,99674 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:121 - Kafka startTimeMs: 1687891238996
2023-06-27 18:40:39,02636 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:119 - Kafka version: 3.4.0
2023-06-27 18:40:39,02651 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:120 - Kafka commitId: 2e1947d240607d53
2023-06-27 18:40:39,02659 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:121 - Kafka startTimeMs: 1687891239026
2023-06-27 18:40:39,05093 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:119 - Kafka version: 3.4.0
2023-06-27 18:40:39,05105 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:120 - Kafka commitId: 2e1947d240607d53
2023-06-27 18:40:39,05111 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:121 - Kafka startTimeMs: 1687891239050
2023-06-27 18:40:39,07905 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:119 - Kafka version: 3.4.0
2023-06-27 18:40:39,07921 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:120 - Kafka commitId: 2e1947d240607d53
2023-06-27 18:40:39,08012 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:121 - Kafka startTimeMs: 1687891239078
2023-06-27 18:40:42,74334 INFO  [__strimzi-topic-operator-kstreams-399fe20a-a867-4d50-a23c-5503d3ed5d2a-StreamThread-1] LoggingStateRestoreListener:16 - restore start: topicPartition=__strimzi-topic-operator-kstreams-topic-store-changelog-0, storeName=topic-store, startingOffset=0, endingOffset=0
2023-06-27 18:40:42,84531 INFO  [__strimzi-topic-operator-kstreams-399fe20a-a867-4d50-a23c-5503d3ed5d2a-StreamThread-1] LoggingStateRestoreListener:27 - restore end: topicPartition=__strimzi-topic-operator-kstreams-topic-store-changelog-0, storeName=topic-store, totalRestored=0
2023-06-27 18:40:42,85002 INFO  [ForkJoinPool.commonPool-worker-1] KafkaStreamsTopicStoreService:81 - Creating topic store ...
2023-06-27 18:40:42,85455 INFO  [ForkJoinPool.commonPool-worker-1] KafkaStreamsTopicStoreService:72 - Started.
2023-06-27 18:40:42,85493 INFO  [kafka-admin-client-thread | adminclient-2] AppInfoParser:83 - App info kafka.admin.client for adminclient-2 unregistered
2023-06-27 18:40:42,85764 INFO  [kafka-admin-client-thread | adminclient-2] Metrics:693 - Metrics scheduler closed
2023-06-27 18:40:42,85776 INFO  [kafka-admin-client-thread | adminclient-2] Metrics:697 - Closing reporter org.apache.kafka.common.metrics.JmxReporter
2023-06-27 18:40:42,85784 INFO  [kafka-admin-client-thread | adminclient-2] Metrics:703 - Metrics reporters closed
2023-06-27 18:40:42,85816 INFO  [blocking-startup-ops-0] Session:264 - Topic store created, took 4250 ms
2023-06-27 18:41:43,32279 WARN  [vertx-blocked-thread-checker] BlockedThreadChecker: - Thread Thread[blocking-startup-ops-0,5,main] has been blocked for 60452 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
	at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:211) ~[?:?]
	at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864) ~[?:?]
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463) ~[?:?]
	at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434) ~[?:?]
	at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072) ~[?:?]
	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:162) ~[io.fabric8.kubernetes-client-api-6.7.0.jar:?]
	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:185) ~[io.fabric8.kubernetes-client-api-6.7.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:611) ~[io.fabric8.kubernetes-client-6.7.0.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.watch(BaseOperation.java:598) ~[io.fabric8.kubernetes-client-6.7.0.jar:?]
	at io.strimzi.operator.topic.Session.startWatcher(Session.java:426) ~[io.strimzi.topic-operator-0.35.1.jar:0.35.1]
	at io.strimzi.operator.topic.Session.lambda$createK8sWatcher$14(Session.java:255) ~[io.strimzi.topic-operator-0.35.1.jar:0.35.1]
	at io.strimzi.operator.topic.Session$$Lambda$847/0x0000000801502f00.handle(Unknown Source) ~[?:?]
	at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.ContextBase$$Lambda$424/0x00000008013ac240.handle(Unknown Source) ~[?:?]
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.ContextBase$$Lambda$421/0x00000008013a6f40.run(Unknown Source) ~[?:?]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.3.8.jar:4.3.8]
	at io.vertx.core.impl.TaskQueue$$Lambda$331/0x00000008012d8000.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.87.Final.jar:4.1.87.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-06-27 18:41:44,32293 WARN  [vertx-blocked-thread-checker] BlockedThreadChecker: - Thread Thread[blocking-startup-ops-0,5,main] has been blocked for 61453 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked

nmichaud2 · 2023-06-27T19:26:00Z

nmichaud2
Jun 27, 2023
Author

Looking at the network policy that strimzi generates makes me think I might be missing something for the cluster-operator which is in another namespace:

spec:
  podSelector:
    matchLabels:
      strimzi.io/cluster: kafka-etl
      strimzi.io/kind: Kafka
      strimzi.io/name: kafka-etl-kafka
  ingress:
    - ports:
        - protocol: TCP
          port: 9090
      from:
        - podSelector:
            matchLabels:
              strimzi.io/cluster: kafka-etl
              strimzi.io/kind: Kafka
              strimzi.io/name: kafka-etl-kafka
    - ports:
        - protocol: TCP
          port: 9091
      from:
        - podSelector:
            matchLabels:
              strimzi.io/kind: cluster-operator
          namespaceSelector: {}
        - podSelector:
            matchLabels:
              strimzi.io/cluster: kafka-etl
              strimzi.io/kind: Kafka
              strimzi.io/name: kafka-etl-kafka
        - podSelector:
            matchLabels:
              strimzi.io/name: kafka-etl-entity-operator
        - podSelector:
            matchLabels:
              strimzi.io/name: kafka-etl-kafka-exporter
        - podSelector:
            matchLabels:
              strimzi.io/name: kafka-etl-cruise-control
    - ports:
        - protocol: TCP
          port: 9092
    - ports:
        - protocol: TCP
          port: 9093
  policyTypes:
    - Ingress

0 replies

scholzj · 2023-06-27T19:26:32Z

scholzj
Jun 27, 2023
Maintainer

Strimzi normally creates its own network policies to ensure the components can talk with each other. So I'm not sure why do you need to create some additional policies for that. The operators obviously also need to talk to Kubernetes APIs.

3 replies

nmichaud2 Jun 27, 2023
Author

It only sets a policy type for Ingress

scholzj Jun 27, 2023
Maintainer

Ahh, ok. Well, the Entity Operator talks to ZooKeeper nodes (port 2181), Kafka brokers(port 9091 IIRC) and Kubernetes API.

nmichaud2 Jun 27, 2023
Author

it's the kubernetes API. We're used to have the operators in separate namespaces, but since the entity operator gets deployed in the dev namespace which we deny all for, it gets blocked. Thanks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

User/topic operator not starting with network policy #8739

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Strimzi

User/topic operator not starting with network policy #8739

Uh oh!

Uh oh!

nmichaud2 Jun 27, 2023

Replies: 2 comments · 3 replies

Uh oh!

Uh oh!

nmichaud2 Jun 27, 2023 Author

Uh oh!

scholzj Jun 27, 2023 Maintainer

Uh oh!

Uh oh!

nmichaud2 Jun 27, 2023 Author

Uh oh!

scholzj Jun 27, 2023 Maintainer

Uh oh!

nmichaud2 Jun 27, 2023 Author

nmichaud2
Jun 27, 2023

Replies: 2 comments 3 replies

nmichaud2
Jun 27, 2023
Author

scholzj
Jun 27, 2023
Maintainer

nmichaud2 Jun 27, 2023
Author

scholzj Jun 27, 2023
Maintainer

nmichaud2 Jun 27, 2023
Author