KafkaConnect in combination with istio

[ufobat]

I have strimzi in use in combination with istio and I struggle in getting the KafkaConnect Resource up and running.

I am planing to create a Kafka Connect Instance with the Splunk Connector. My Understanding iss that it needs to create an image first with the required software in it. This Image will be uploaded to a Container Registry. Right?

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: strimzi
  annotations:
    strimzi.io/use-connector-resources: "true"
  labels:
    app: KafkaConnect
spec:
  version: 3.2.3
  replicas: 1
  bootstrapServers: strimzi-kafka-bootstrap:9093
  authentication:
    type: scram-sha-512
    username: strimzi-connect-cluster-account
    passwordSecret:
      secretName: strimzi-connect-cluster-account
      password: password
  config:
    group.id: strimzi-connect-cluster
    offset.storage.topic: strimzi-connect-cluster-offsets
    config.storage.topic: strimzi-connect-cluster-configs
    status.storage.topic: strimzi-connect-cluster-status
    # -1 means it will use the default replication factor configured in the broker
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1
  build:
    output:
      type: docker
      image: harbor.k8s.loc/test/strimzi-connect-image:latest
      pushSecret: harbor-registry-secret
      additionalKanikoOptions:
        - '--skip-tls-verify'
        - '--skip-tls-verify-pull'
        - '--verbosity=trace'
        - '--skip-tls-verify-registry=harbor.k8s.loc'
        # - --insecure # harbor is reachable via https, no --insecure!
    plugins: []
      - name: splunk-sink
        artifacts:
          - type: zip
            url: https://d1i4a15mxbxib1.cloudfront.net/api/plugins/splunk/kafka-connect-splunk/versions/2.0.9/splunk-kafka-connect-splunk-2.0.9.zip
            sha512sum: a2435f3fda01c97ef267931bbab341daa592a22e4c6384213b025d5a445f1e5c179bf3510452d112c304e5be312ffd0a54498ddae5113423a5b63f3f95ecd3b7

Unfortunately I am not able to push the image and I dont know how to troubleshoot it or fix it.

As you can See i set verbosity to trace.

$ kubectl logs -f strimzi-connect-build
DEBU[0000] Copying file /dockerfile/Dockerfile to /kaniko/Dockerfile 
TRAC[0000] Adding /var/run to default ignore list       
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "harbor.k8s.loc/test/strimzi-connect-image:latest": creating push check transport for harbor.k8s.loc failed: Get "https://harbor.k8s.loc/v2/": dial tcp 172.20.8.41:443: connect: connection refused

I am aware that istio might be causing issues, but in the same kubernetes namespaces i've created an pod and check if that url is really not resolvable:

$ kubectl run my-shell --rm -i --tty --image alpine -- sh
If you don't see a command prompt, try pressing enter.
/ # apk upgrade; apk add curl;
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
OK: 7 MiB in 15 packages
(1/5) Installing ca-certificates (20220614-r4)
(2/5) Installing brotli-libs (1.0.9-r9)
(3/5) Installing nghttp2-libs (1.51.0-r0)
(4/5) Installing libcurl (7.87.0-r1)
(5/5) Installing curl (7.87.0-r1)
Executing busybox-1.35.0-r29.trigger
Executing ca-certificates-20220614-r4.trigger
OK: 9 MiB in 20 packages
/ # curl https://harbor.k8s.loc/v2/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
/ # curl -kv https://harbor.k8s.loc/v2/
*   Trying 172.20.8.41:443...
* Connected to harbor.k8s.loc (172.20.8.41) port 443 (#0)

This seems to work! I am aware that he Cert does not work, thats because the pod container does not know our company CA.

(Harbor itself works, i can use it from my laptop or in other contexts without problem)

How can I get more details in order to get this working?

[scholzj]

dial tcp 172.20.8.41:443: connect: connection refused sounds to me like it cannot connect to the registry -> so maybe it fails even before it gets to the TLS stuff?

[ufobat]

Yeah but why could that be the case!? I was not sure if this error message is indeed meaning what it was saying, since my-shell pod works in the same namespace with the same istio settings.

Is there a way to confirm / troubleshoot this?
I cant invoke a command in that pod, since it's crashing so fast.

[scholzj]

I do not know I'm afraid - I do not use Istio so I have no idea what it could or couldn't cause.

If needed, you can just build the container image with the additional plugins manually to work around it: https://strimzi.io/docs/operators/latest/full/deploying.html#creating-new-image-from-base-str

[ufobat]

I can not confirm that's a connectivity issue...

( it didnt work the very first time, but since then i did invoke the same kubectl command, and i do have internet access...
I will continue to look if I can figure out someting)

$ kubectl debug -it strimzi-connect-build --image=alpine 
Defaulting debug container name to debugger-grtcg.
If you don't see a command prompt, try pressing enter.
/ # apk upgrade
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
OK: 7 MiB in 15 packages
/ # apk add curl
(1/5) Installing ca-certificates (20220614-r4)
(2/5) Installing brotli-libs (1.0.9-r9)
(3/5) Installing nghttp2-libs (1.51.0-r0)
(4/5) Installing libcurl (7.87.0-r1)
(5/5) Installing curl (7.87.0-r1)
Executing busybox-1.35.0-r29.trigger
Executing ca-certificates-20220614-r4.trigger
OK: 9 MiB in 20 packages
/ # curl https://harbor.k8s.loc
curl: (60) SSL certificate problem: unable to get local issuer certificate

[ufobat]

After a day, without anything done by us, on the saturday morning it succeed to build and now everything works.
before that it was in a loop with dying builder container..

I would appreciate if there could be added more debugging posibilities.