Strimzi
Operation: [get] for kind: [Secret] with name: [my-secret] in namespace: [my-namespace] failed. #8963
Answered
by
scholzj
ndellosa95
asked this question in
Q&A
Operation: [get] for kind: [Secret] with name: [my-secret] in namespace: [my-namespace] failed.
#8963
-
|
I am attempting to use the Kubernetes secret config provider based off of the guide here. However, I am getting the following error when the Kafka Connect pod starts up: I have followed the guide and can see on Argo that I have the role with the following manifest: apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"Role","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"my-namespace"},"name":"my-namespace-role","namespace":"my-namespace"},"rules":[{"apiGroups":[""],"resourceNames":["my-secret","value.converter.schema.registry.basic.auth.user.info"],"resources":["secrets"],"verbs":["get"]}]}
creationTimestamp: '2023-08-07T12:15:22Z'
labels:
app.kubernetes.io/instance: my-namespace
name: my-namespace-role
namespace: my-namespace
resourceVersion: '68943993'
uid: 6eb752f8-2f0d-4a57-82bf-178af96d9320
rules:
- apiGroups:
- ''
resourceNames:
- my-secret
- value.converter.schema.registry.basic.auth.user.info
resources:
- secrets
verbs:
- getAnd the associated role binding: apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"my-namespace"},"name":"my-namespace-role-binding","namespace":"my-namespace"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"Role","name":"my-namespace-role"},"subjects":[{"kind":"ServiceAccount","name":"my-namespace-connect","namespace":"my-namespace"}]}
creationTimestamp: '2023-08-08T19:29:03Z'
labels:
app.kubernetes.io/instance: my-namespace
name: my-namespace-role-binding
namespace: my-namespace
resourceVersion: '69042481'
uid: f486aafc-fd70-4ba9-bc17-5ab3f719d87c
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: my-namespace-role
subjects:
- kind: ServiceAccount
name: my-namespace-connect
namespace: my-namespaceAnd the service account created by Strimzi: apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: '2023-08-07T21:11:45Z'
labels:
app.kubernetes.io/instance: my-namespace
app.kubernetes.io/managed-by: strimzi-cluster-operator
app.kubernetes.io/name: kafka-connect
app.kubernetes.io/part-of: strimzi-my-namespace
strimzi.io/cluster: my-namespace
strimzi.io/component-type: kafka-connect
strimzi.io/kind: KafkaConnect
strimzi.io/name: my-namespace-connect
name: my-namespace-connect
namespace: my-namespace
ownerReferences:
- apiVersion: kafka.strimzi.io/v1beta2
blockOwnerDeletion: false
controller: false
kind: KafkaConnect
name: my-namespace
uid: 2fbe4a31-dffc-42ef-957e-9cf9cd7885a3
resourceVersion: '68051682'
uid: c5193644-4482-4f8f-8c03-26dc5fee8135Finally, I can confirm that the pod has the following in its spec configuration: serviceAccount: my-namespace-connect
serviceAccountName: my-namespace-connectI honestly am completely stumped on what to do from here and the error messaging doesn't provide any clues, though I at least know it sees that the secret exists because that is a separate error message. Any ideas on how to get more details on what exactly the issue here is? |
Beta Was this translation helpful? Give feedback.
Answered by
scholzj
Aug 14, 2023
Replies: 1 comment 9 replies
-
|
I guess you should share the KafkaConnect / KafkaConnector resources as well as the full log from Connect. The exception seems to be missing a proper stack trace, not sure that is normal - maybe there is something more. You should also share what version are you using. |
Beta Was this translation helpful? Give feedback.
9 replies
-
|
I tried with a plaintext username and enabled trace level logging but am getting the same error for |
Beta Was this translation helpful? Give feedback.
-
|
I failed to reproduce it. I did not get a similar log with any error states which come to my mind. In any case, the logging is broken, so it is not really clear what the exception is. I opened strimzi/kafka-kubernetes-config-provider#24 to fix it. If you want, |
Beta Was this translation helpful? Give feedback.
-
|
I'm back from vacation and looking at this again. Thank you for providing that image, I'm able to see the full stack trace now, which is as follows. I'm going to check with our cloud engineers, but figured I'd post here to in case 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Well, at least we now get the full exception. But it looks like some weird networking or TLS issue which to be honest might mean anything. Is anything special about your environment? Such as using IPv6 only or something like that? |
Beta Was this translation helpful? Give feedback.
Answer selected by
ndellosa95
-
|
Took us a few days but we had to carve out some kind of exception in our service mesh sidecar (or at least, that's my understanding of what we did, I'm not a network engineer 😅) to get it to work. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting. Thanks for sharing back ... its good to have an idea about these issues if it happens to someone else in the future 👍 |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Well, at least we now get the full exception. But it looks like some weird networking or TLS issue which to be honest might mean anything. Is anything special about your environment? Such as using IPv6 only or something like that?