KubernetesSecretConfigProvider Looking in Multiple Namespaces for Secret

[shubhamg7]

Hello! We are setting up Kafka Connectors that provide database passwords via the KubernetesSecretConfigProvider. Originally we had one connector running perfectly fine using the KubernetesSecretConfigProvider to provide the secret from the same namespace as the connector. However when we have setup two connectors in different namespaces they are trying to read the secret value not only in its own namespace but also the other connectors namespace. Is this expected behavior?

I have been following this blog post for guidance.

Here's the exact config:

Connector A:

# ROLE
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: connector-configuration-role-A
  namespace: namespace-A
rules:
- apiGroups:
  - ""
  resourceNames:
  - my-secret
  resources:
  - secrets
  verbs:
  - get

# Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: connector-configuration-role-binding
  namespace: namespace-A
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: connector-configuration-role-A
subjects:
- kind: ServiceAccount
  name: my-connect-A-connect  # created by strimzi
  namespace: namespace-A

# Connector
database.password: ${secrets:namespace-A/my-secret:my-key}

Connector B:

# ROLE
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: connector-configuration-role-B
  namespace: namespace-B
rules:
- apiGroups:
  - ""
  resourceNames:
  - my-secret
  resources:
  - secrets
  verbs:
  - get

# Role Binding
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: connector-configuration-role-binding
  namespace: namespace-B
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: connector-configuration-role-B
subjects:
- kind: ServiceAccount
  name: my-connect-B-connect  # created by strimzi
  namespace: namespace-B

# Connector
database.password: ${secrets:namespace-B/my-secret:my-key}

In both of the pods' logs I can also see the pod trying to retrieve the secret from the two distinct namespaces:

INFO Retrieving configuration from Secret my-secret in namespace namespace-A
INFO Retrieving configuration from Secret my-secret in namespace namespace-B

One of these retrievals succeeds but the other causes the pods for each connector to crash with an error like so:

org.apache.kafka.common.config.ConfigException: Invalid value io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://172.20.0.1/api/v1/namespaces/<namespace-name>/secrets/<secret-name>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. secrets "<secret-name>" is forbidden: User "system:serviceaccount:<service-account-name>" cannot get resource "secrets" in API group "" in the namespace "<namespace-name>".

Ideally each pod should only look for the secret in its own namespace rather than trying to fetch the secret in another namespace. Please let me know if there is a fix for this and if theres anything I can expand on!

[scholzj]

Well, that is what you are asking it to do, to read it from both namespaces, or? So that seems correct. But you configured the RoleBinding incorrectly. The service account namespace (.subjects[].namespace) needs to point to the namespace where your KafkaConnect is deployed => so it should be the same in both cases.

[shubhamg7]

The connector is trying to read from both namespaces but I would like for it to only read from the namespace that the connector exists at.

For example:

• if the connector is running in namespace A, it should only look for the secret in namespace A
• if the connector that is running in namespace B should only look for the secret in namespace B

Currently the connector running in namespace A gets the secret from namespace A and also tries to fetch the secret in namespace B. This violates the permission boundary we have set and would like to stop the second secret retrieval from happening since it is not needed.

As for your comment about the service account namespace, I have two separate deployments of Kafka Connect (one in each namespace) so it right for the service account namespaces to be different? Please let me know if I am misunderstanding something.

[scholzj]

The KafkaConnector resource is only virtual. The connector as well as the config provider whcih is getting the secrets are all running inside the same pod. If you want a real segregation between the connectors, you would need to use separate Connects.

Now the second part of your previous comment suggests that you already have it, so that makes it more intresting. Can you share all the YAML files you have to doublecheck them?

Did you followed this guide - https://strimzi.io/docs/operators/latest/full/deploying.html#con-kafka-connect-multiple-instances-str - to make sure the Connect clusters are configured as independent?

[shubhamg7]

Ahh thank you for the suggestion. Yes I do have two separate Kafka Connect deployments, but they do share the same config values for due to how I was templating these connectors.

offset.storage.topic
config.storage.topic
status.storage.topic

I will try changing this and report back with the results!

[shubhamg7]

This seems to have worked thank you!