Best way to integrate Vault Secrets Operator from multiple Kubernetes clusters with an external Vault

[SergeiCherevko]

Hi everyone,

I have a setup where Vault runs in its own dedicated cluster, and there are about 10 other Kubernetes clusters, each running VSO.

I’m trying to figure out the best way to handle authentication and TokenReview between these clusters.

From what I understand, there are a few options — but all of them have caveats in a multi-cluster setup:

• Use local token as reviewer JWT – not possible, since Vault and VSO are in different clusters.

• Use client JWT as reviewer JWT – not ideal, because it requires creating many additional auth-delegator roles for each VaultAuth resource (we have hundreds).

• Use a long-lived token as reviewer JWT – seems like the only viable option?
  Is there a way to configure VSO (via your VSO Helm) so that it generates or uses a persistent long-lived token from the default service account, which could be reused across clusters?

• Use JWT auth instead? – would this be a better fit for external Vault setups?

We don’t plan to have shared VaultAuth per cluster — permissions are granted very specifically on a per-resource basis.

[mlec1]

We went for a service account with a long lived token in each kubernetes cluster running VSO.
Each cluster has its own kubernetes auth

[SergeiCherevko]

@mlec1 Thank you for your answer! We do the same.

I can’t understand why in the VSO Helm chart we can’t set up a long-lived token and RBAC for the service account via values (like in BanzaiCloud). I mean something like:

enable_rbac: true
enable_long_live_token: true

So I have to manually create two additional manifests in each cluster — this is inconvenient.

---
apiVersion: v1
kind: Secret
metadata:
  name: vso-token-reviewer-secret
  annotations:
    kubernetes.io/service-account.name: vault-vault-secrets-operator-controller-manager
type: kubernetes.io/service-account-token

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vso-token-reviewer
subjects:
  - kind: ServiceAccount
    name: vault-vault-secrets-operator-controller-manager
    namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator

[mlec1]

Yes we had to do the same, creating the Service Account and ClusterRoleBinding manually