Kubernetes auth method does not work with "Use client JWT as reviewer JWT" option - with workaround

[jtv8]

Describe the bug
One of the options for configuring the Kubernetes auth method described in Kubernetes auth method is "Use client JWT as reviewer JWT". This does not work out-of-the-box with VSO and will result in a HTTP 403 error for PUT https://{VAULT_ADDR}/v1/auth/{MOUNT_PATH} . This is because Vault attempts to authenticate to the Kubernetes TokenReview API with the client JWT, but this JWT is missing the appropriate audience claim for the Kubernetes APIServer.

To Reproduce
Steps to reproduce the behavior:

1. Configure the Kubernetes Auth backend according to the supplied Terraform example
2. Deploy VSO using the supplied values.yaml
3. Apply the supplied CustomResources
4. Run kubectl describe VaultDynamicSecret app-db-creds -n demo-ns to see the following error:

Events:
  Type     Reason                  Age              From                Message
  ----     ------                  ----             ----                -------
  Warning  VaultClientConfigError  1s (x7 over 1s)  VaultDynamicSecret  Failed to get Vault client: Error making API request.

URL: PUT https://{VAULT_ADDR}/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied, lease_id=

Application deployment:

Auth config in Terraform:

resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
  path = "kubernetes"
}

resource "vault_kubernetes_auth_backend_config" "kubernetes_auth" {
  backend         = vault_auth_backend.kubernetes.path
  kubernetes_host = one(data.azurerm_kubernetes_cluster.aks.kube_config).host
  kubernetes_ca_cert = base64decode(
    one(data.azurerm_kubernetes_cluster.aks.kube_config).cluster_ca_certificate
  )
  disable_local_ca_jwt = true
}

values.yaml

defaultVaultConnection:
  enabled: true
  address: https://{VAULT_ADDR}

CustomResources:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: app
  namespace: demo-ns
spec:
  kubernetes:
    role: demo-ns-app
    serviceAccount: default
    tokenExpirationSeconds: 600
    audiences:
     - vault
  method: kubernetes
  mount: kubernetes
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: app-db-creds
  namespace: demo-ns
spec:
  destination:
    name: app-db-creds
    create: true
  mount: database
  path: creds/app-dbuser
  rolloutRestartTargets:
  - kind: Deployment
    name: app
  vaultAuthRef: app

Other useful info to include: kubectl describe deployment <app> and kubectl describe <vso-custom-resource> <app> output.

Expected behavior
When VSO generates the JWT for the ServiceAccount it should (by default) include the Kubernetes APIServer audience for the cluster in the aud claim. The VaultDynamicSecret will then be created as expected.

Alternatively, this behavior could be configured by an additional parameter in the VaultAuth CRD to tell VSO that the client JWT is being used as the token reviewer JWT.

Environment

• Kubernetes version: 1.26.3
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): AKS
  • Other configuration options or runtime services (istio, etc.): istio 1.18.2
• vault-secrets-operator version: 0.2.0

Additional context

Workaround

For now, you can resolve this by manually adding the APIServer audience to the VaultAuth CustomResource. For example, for Azure Kubernetes Service this looks like:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: app
  namespace: demo-ns
spec:
  kubernetes:
    role: demo-ns-app
    serviceAccount: default
    tokenExpirationSeconds: 600
    audiences:
     - vault
     - "https://{REDACTED}.hcp.westeurope.azmk8s.io"
  method: kubernetes
  mount: kubernetes

If you are unsure of the APIServer audience for your cluster, use one of these examples to decode the JWT mounted in /run/secrets/kubernetes.io/serviceaccount/token for an existing pod and inspect the contents of the aud claim.

[benashz]

Hi @jtv8, thank you for the detailed report. We will definitely look into getting an ideal fix out to address the issue. If you could confirm that you have set the audience to vault on the k8s-auth role, that would be great!

Thanks,

Ben

[jtv8]

Hi @benashz - thanks for looking into this! Yes, I can confirm that the audience was set to vault on the role. During troubleshooting I tried both with and without - the resulting error message was the same, as I believe the issue occurs with Vault's authentication with the APIServer, and this would prevent the call to the TokenReview API which I assume is where Vault's token audience gets checked.

[telchak]

Hi @benashz @jtv8 !
Thank you very much for this feedback, you avoided me some headache ^^
For information, and for anyone having the same issue, I encontered the same error on a GKE cluster using this Kubernetes auth method.

For me, the audience string to add in VaultAuth resource was like that:

• https://container.googleapis.com/v1/projects/{PROJECT_ID}/locations/{ZONE/REGION}/clusters/{CLUSTER_NAME}