-
Notifications
You must be signed in to change notification settings - Fork 1k
Description
Description
When using the Kubernetes and Helm Terraform providers with exec-based authentication (e.g. kubelogin in workload identity / OIDC mode), authentication fails during long-running Terraform applies because the OIDC federated token (e.g. from AZURE_FEDERATED_TOKEN_FILE) expires and is never refreshed for the duration of the apply. The azurerm provider (and azurerm backend) already support ID token refresh in this scenario; the same behavior is needed for providers that rely on kubelogin (or similar exec plugins) reading a federated token file.
Environment: Terraform 1.14.x, hashicorp/kubernetes (2.38.x), hashicorp/helm (3.1.x), kubelogin for AKS, Azure DevOps with Workload Identity Federation (OIDC). The pipeline writes the OIDC ID token to a file and sets AZURE_FEDERATED_TOKEN_FILE; kubelogin is used as the exec plugin and reads that file.
Current behavior: Before apply, the pipeline writes the OIDC token once. If apply runs longer than the token lifetime (~10 minutes), the federated token file is never updated. When Terraform later invokes the exec plugin (e.g. for a Kubernetes or Helm resource), kubelogin reads the expired token and fails:
Error: ... failed to create kubernetes rest client for update of resource: Get "https://<cluster>.hcp.<region>.azmk8s.io:443/api?timeout=32s": getting credentials: exec: executable kubelogin failed with exit code 1
Desired behavior (aligned with azurerm): Support OIDC / federated ID token refresh so that when the exec plugin (e.g. kubelogin) is invoked and the current federated token is expired, the provider (or the exec plugin) can obtain a new ID token (e.g. via the same mechanism as Terraform’s Azure DevOps OIDC refresh) and continue. Long-running applies using Kubernetes/Helm with OIDC (kubelogin) should keep working after the initial token expires.
Why it matters: Many users run Terraform in Azure DevOps (or GitHub Actions) with OIDC / workload identity and no long-lived secrets. Applies that exceed the token lifetime fail partway through, forcing workarounds (phased apply, reordering) instead of first-class token refresh like azurerm.
Potential Terraform Configuration
# Kubernetes and Helm providers using kubelogin with workload identity (OIDC).
# AZURE_FEDERATED_TOKEN_FILE is set by the pipeline and points to the OIDC token file.
# When apply runs longer than the token lifetime (~10 min), exec fails because the token is never refreshed.
provider "kubernetes" {
host = azurerm_kubernetes_cluster.this.kube_config[0].host
cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.this.kube_config[0].cluster_ca_certificate)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "kubelogin"
args = [
"get-token",
"--login", "workloadidentity",
"--environment", "AzurePublicCloud",
"--server-id", "<azure-ad-server-app-id>"
]
env = {
AZURE_CLIENT_ID = data.azurerm_client_config.current.client_id
AZURE_TENANT_ID = data.azurerm_client_config.current.tenant_id
AZURE_AUTHORITY_HOST = "https://login.microsoftonline.com/"
# AZURE_FEDERATED_TOKEN_FILE is set by CI (e.g. Azure DevOps); token expires ~10 min, never refreshed during apply
}
}
}References
- Azure DevOps: Introducing Azure DevOps ID Token Refresh and Terraform Task Version 5
- azurerm: Authenticating via Service Principal and OpenID Connect
- Authentication flow: Azure DevOps OIDC →
AZURE_FEDERATED_TOKEN_FILE→ kubelogin (exec) → Kubernetes/Helm providers.
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment