Terraform.io to EKS "Error: Kubernetes cluster unreachable"

[eeeschwartz]

Terraform Version

0.12.19

Affected Resource(s)

• helm_release

Terraform Configuration Files

locals {
 kubeconfig = <<KUBECONFIG
apiVersion: v1
clusters:
- cluster:
    server: ${aws_eks_cluster.my_cluster.endpoint}
    certificate-authority-data: ${aws_eks_cluster.my_cluster.certificate_authority.0.data}
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "${aws_eks_cluster.my_cluster.name}"
KUBECONFIG
}

resource "local_file" "kubeconfig" {
  content  = local.kubeconfig
  filename = "/home/terraform/.kube/config"
}

resource "null_resource" "custom" {
  depends_on    = [local_file.kubeconfig]

  # change trigger to run every time
  triggers = {
    build_number = "${timestamp()}"
  }

  # download kubectl
  provisioner "local-exec" {
    command = <<EOF
      set -e

      curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.14.6/2019-08-22/bin/linux/amd64/aws-iam-authenticator
      chmod +x aws-iam-authenticator
      mkdir -p $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$PATH:$HOME/bin

      echo $PATH

      aws-iam-authenticator

      curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
      chmod +x kubectl

      ./kubectl get po
    EOF
  }
}

resource "helm_release" "testchart" {
  depends_on    = [local_file.kubeconfig]
  name          = "testchart"
  chart         = "../../../resources/testchart"
  namespace     = "default"
}

Debug Output

Note that

• kubectl get po reaches the cluster and reports "No resources found in default namespace."
• while helm_release reports: "Error: Kubernetes cluster unreachable"
• In earlier testing it errored with "Error: stat /home/terraform/.kube/config". Now that I write the local file to that location, it no longer errors. I assume that means it successfully reads the kube config.

https://gist.github.com/eeeschwartz/021c7b0ca66a1b102970f36c42b23a59

Expected Behavior

The testchart is applied

Actual Behavior

The helm provider is unable to reach the EKS cluster.

Steps to Reproduce

On terraform.io:

1. terraform apply

Important Factoids

Note that kubectl is able to communicate with the cluster. But something about the terraform.io environment, the .helm/config, or the helm provider itself renders the cluster unreachable.

Note of Gratitude

Thanks for all the work getting helm 3 support out the door. Holler if I'm missing anything obvious or can help diagnose further.

[eeeschwartz]

The token auth configuration below ultimately worked for me. Perhaps this should be the canonical approach for Terraform Cloud -> EKS, rather than using ~/.kube/config

provider "aws" {
  region = "us-east-1"
}

data "aws_eks_cluster_auth" "cluster-auth" {
  depends_on = [aws_eks_cluster.my_cluster]
  name       = aws_eks_cluster.my_cluster.name
}

provider "helm" {
  alias = "my_cluster"
  kubernetes {
    host                   = aws_eks_cluster.my_cluster.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.my_cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster-auth.token
    load_config_file       = false
  }
}

resource "helm_release" "testchart" {
  provider  = helm.my_cluster
  name       = "testchart"
  chart      = "../../../resources/testchart"
  namespace  = "default"
}

[kinihun]

I don't see how ths could possibly work, with Helm 3, it seems to be completely broken.
Below is my configuration and I can't connect to the cluster.
My kubernetes provider works but not the kubernetes block within Helm which has the same settings.

data "aws_eks_cluster" "cluster" {
  name = "foobar"
}

data "aws_eks_cluster_auth" "cluster" {
  name = "foobar"
}

provider "kubernetes" {
  version                = "1.10.0"
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
}

provider "helm" {
  version                = "1.0.0"

  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
    load_config_file       = false
  }
}```

[venky999]

seeing same issue

[kharandziuk]

@eeeschwartz can confirm it's working with newly created cluster. Here a sample configuration to prove it.

From other perspective with already created cluster I see the same issue and debug=true doesn't help at all

[vfiset]

anyone found a workaround yet ?

[kharandziuk]

@vfiset I believe there is no workaround. It's just an issue with policies(my guess).

The actual provider doesn't give you enough debug information. So, you will probably need to run helm install manually to find an issue.

[rgardam]

My guess is that the aws-auth config map is blocking access. In the example that @kharandziuk has show here there's no aws-auth configmap defined. Also it's worth noting that the usage of helm here is in the same terraform run as the eks run which means that the default credentials for eks are the ones being used to deploy helm.

I have a fairly complicated setup where i'm assuming roles between the different stages of the EKS cluster deployment.

• https://aws.amazon.com/premiumsupport/knowledge-center/amazon-eks-cluster-access/
  Have a look at how you have to configure aws-auth to allow access to your cluster.

[leoddias]

Seeing same issue using Helm3. My tf looks like as @kinihun ...
Its happens in the first run of "terraform apply", when I try to exec again everything goes well.

[netflash]

I have created deployed a helm chart via helm provider ages ago. It works fine, I can change things here and there, etc.
Today I wanted to "migrate" a standalone-deployed helm chart to be managed under terraform. So when I tried to run terraform import helm_release.chart namespace/chart, I've got this error.