EKS Deployment fails with Unauthorized Error

[cmarquezrusso]

Hi,

We are currently deploying EKS clusters on Dev and Prod following https://docs.cloudposse.com/layers/eks/deploy-clusters/

While running the eks workflows, we are getting errors with the eks/storage-class

Error: Unauthorized

I think this is caused by not having access to EKS itself. I tried kubectl commands after configuring my kubeconfig on AWS cloudshell and I got this response:

~ $ kubectl get pods -A 
E0508 08:21:36.350576     400 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Note: This cluster in particular was provisioned with public endpoints so I can test AUTHZ from cloudshell

This issue can be reproduced on both Dev and Prod platform accounts.

Are we missing any configuration that allow us to get access to the kube api?

[cmarquezrusso]

More context information:

• We have AWS VPN up and running and I can confirm that the issue persists even with private access to eks.
• We are executing this as superadmin. There were no errors while deploying the clusters (aka making AWS API calls with Terraform)

[milldr]

EKS connectivity can be quite tedious to debug. I spent quite a bit of time debugging cluster connectivity, so we added these tips. Please take a look and let us know if it helps:
https://docs.cloudposse.com/layers/eks/faq/#common-connectivity-issues-and-solutions

[cmarquezrusso]

Thanks for that. The network layer is working fine. We are also using AWS VPN Client and I tested it against a bastion host.

The issue is with EKS Authorization (user is logged in) when applying K8s resources. In this case in particular, when its trying to create storage classes.

This is the debug info

2025-05-09T02:22:29.086Z [DEBUG] provider.terraform-provider-kubernetes: 2025/05/09 02:22:29 [INFO] Checking storage class gp2
2025-05-09T02:22:30.085Z [DEBUG] provider.terraform-provider-kubernetes: Sending HTTP Request: Authorization="Bearer k8s-" tf_http_req_body="" User-Agent="HashiCorp/1.0 Terraform/1.9.0" tf_http_req_version=HTTP/1.1 @caller=/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.36.0/helper/logging/logging_http_transport.go:160 Accept-Encoding=gzip tf_http_op_type=request tf_http_req_uri=/apis/storage.k8s.io/v1/storageclasses/gp2 @module=kubernetes.Kubernetes Accept="application/json, */*" Host=9D9A9F3873603466051DE157448C3E2E.gr7.us-west-2.eks.amazonaws.com new_logger_warning="This log was generated by a subsystem logger that wasn't created before being used. Use tflog.NewSubsystem to create this logger before it is used." tf_http_req_method=GET tf_http_trans_id=83dba29f-2b9d-9e58-d2af-b6c994cbe5a7 timestamp=2025-05-09T02:22:30.085Z
2025-05-09T02:22:30.800Z [DEBUG] provider.terraform-provider-kubernetes: Received HTTP Response: Content-Type=application/json Date="Fri, 09 May 2025 02:22:30 GMT"
  tf_http_res_body=
  | {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
   tf_http_res_status_code=401 @module=kubernetes.Kubernetes Cache-Control="no-cache, private" Content-Length=129 tf_http_res_status_reason="401 Unauthorized" Audit-Id=97411b43-f298-4f42-bb72-9a7f3fa40eb1 tf_http_trans_id=83dba29f-2b9d-9e58-d2af-b6c994cbe5a7 tf_http_op_type=response tf_http_res_version=HTTP/2.0 @caller=/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.36.0/helper/logging/logging_http_transport.go:160 new_logger_warning="This log was generated by a subsystem logger that wasn't created before being used. Use tflog.NewSubsystem to create this logger before it is used." timestamp=2025-05-09T02:22:30.799Z
2025-05-09T02:22:31.860Z [DEBUG] provider.terraform-provider-kubernetes: 2025/05/09 02:22:31 [DEBUG] Received error: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:"", RemainingItemCount:(*int64)(nil)}, Status:"Failure", Message:"Unauthorized", Reason:"Unauthorized", Details:(*v1.StatusDetails)(nil), Code:401}}
2025-05-09T02:22:31.860Z [ERROR] provider.terraform-provider-kubernetes: Response contains error diagnostic: diagnostic_summary=Unauthorized tf_rpc=ReadResource @module=sdk.proto diagnostic_detail="" diagnostic_severity=ERROR tf_proto_version=5.8 tf_provider_addr=registry.terraform.io/hashicorp/kubernetes tf_req_id=c85b1ba1-259a-5ccc-665f-1c165a4a1237 @caller=/home/runner/go/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.26.0/tfprotov5/internal/diag/diagnostics.go:58 tf_resource_type=kubernetes_storage_class_v1 timestamp=2025-05-09T02:22:31.860Z
2025-05-09T02:22:31.860Z [ERROR] vertex "kubernetes_storage_class_v1.ebs[\"gp2\"]" error: Unauthorized
2025-05-09T02:22:31.861Z [ERROR] vertex "kubernetes_storage_class_v1.ebs (expand)" error: Unauthorized

I tried tracking down the STS Token to understand why that session doesnt have access to the cluster. Seems that its using the OrganizationAccountAccessRole role

 <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  |   <GetCallerIdentityResult>
  |     <Arn>arn:aws:sts:::assumed-role/OrganizationAccountAccessRole/aws-go-sdk-1746763595726494877</Arn>
  |   </GetCallerIdentityResult>
  | </GetCallerIdentityResponse>

I went ahead and I've created a new IAM access entry on the EKS Cluster for the OrganizationAccountAccessRole with 3 policies attached:

• AmazonEKSAdminPolicy
• AmazonEKSAdminViewPolicy and
• AmazonEKSClusterAdminPolicy

I can confirm that the issue is there, as atmos workflow deploy/cluster -s plat-usw2-dev -f eks worked successfully after that.

[milldr]

I see. It's likely a result of attempting to access the cluster with the SuperAdmin user rather than a Permission Set or AWS Teams / Team role. Those are both typically included with auth map for the cluster.

ie

# stacks/catalog/eks.yaml
components:
  terraform:
    eks:
      vars:
        ...
        
        # List of `aws-teams-roles` (in the account where the EKS cluster is deployed) to map to Kubernetes RBAC groups
        aws_team_roles_rbac:
          - aws_team_role: admin
            groups:
              - system:masters
          - aws_team_role: poweruser
            groups:
              - idp:poweruser
          - aws_team_role: observer
            groups:
              - idp:observer
          - aws_team_role: planner
            groups:
              - idp:reader
          - aws_team_role: terraform
            groups:
              - system:masters

        # Permission sets from AWS SSO allowing cluster access
        # See `aws-sso` component.
        aws_sso_permission_sets_rbac:
        - aws_sso_permission_set: PowerUserAccess
          groups:
          - idp:poweruser
        - aws_sso_permission_set: ReadOnlyAccess
          groups:
          - idp:observer

[cmarquezrusso]

Agree. Thus is why I tested with AWS Cloudshell so I can get my own sts session using my SSO profile.

I tried with Leapp.app but I am getting errors assuming my SSO Admin Role

I am currently testing all of this from scratch (starting from DNS) I will add more info if the issue persists.

[cmarquezrusso]

The issue persists after re-running the steps required to create a k8s cluster from scratch, including DNS and ACM.

These are the access entries that are provisioned on EKS with EKS Admin Access

Some questions:

• Could you please confirm which user must be used for atmos workflow deploy/cluster -s plat-usw2-dev -f eks? I checked https://docs.cloudposse.com/layers/eks/deploy-clusters/ but there are no references to using a different user to SuperAdmin.
  TF logs shows arn:aws:sts::500000000:assumed-role/OrganizationAccountAccessRole as the role that is trying to connect with EKS K8S API.

• Do I need to create a new chain identity on Leapp for this? Or should I use identity as configured during https://docs.cloudposse.com/layers/identity/how-to-log-into-aws/

• Is this documented anywhere in the QuickStart guide? Just to be sure that I didn't miss an important section/step.

Thanks in advance.

[cmarquezrusso]

Also, I tried with cloudshell using my poweruser role, with an EKS cluster that has public endpoints for the K8S API (so I can authenticate)

Here is the log

~ $ aws sts get-caller-identity                                                                                                                                                                                                                                                                                                                                                                                              
{
    "UserId": "AROACCCCCCC:cristian@domain.net",
    "Account": "5000000000",
    "Arn": "arn:aws:sts::5000000000:assumed-role/AWSReservedSSO_PowerUserAccess_99CCCCCCCC/cristian@domain"
}
~ $ kubectl get pods -A 
Error from server (Forbidden): pods is forbidden: User "arn:aws:sts::5000000:assumed-role/AWSReservedSSO_PowerUserAccess_99CCCCCCCC/cristian-domain.net" cannot list resource "pods" in API group "" at the cluster scope
~ $ 

[dukarc]

Ok i think we figured this out. It probably is likely your guys bread and butter .. but it was confusing to us. Took a bit to get that we always use the core-identity in LEAPP and assume roles from there. The order of operations we followed is.

1. We needed to first log in using a chained role in leap to core-identity
2. Then we need to in geodesic run assume-role for the plat-dev as poweruser
3. Then set up kubectl config and all worked aws eks update-kubeconfig --name vbpt-plat-usw2-dev-eks-cluster --region us-west-2 -> kubectl get pods -n echo

[cmarquezrusso]

After additional testing, I can confirm that I can query K8S Resources withxxxx-plat-gbl-dev-poweruser

aws sts get-caller-identity
{
    "UserId": "AROXXXXXXXXX:botocore-session-174NNNNNN",
    "Account": "5000000",
    "Arn": "arn:aws:sts::500000:assumed-role/xxxx-plat-gbl-dev-poweruser/botocore-session-174NNNNNN"
}

k get pods -A
NAMESPACE        NAME                                               READY   STATUS    RESTARTS   AGE
alb-controller   aws-load-balancer-controller-8fd55c865-zcrdl       1/1     Running   0          23h
alb-controller   aws-load-balancer-controller-8fd55c865-zp8v2       1/1     Running   0          23h
[...]

However, if I try to apply eks/resources (or even to create a k8s cluster following the eks workflow) I get assume_role issues:

Error: Cannot assume IAM Role
│ 
│ IAM Role (arn:aws:iam::400000000:role/vbpt-core-gbl-root-tfstate) cannot be assumed.
│ 
│ There are a number of possible causes of this - the most common are:
│   * The credentials used in order to assume the role are invalid
│   * The credentials do not have appropriate permission to assume the role
│   * The role ARN is not valid
│ 
│ Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 469d67ba-0aaf-4aae-95e0-a9f79ad05100, api error AccessDenied:
│ User: arn:aws:sts::5000000:assumed-role/vbpt-plat-gbl-dev-poweruser/aws-go-sdk-1747117359031053462 is not authorized to perform: sts:AssumeRole on
│ resource: arn:aws:iam::4000000:role/vbpt-core-gbl-root-tfstate
│ 

These issues are gone when using vbpt-core-gbl-identity-devops so we will continue the next steps using this instead of SuperAdmin.