Support AWS EKS Pod Identity Associations

[jtele2]

Description

Feature Request: Enable Elastic Cloud on Kubernetes (ECK) to natively support AWS Pod Identity Association (PIA) for managing IAM roles and credentials when running Elasticsearch on Amazon EKS.

Background: Currently, ECK supports IAM Roles for Service Accounts (IRSA) to manage AWS permissions, especially for S3 Snapshots. However, AWS EKS introduced Pod Identity Association (PIA), which allows managing IAM role associations at the cluster level without requiring service account annotations. This is beneficial for simplifying permissions management and avoiding hard-coded annotations.

When using PIA, the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable set by ECK's init containers conflicts with the expected behavior of PIA, causing pods to fall back to ECS-style credential retrieval rather than using the IAM role associated via PIA.

Supporting PIA would allow smoother integration and avoid manual overrides or workarounds for environment variables.

Current Behavior

Platform: Amazon EKS
ECK Version: 2.14.0
Elasticsearch Version: 8.15.3
EKS Version: 1.30
eksctl Version: 0.194.0-dev+02ef28ee3.2024-10-22T18:42:34Z
kubectl Version

• Client Version: v1.31.2
• Kustomize Version: v5.4.2
• Server Version: v1.30.6-eks-7f9249a

When deploying an Elasticsearch cluster using ECK with PIA, the elastic-internal-init-filesystem init container sets the following environment variables:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

These variables cause the AWS SDK to attempt credential retrieval via the ECS metadata endpoint, which conflicts with the PIA mechanism and leads to permission issues. The expected behavior for PIA is to use the Web Identity Token mounted by the service account to assume the IAM role.

Expected Behavior

ECK should detect and respect Pod Identity Associations (PIA) when deployed on EKS.
The init containers should not set ECS-specific environment variables (AWS_CONTAINER_CREDENTIALS_FULL_URI) when PIA is being used.
Elasticsearch should seamlessly use the IAM role associated via PIA, leveraging the Web Identity Token.

Steps to Reproduce

Create an EKS cluster with Pod Identity Association enabled using eksctl:

eksctl create cluster -f cluster-config.yml
Create a Pod Identity Association for the aws-sa service account in the eck namespace:

eksctl create podidentityassociation \
  --cluster DevCluster02 \
  --namespace eck \
  --service-account aws-sa \
  --role-arn arn:aws:iam::YOUR_ACCOUNT_ID:role/eck-snapshots-role

Deploy Elasticsearch with ECK using the following configuration:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es
  namespace: eck
spec:
  version: 8.15.3
  nodeSets:
    - name: master
      count: 3
      podTemplate:
        spec:
          serviceAccountName: aws-sa

Check the environment variables in the init container:

kubectl exec -n eck es-es-master-0 -- env | grep AWS

You will see:

AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials

Error Logs

Attempt to perform an S3 snapshot operation, which fails due to incorrect credential retrieval. Relevant error logs:

[YOUR_BUCKET] path  is not accessible on master node","error.stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [YOUR_BUCKET] path  is not accessible on master node
Caused by: com.amazonaws.SdkClientException: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host can only be one of [localhost, 127.0.0.1]

Click to Expand Error Logs

[YOUR_BUCKET] path is not accessible on master node","error.stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [YOUR_BUCKET] path is not accessible on master node Caused by: java.io.IOException: Exception when listing blobs by prefix [index-] \tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:356) \tat org.elasticsearch.server@8.15.3/org.elasticsearch.repositories.blobstore.BlobStoreRepository.listBlobsToGetLatestIndexId(BlobStoreRepository.java:3099) \tat org.elasticsearch.server@8.15.3/org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:3069) \tat org.elasticsearch.server@8.15.3/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2148) \tat org.elasticsearch.server@8.15.3/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:361) \tat org.elasticsearch.server@8.15.3/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:36) \tat org.elasticsearch.server@8.15.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) \tat org.elasticsearch.server@8.15.3/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) \tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) \tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) \tat java.base/java.lang.Thread.run(Thread.java:1570) Caused by: com.amazonaws.SdkClientException: The full URI (http://169.254.170.23/v1/credentials) contained withing environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host can only be one of [localhost, 127.0.0.1] \tat com.amazonaws.auth.ContainerCredentialsProvider$FullUriCredentialsEndpointProvider.getCredentialsEndpoint(ContainerCredentialsProvider.java:121) \tat com.amazonaws.auth.ContainerCredentialsFetcher.getCredentialsResponse(ContainerCredentialsFetcher.java:38) \tat com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124) \tat com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80) \tat com.amazonaws.auth.ContainerCredentialsProvider.getCredentials(ContainerCredentialsProvider.java:71) \tat com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75) \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319) \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) \tat org.elasticsearch.repositories.s3.S3Service$PrivilegedAWSCredentialsProvider.getCredentials(S3Service.java:313) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:845) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:794) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) \tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) \tat com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6431) \tat com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:6404) \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5441) \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397) \tat com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928) \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$15(S3BlobContainer.java:403) \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:319) \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) \tat org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:403) \tat org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:347) \t... 10 more

[pebrc]

ECK is not setting the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable.

[jtele2]

ECK is not setting the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable.

Is it set in the base elastic image then? Why doesn't PIA work OOB with ECK managed elastic clusters?

[barkbay]

Is it set in the base elastic image then?

Or by a webhook: https://github.com/aws/amazon-eks-pod-identity-webhook/blob/90b4c5247a1c73a5508f209a792e1efa67f2bf07/pkg/handler/handler.go#L234-L243

[alex-kattathra-johnson]

Elasticsearch uses a version of the AWS SDK that doesn't support pod identity. Reference: elastic/elasticsearch#122431

[jtele2]

Elasticsearch uses a version of the AWS SDK that doesn't support pod identity. Reference: elastic/elasticsearch#122431

They’re still on v1! Good find. Now this all makes sense. As a work around, I did what was suggested in the above thread.

Hope to get v2 soon, very surprised @elasticsearch!

[jtele2]

Also @pebrc - I don’t understand why this was removed from triage and assigned a non-issue. Not being able to use PIAs is a huge issue. Just because I didn’t diagnose the root cause of the problem correctly, doesn’t make the whole issue a non-issue…

[alex-kattathra-johnson]

IRSA is the recommended/documented way at the moment, but the pod identity is the AWS best practice because it uses the metadata server to authenticate, is more streamlined/k8s native and secure.

[pebrc]

@jtele2 I did not mean to downplay the severity of this issue. The reason I labeled this is a non-issue is that the underlying issue is in Elasticsearch itself as @alex-kattathra-johnson has found out. There is nothing actionable on the side of the ECK operator for this problem based on my current understanding. We use this repository exclusively for issues related to the orchestration of Elasticsearch and other Elastic stack applications through the ECK operator e.g. bugs or feature requests. I hope this makes sense.

[jtele2]

@pebrc I understand, that makes sense. Thanks for explaining yourself. I guess we can close this then, seeing there's a related gh issue already.

[siminn-arnorgj]

I can confirm that EKS Pod Identity works on Elasticsearch 9.1.0 if you set the following on all containers/pods using the plugin, of course in addition to using a service account associated with a pod identity.

          containers:
            - name: elasticsearch
              env:
                - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
                  value: /usr/share/elasticsearch/config/repository-s3/eks-pod-identity-token
                - name: AWS_CONTAINER_CREDENTIALS_FULL_URI
                  value: http://169.254.170.23/v1/credentials
          initContainers:
            - name: symlink-token
              command:
                - sh
                - -c
                - mkdir -p "/usr/share/elasticsearch/config/repository-s3"; ln -s /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
                  "/usr/share/elasticsearch/config/repository-s3/eks-pod-identity-token"

[alex-kattathra-johnson]

The token has to be in the specific /usr path? Can we set the env var to /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token?