Snapshot/restore repository-s3 –> Using IAM roles for Kubernetes service accounts for authentication is not working

[ahmetdd]

Elasticsearch Version

Version 8.9.0

Installed Plugins

No response

Java Version

bundled

OS Version

debian-sid-slim:v23.29.0

Problem Description

Hi,

Currently we are following the guide here[1]. We can successfully assign proper service account(hence AWS iam role) to the es pods. If relevant here[2] you can see our redacted elasticsearch CRD file.

This correctly results in assigning the proper role to the es nodes(I can test if the role works simply by logging in to pod console & test some s3 copy operations.)

Next I refer to [1] to create the symlinks in the folder - here is a catch, "ES_PATH_CONF" is not set, hence we set it to " /usr/share/elasticsearch/config" & create a link to the AWS TOKEN as written in same documentation.

But still when we try to verify the repository connection status we get the error [3]

Any idea or guidance is appreciated, we are stuck.

Thanks!
Ahmet

Steps to Reproduce

1. S3 repository | Elasticsearch Guide [8.5] | Elastic

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name:  <REDACTED>
spec:
  version: <REDACTED>
  image: <REDACTED>
  auth:
    fileRealm:
      - secretName: <REDACTED>
      - secretName: <REDACTED>
  nodeSets:
  - name: default
    count: 2
    podTemplate:
      spec:
        serviceAccount: <SA Name that binds to proper AWS Role>
        serviceAccountName: <SA Name that binds to proper AWS Role>
        imagePullSecrets:
          - name: <REDACTED>
        containers:
          - name: elasticsearch
            resources: <REDACTED>
    volumeClaimTemplates:
      - metadata:
          name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: <REDACTED>
    config:
      node.store.allow_mmap: <REDACTED>

{
  "name": "ResponseError",
  "meta": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "repository_verification_exception",
            "reason": "[repository-s3] path [<REDACTED>] is not accessible on master node"
          }
        ],
        "type": "repository_verification_exception",
        "reason": "[repository-s3] path [<REDACTED>] is not accessible on master node",
        "caused_by": {
          "type": "i_o_exception",
          "reason": "Unable to upload object [<REDACTED>/tests-hQo9BhOSQ5S1VGbd2mCXoA/master.dat] using a single upload",
          "caused_by": {
            "type": "amazon_s3_exception",
            "reason": "amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: <REDACTED>; S3 Extended Request ID: <REDACTED>; Proxy: null)"
          }
        }
      },
      "status": 500
    },
    "statusCode": 500,
    "headers": {
      "x-opaque-id": "<REDACTED>;kibana:application:management:",
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "732"
    },
    "meta": {
      "context": null,
      "request": {
        "params": {
          "method": "POST",
          "path": "/_snapshot/repository-s3/_verify",
          "querystring": "",
          "headers": {
            "user-agent": "Kibana/8.9.0",
            "x-elastic-product-origin": "kibana",
            "authorization": "Basic <REDACTED>",
            "x-opaque-id": "<REDACTED>;kibana:application:management:",
            "x-elastic-client-meta": "es=8.8.0p,js=16.20.1,t=8.3.1,hc=16.20.1",
            "accept": "application/vnd.elasticsearch+json; compatible-with=8,text/plain"
          }
        },
        "options": {
          "opaqueId": "<REDACTED>;kibana:application:management:",
          "headers": {
            "x-elastic-product-origin": "kibana",
            "user-agent": "Kibana/8.9.0",
            "authorization": "Basic <REDACTED>",
            "x-opaque-id": "<REDACTED>",
            "x-elastic-client-meta": "es=8.8.0p,js=16.20.1,t=8.3.1,hc=16.20.1"
          }
        },
        "id": 1
      },
      "name": "elasticsearch-js",
      "connection": {
        "url": "<REDACTED>",
        "id": "<REDACTED>",
        "headers": {},
        "status": "alive"
      },
      "attempts": 0,
      "aborted": false
    },
    "warnings": null
  }
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[akumar-99]

Way around which worked for me (Found it here and later realised it should be enabled on all nodes from here):

• IRSA created
• Add the following as extra to all the nodes:

          containers:
            - name: elasticsearch
              env:
                - name: AWS_ROLE_SESSION_NAME
                  value: repository-s3 (Any name works)
          initContainers:
            - name: symlink-token (Any name works)
              command:
                - sh
                - -c
                - mkdir -p "/usr/share/elasticsearch/config/repository-s3"; ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "/usr/share/elasticsearch/config/repository-s3/aws-web-identity-token-file"
          serviceAccount: us-east-1-elasticsearch-nodes```

[vatsal-kavida]

Way around which worked for me (Found it here and later realised it should be enabled on all nodes from here):

* IRSA created

* Add the following as extra to all the nodes:

          containers:
            - name: elasticsearch
              env:
                - name: AWS_ROLE_SESSION_NAME
                  value: repository-s3 (Any name works)
          initContainers:
            - name: symlink-token (Any name works)
              command:
                - sh
                - -c
                - mkdir -p "/usr/share/elasticsearch/config/repository-s3"; ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "/usr/share/elasticsearch/config/repository-s3/aws-web-identity-token-file"
          serviceAccount: us-east-1-elasticsearch-nodes```

After trying your solution I started getting the following eror, I am attaching the logs and mainfest below ->
I have deployed a elasticsearch cluster using eck-operator on my eks cluster. I am trying to restore a snapshot stored in my s3 by trying to connect to it. here is the manifest I used ->

---
# Default values for eck-elasticsearch.
# This is a YAML-formatted file.

# Overridable names of the Elasticsearch resource.
# By default, this is the Release name set for the chart,
# followed by 'eck-elasticsearch'.
#
# nameOverride will override the name of the Chart with the name set here,
# so nameOverride: quickstart, would convert to '{{ Release.name }}-quickstart'
#
# nameOverride: "quickstart"
#
# fullnameOverride will override both the release name, and the chart name,
# and will name the Elasticsearch resource exactly as specified.
#
# fullnameOverride: "quickstart"

# Version of Elasticsearch.
#
version: 8.11.1

# Elasticsearch Docker image to deploy
#
# image:

# Labels that will be applied to Elasticsearch.
#
labels:
  deployment : dev

# Annotations that will be applied to Elasticsearch.
#
annotations:
  eck.k8s.elastic.co/license: basic

# Settings for configuring Elasticsearch users and roles.
# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-users-and-roles.html
#
auth: {}

# Settings for configuring stack monitoring.
# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-stack-monitoring.html
#
monitoring: {}
  # metrics:
  #   elasticsearchRefs:
  #   - name: monitoring
  #     namespace: observability
  # logs:
  #   elasticsearchRefs:
  #   - name: monitoring
  #     namespace: observability

# Control the Elasticsearch transport module used for internal communication between nodes.
# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-transport-settings.html
#
transport: {}
  # service:
  #   metadata:
  #     labels:
  #       my-custom: label
  #   spec:
  #     type: LoadBalancer
  # tls:
  #   subjectAltNames:
  #     - ip: 1.2.3.4
  #     - dns: hulk.example.com
  #   certificate:
  #     secretName: custom-ca

# Settings to control how Elasticsearch will be accessed.
# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-accessing-elastic-services.html
#
http:
   service:
     spec:
       # expose this cluster Service with a LoadBalancer
       type: NodePort
   tls:
     selfSignedCertificate:
       disabled: true
  # service:
  #   metadata:
  #     labels:
  #       my-custom: label
  #   spec:
  #     type: LoadBalancer
  # tls:
  #   selfSignedCertificate:
  #     # To fully disable TLS for the HTTP layer of Elasticsearch, simply
  #     # set the below field to 'true', removing all other fields.
  #     disabled: false
  #     subjectAltNames:
  #       - ip: 1.2.3.4
  #       - dns: hulk.example.com
  #   certificate:
  #     secretName: custom-ca

# Control Elasticsearch Secure Settings.
# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-es-secure-settings.html#k8s-es-secure-settings
#
secureSettings: []
  # - secretName: one-secure-settings-secret
  # Projection of secret keys to specific paths
  # - secretName: gcs-secure-settings
  #   entries:
  #   - key: gcs.client.default.credentials_file
  #   - key: gcs_client_1
  #     path: gcs.client.client_1.credentials_file
  #   - key: gcs_client_2
  #     path: gcs.client.client_2.credentials_file

# Settings for limiting the number of simultaneous changes to an Elasticsearch resource.
# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-update-strategy.html
#
updateStrategy: {}
  # changeBudget:
  #   maxSurge: 3
  #   maxUnavailable: 1

# Controlling of connectivity between remote clusters within the same kubernetes cluster.
# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-remote-clusters.html
#
remoteClusters: {}
  # - name: cluster-two
  #   elasticsearchRef:
  #     name: cluster-two
  #     namespace: ns-two

# VolumeClaimDeletePolicy sets the policy for handling deletion of PersistentVolumeClaims for all NodeSets.
# Possible values are DeleteOnScaledownOnly and DeleteOnScaledownAndClusterDeletion.
# By default, if not set or empty, the operator sets DeleteOnScaledownAndClusterDeletion.
#
volumeClaimDeletePolicy: ""

# Settings to limit the disruption when pods need to be rescheduled for some reason such as upgrades or routine maintenance.
# By default, if not set, the operator sets a budget that doesn't allow any pod to be removed in case the cluster is not green or if there is only one node of type `data` or `master`.
# In all other cases the default PodDisruptionBudget sets `minUnavailable` equal to the total number of nodes minus 1.
# To completely disable the pod disruption budget set `disabled` to true.
#
# podDisruptionBudget:
#   spec:
#     minAvailable: 2
#     selector:
#       matchLabels:
#         elasticsearch.k8s.elastic.co/cluster-name: quickstart
#   disabled: true

# Used to check access from the current resource to a resource (for ex. a remote Elasticsearch cluster) in a different namespace.
# Can only be used if ECK is enforcing RBAC on references.
#
# serviceAccountName: ""

# Number of revisions to retain to allow rollback in the underlying StatefulSets.
# By default, if not set, Kubernetes sets 10.
#
# revisionHistoryLimit: 2

# Node configuration settings.
# The node roles which can be configured here are:
# - "master"
# - "data_hot"
# - "data_cold"
# - "data_frozen"
# - "data_content"
# - "ml"
# - "ingest"
# ref: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-node-configuration.html
#
nodeSets:
- name: elk
  config:
    # most Elasticsearch configuration parameters are possible to set, e.g: node.attr.attr_name: attr_value
    node.roles: ["master", "data", "ingest", "ml"]
    # this allows ES to run on nodes even if their vm.max_map_count has not been increased, at a performance cost
    node.store.allow_mmap: false
    # uncomment the lines below to use the zone attribute from the node labels
    #cluster.routing.allocation.awareness.attributes: k8s_node_name,zone
    #node.attr.zone: ${ZONE}
  podTemplate:
    metadata:
      labels:
        # additional labels for pods
        deployment: dev
    spec:
      serviceAccountName: "elastic-operator"
      initContainers:
        - name: symlink-token
          command:
            - sh
            - -c
            - mkdir -p "/usr/share/elasticsearch/config/repository-s3"; ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "/usr/share/elasticsearch/config/repository-s3/aws-web-identity-token-file"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: "Deployment"
                    operator: In
                    values:
                      - "kafka"
      # this changes the kernel setting on the node to allow ES to use mmap
      # if you uncomment this init container you will likely also want to remove the
      # "node.store.allow_mmap: false" setting above
      # initContainers:
      # - name: sysctl
      #   securityContext:
      #     privileged: true
      #     runAsUser: 0
      #   command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
      ###
      # uncomment the line below if you are using a service mesh such as linkerd2 that uses service account tokens for pod identification.
      # automountServiceAccountToken: true
      containers:
      - name: elasticsearch
        # specify resource limits and requests
        resources:
          limits:
            memory: 6Gi
            cpu: 1
          requests:
            memory: 2Gi
            cpu: 200m
        env:
        # uncomment the lines below to make the topology.kubernetes.io/zone annotation available as an environment variable and
        # use it as a cluster routing allocation attribute.
        - name: AWS_ROLE_SESSION_NAME
          value: repository-s3
        #  valueFrom:
        #    fieldRef:
        #      fieldPath: metadata.annotations['topology.kubernetes.io/zone']
        - name: ES_JAVA_OPTS
          value: "-Xms2g -Xmx2g"
      #topologySpreadConstraints:
      #  - maxSkew: 1
      #    topologyKey: topology.kubernetes.io/zone
      #    whenUnsatisfiable: DoNotSchedule
      #    labelSelector:
      #      matchLabels:
      #        elasticsearch.k8s.elastic.co/cluster-name: elasticsearch-sample
      #        elasticsearch.k8s.elastic.co/statefulset-name: elasticsearch-sample-es-default
  count: 1
#   # request 2Gi of persistent data storage for pods in this topology element
  volumeClaimTemplates:
   - metadata:
       name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
     spec:
       accessModes:
       - ReadWriteOnce
       resources:
         requests:
           storage: 150Gi
       storageClassName: gp3
        # SecurityContext defines the security options the container should be run with.
        # If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.
        #
        # These typically are set automatically by the ECK Operator, and should only be adjusted
        # with the full knowledge of the effects of each field.
        #
        # securityContext:

          # Whether this container has a read-only root filesystem. Default is false.
          # readOnlyRootFilesystem: false

          # The GID to run the entrypoint of the container process. Uses runtime default if unset.
          # runAsGroup: 1000

          # Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure
          # that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed.
          # runAsNonRoot: true

          # The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified.
          # runAsUser: 1000

    # ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec.
    # https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod
    # imagePullSecrets:
    # - name: "image-pull-secret"

    # List of initialization containers belonging to the pod.
    #
    # Common initContainers include setting sysctl, or in 7.x versions of Elasticsearch,
    # installing Elasticsearch plugins.
    #
    # https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
#     - command:
#       - sh
#       - "-c"
#       - sysctl -w vm.max_map_count=262144
#       name: sysctl
#       securityContext:
#         privileged: true
    # - command:
    #   - sh
    #   - "-c"
    #   - bin/elasticsearch-plugin remove --purge analysis-icu ; bin/elasticsearch-plugin install --batch analysis-icu
    #   name: install-plugins
    #   securityContext:
    #     privileged: true


    # NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node.
    # https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
    # https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-advanced-node-scheduling.html
    # nodeSelector:
    #   diskType: ssd
    #   environment: production

    # If specified, indicates the pod's priority. "system-node-critical" and "system-cluster-critical" are two special keywords which indicate the highest priorities with the former being the highest priority.
    # Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default.
    # https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/
    # priorityClassName: ""

    # SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field.
    # See previously defined 'securityContext' within 'podTemplate' for all available fields.
    # securityContext: {}

    # ServiceAccountName is the name of the ServiceAccount to use to run this pod.
    # https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
    # serviceAccountName: ""

    # Optional duration in seconds to wait for the Elasticsearch pod to terminate gracefully.
    # terminationGracePeriodSeconds: 30s

    # If specified, the pod's tolerations that will apply to all containers within the pod.
    # https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
    # tolerations:
    # - key: "node-role.kubernetes.io/elasticsearch"
    #   effect: "NoSchedule"
    #   operator: "Exists"

    # TopologySpreadConstraints describes how a group of pods ought to spread across topology domains.
    # Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed.
    #
    # These settings are generally applied within each `nodeSets[].podTemplate` field to apply to a specific Elasticsearch nodeset.
    #
    # https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-advanced-node-scheduling.html
    # topologySpreadConstraints: {}

    # List of volumes that can be mounted by containers belonging to the pod.
    # https://kubernetes.io/docs/concepts/storage/volumes
    # volumes: []

Once I applied this manifest, I got following error on starting of the pod ->

{"@timestamp":"2024-02-29T22:46:26.154Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch-eck-elasticsearch-es-elk-0","elasticsearch.cluster.name":"elasticsearch-eck-elasticsearch","error.type":"java.security.AccessControlException","error.message":"access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")","error.stack_trace":"java.security.AccessControlException: access denied (\"java.lang.RuntimePermission\" \"accessDeclaredMembers\")\n\tat java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488)\n\tat java.base/java.security.AccessController.checkPermission(AccessController.java:1071)\n\tat java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411)\n\tat java.base/java.lang.Class.checkMemberAccess(Class.java:3227)\n\tat java.base/java.lang.Class.getDeclaredConstructors(Class.java:2725)\n\tat com.fasterxml.jackson.databind.util.ClassUtil.getConstructors(ClassUtil.java:1331)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector._findPotentialConstructors(AnnotatedCreatorCollector.java:115)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collect(AnnotatedCreatorCollector.java:70)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedCreatorCollector.collectCreators(AnnotatedCreatorCollector.java:61)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedClass._creators(AnnotatedClass.java:403)\n\tat com.fasterxml.jackson.databind.introspect.AnnotatedClass.getFactoryMethods(AnnotatedClass.java:315)\n\tat com.fasterxml.jackson.databind.introspect.BasicBeanDescription.getFactoryMethods(BasicBeanDescription.java:573)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._addExplicitFactoryCreators(BasicDeserializerFactory.java:641)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory._constructDefaultValueInstantiator(BasicDeserializerFactory.java:278)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findValueInstantiator(BasicDeserializerFactory.java:222)\n\tat com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.createCollectionDeserializer(BasicDeserializerFactory.java:1421)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:403)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:350)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:644)\n\tat com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findNonContextualValueDeserializer(DeserializationContext.java:644)\n\tat com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:539)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:621)\n\tat com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:188)\n\tat com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:28)\n\tat com.fasterxml.jackson.databind.DeserializationContext.handlePrimaryContextualization(DeserializationContext.java:836)\n\tat com.fasterxml.jackson.databind.deser.BeanDeserializerBase.resolve(BeanDeserializerBase.java:550)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:294)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)\n\tat com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)\n\tat com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:654)\n\tat com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4956)\n\tat com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4826)\n\tat com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3809)\n\tat com.amazonaws.partitions.PartitionsLoader.loadPartitionFromStream(PartitionsLoader.java:92)\n\tat com.amazonaws.partitions.PartitionsLoader.build(PartitionsLoader.java:84)\n\tat com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30)\n\tat com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64)\n\tat com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:52)\n\tat com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:106)\n\tat com.amazonaws.client.builder.AwsClientBuilder.getRegionObject(AwsClientBuilder.java:256)\n\tat com.amazonaws.client.builder.AwsClientBuilder.withRegion(AwsClientBuilder.java:245)\n\tat org.elasticsearch.repositories.s3.S3Service$CustomWebIdentityTokenCredentialsProvider.<init>(S3Service.java:373)\n\tat org.elasticsearch.repositories.s3.S3Service.<init>(S3Service.java:98)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.s3Service(S3RepositoryPlugin.java:115)\n\tat org.elasticsearch.repositories.s3.S3RepositoryPlugin.createComponents(S3RepositoryPlugin.java:109)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.node.Node.lambda$new$17(Node.java:759)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.plugins.PluginsService.lambda$flatMap$1(PluginsService.java:263)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)\n\tat java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)\n\tat java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)\n\tat java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)\n\tat java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)\n\tat java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.node.Node.<init>(Node.java:775)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.node.Node.<init>(Node.java:344)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:236)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:236)\n\tat org.elasticsearch.server@8.11.1/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:73)\n"}  
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elasticsearch-eck-elasticsearch.log  
ERROR: Elasticsearch exited unexpectedly, with exit code 1

In the manifest the serviceAccount being used is creted by eck-operartor and I have annotated that with the below given permisison ->

    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": "*"
        }
    ]
}

I have annoated the service accoiuunt with arole thatt has this permiison . The goal is to restore the snapshot from the desired s3 repo all the indices at once.
Please help with this.

[blunckr-aj]

@vatsal-kavida were you able to figure out what was causing that error? I'm encountering something similar

[alex-kattathra-johnson]

I think this might be because the aws sdk v1 (1.12.270) elastic uses doesn't support pod identity.

[alex-kattathra-johnson]

https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html

[DaveCTurner]

Closed by #122431