403 Forbidden Error - Validate IMDSv2 Support in OpenSearch Connector for Apache Kafka 3.1.1

[pdp-jsanabria]

Hello,

I am using OpenSearch Connector for Apache Kafka 3.1.1 to send logs to Amazon OpenSearch Service with IAM authentication. However, I am encountering a 403 Forbidden error when the connector tries to verify the existence of an index in OpenSearch.

Issue Details:

• The connector is running on an EC2 instance with an assigned IAM role.
• The IAM role has the correct permissions (es:DescribeIndex, es:ListIndices, es:CreateIndex, es:Write, es:Read), and security group rules allow HTTPS (port 443) access.
• The EC2 instance is in a private subnet, and AWS IMDSv2 is enabled.
• A manual curl request using an IMDSv2 token successfully retrieves IAM credentials.
• The connector fails to authenticate and logs the following error:

{
  "name": "connector-name",
  "connector": {
    "state": "UNASSIGNED",
    "worker_id": "XX.XX.XX.XX:8083"
  },
  "tasks": [
    {
      "id": 0,
      "state": "FAILED",
      "worker_id": "XX.XX.XX.XX:8083",
      "trace": "org.apache.kafka.connect.errors.ConnectException: Exiting WorkerSinkTask due to unrecoverable exception.\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:636)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:345)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:247)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:216)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:226)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:281)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:238)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: org.apache.kafka.connect.errors.ConnectException: Failed to check index logs-all-apps exists after total of 6 attempt(s)\n\tat io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:144)\n\tat io.aiven.kafka.connect.opensearch.RetryUtil.callWithRetry(RetryUtil.java:105)\n\tat io.aiven.kafka.connect.opensearch.OpensearchClient.withRetry(OpensearchClient.java:339)\n\tat io.aiven.kafka.connect.opensearch.OpensearchClient.indexOrDataStreamExists(OpensearchClient.java:133)\n\tat io.aiven.kafka.connect.opensearch.OpensearchSinkTask.ensureIndexOrDataStreamExists(OpensearchSinkTask.java:156)\n\tat io.aiven.kafka.connect.opensearch.OpensearchSinkTask.tryWriteRecord(OpensearchSinkTask.java:132)\n\tat io.aiven.kafka.connect.opensearch.OpensearchSinkTask.put(OpensearchSinkTask.java:122)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:606)\n\t... 11 more\nCaused by: OpenSearchStatusException[method [HEAD], host [https://host_opensearch.us-east-x.es.amazonaws.com], URI [/topic_name?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 **403 Forbidden**]]; nested: ResponseException[method [HEAD], host [https://host_opensearch.us-east-x.es.amazonaws.com], URI [/topic_name?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 403 Forbidden]];"
    }
  ]
}

Questions & Requests:

1. Does OpenSearch Connector for Apache Kafka 3.1.1 fully support AWS IMDSv2 for IAM authentication?
2. Is there any additional configuration needed in the connector to properly use IMDSv2 when retrieving AWS credentials?
3. Could this issue be related to how the connector signs requests using AWS SigV4 when IMDSv2 is enabled?
   Any guidance on this issue would be greatly appreciated. Thank you!