SecretManager | List secrets on `ap-northeast-1`: connect ETIMEDOUT/ENETUNREACH

[loganmzz]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

I have written a tool trying to resolve existing Terraform resource.

Everything is working fine with many services (Secret Manager, Security Group, EventBridge) and two regions (eu-central-1 and ap-northeast-1) but listing secrets on ap-northeast-1.

const sm = new SecretsManager({region: 'ap-northeast-1'});
const listSecrets = await sm.listSecrets();

It results in:

AggregateError
        at internalConnectMultiple (node:net:1114:18)
        at internalConnectMultiple (node:net:1177:5)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      code: 'ETIMEDOUT',
      name: 'TimeoutError',
      '$metadata': { attempts: 3, totalRetryDelay: 209 },
      [errors]: [
        Error: connect ETIMEDOUT 18.179.209.155:443
            at createConnectionError (node:net:1634:14)
            at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'connect',
          address: '18.179.209.155',
          port: 443
        },
        Error: connect ENETUNREACH 2406:da14:afa:8a00:e3de:c82b:7b57:b9a4:443 - Local (:::0)
            at internalConnectMultiple (node:net:1176:40)
            at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -101,
          code: 'ENETUNREACH',
          syscall: 'connect',
          address: '2406:da14:afa:8a00:e3de:c82b:7b57:b9a4',
          port: 443
        },
        Error: connect ETIMEDOUT 18.179.211.244:443
            at createConnectionError (node:net:1634:14)
            at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'connect',
          address: '18.179.211.244',
          port: 443
        },
        Error: connect ENETUNREACH 2406:da14:afa:8a01:bfa8:ff5e:a40a:74af:443 - Local (:::0)
            at internalConnectMultiple (node:net:1176:40)
            at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -101,
          code: 'ENETUNREACH',
          syscall: 'connect',
          address: '2406:da14:afa:8a01:bfa8:ff5e:a40a:74af',
          port: 443
        },
        Error: connect ETIMEDOUT 52.198.52.244:443
            at createConnectionError (node:net:1634:14)
            at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'connect',
          address: '52.198.52.244',
          port: 443
        },
        Error: connect ENETUNREACH 2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13:443 - Local (:::0)
            at internalConnectMultiple (node:net:1176:40)
            at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -101,
          code: 'ENETUNREACH',
          syscall: 'connect',
          address: '2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13',
          port: 443
        }
      ]
    }

Tried with AWS CLI I have no issue:

AWS_REGION=ap-northeast-1 aws secretsmanager list-secrets --filters 'Key=name,Values=...'

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-secrets-manager@3.687.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.10.0

Reproduction Steps

const sm = new SecretsManager({region: 'ap-northeast-1'});
const listSecrets = await sm.listSecrets();

Observed Behavior

Error:

AggregateError
    at internalConnectMultiple (node:net:1114:18)
    at internalConnectMultiple (node:net:1177:5)
    at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7) {
  code: 'ETIMEDOUT',
  name: 'TimeoutError',
  '$metadata': { attempts: 3, totalRetryDelay: 157 },
  [errors]: [
    Error: connect ETIMEDOUT 18.179.209.155:443
        at createConnectionError (node:net:1634:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '18.179.209.155',
      port: 443
    },
    Error: connect ENETUNREACH 2406:da14:afa:8a00:e3de:c82b:7b57:b9a4:443 - Local (:::0)
        at internalConnectMultiple (node:net:1176:40)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2406:da14:afa:8a00:e3de:c82b:7b57:b9a4',
      port: 443
    },
    Error: connect ETIMEDOUT 18.179.211.244:443
        at createConnectionError (node:net:1634:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '18.179.211.244',
      port: 443
    },
    Error: connect ENETUNREACH 2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13:443 - Local (:::0)
        at internalConnectMultiple (node:net:1176:40)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13',
      port: 443
    },
    Error: connect ETIMEDOUT 52.198.52.244:443
        at createConnectionError (node:net:1634:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '52.198.52.244',
      port: 443
    },
    Error: connect ENETUNREACH 2406:da14:afa:8a01:bfa8:ff5e:a40a:74af:443 - Local (:::0)
        at internalConnectMultiple (node:net:1176:40)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2406:da14:afa:8a01:bfa8:ff5e:a40a:74af',
      port: 443
    }
  ]
}

Expected Behavior

Valid HTTP response

Possible Solution

N/A

Additional Information/Context

No response

[zshzbh]

Hey @loganmzz ,

Thanks for the feedback! However I can't reproduce this issue.

This is the code I have -

import { SecretsManager } from '@aws-sdk/client-secrets-manager';

const client = new SecretsManager({region: 'ap-northeast-1'});

const response = await client.listSecrets();
console.log(response)

And the result is 200.

{
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'f40c7c36-63d2-459b-a6e3-XXX',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  SecretList: [
    {
      ARN: 'arn:aws:secretsmanager:ap-northeast-1:XXX:secret:prod/test/secret-XXX',
      CreatedDate: 2024-12-04T23:07:09.893Z,
      LastChangedDate: 2024-12-04T23:07:10.145Z,
      Name: 'prod/test/secret',
      SecretVersionsToStages: [Object],
      Tags: []
    }
  ]
}

This error indicates a network timeout when trying to connect to AWS Secrets Manager. You can manually setup timeout and retry attempts -

const client = new SecretsManager({
    region: 'ap-northeast-1',
    logger: console,
    maxAttempts: 3,
    retryMode: 'standard',
    requestTimeout: 3000,
    connectTimeout: 3000
});

Or if you think I miss anything, please add additional info.

Thanks!

[loganmzz]

Looks like timeouts are ignored. I increased them to 50000 and I still facing same issue in only 3s.

I run it several times (~ 10), and it passed once... May be just need to "really" increase timeout :( Any other way to increase them?

[loganmzz]

Just to be sure the information is shared, I have no issue with CLI (Python?)

[loganmzz]

@zshzbh At stated previously your setup doesn't seem to affect timeout. Any other suggestion on how to setup custom timeouts?

[zshzbh]

There's some steps I recommend -

1. Checking secrets in ap-northeast-1:
   I don't actually have access to check AWS secrets or configurations. You'll need to verify this in your own AWS account using the AWS CLI or console.

2. Terraform and network connectivity:
   You're right that Terraform is a third-party tool. When using it with AWS, it's a good practice to check network connectivity. You can do this by:

   • Ensuring your local machine can reach AWS endpoints
   • Verifying your AWS credentials are correctly configured
   • Running a simple AWS CLI command to test connectivity

3. Adding middleware to the code:
   The middleware you provided looks good. It will log the context, input, and output of AWS SDK calls. Here's how you can add it to your AWS SDK client:

import { S3Client } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "ap-northeast-1" });

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

4. Setting up higher max attempts:
   You can increase the maximum number of retry attempts by configuring the retry strategy. Here's an example:

import { S3Client } from "@aws-sdk/client-s3";
import { NodeHttpHandler } from "@aws-sdk/node-http-handler";
import { RetryStrategy } from "@aws-sdk/util-retry";

const client = new S3Client({
  region: "ap-northeast-1",
  maxAttempts: 5,  // Increase this number as needed
  requestHandler: new NodeHttpHandler({
    connectionTimeout: 5000,  // 5 seconds
    socketTimeout: 5000,  // 5 seconds
  }),
  retryStrategy: new RetryStrategy(() => Promise.resolve(3000)),  // 3 seconds delay between retries
});

5. Sharing reproducible code:
   If the above steps don't resolve the issue, please provide:
   • The complete Terraform configuration file
   • Any relevant AWS SDK code you're using
   • Full error messages and stack traces
   • The exact CLI commands you're running
   • Any environment variables or AWS credentials setup

[loganmzz]

There's some steps I recommend -

1. Checking secrets in ap-northeast-1:
   I don't actually have access to check AWS secrets or configurations. You'll need to verify this in your own AWS account using the AWS CLI or console.

No issue here. Everything is working fine with other clients (CLI, Terraform) and other services.

2. Terraform and network connectivity:
   You're right that Terraform is a third-party tool. When using it with AWS, it's a good practice to check network connectivity. You can do this by:

   • Ensuring your local machine can reach AWS endpoints
   • Verifying your AWS credentials are correctly configured
   • Running a simple AWS CLI command to test connectivity

As stated it was already validated.

3. Adding middleware to the code:
   The middleware you provided looks good. It will log the context, input, and output of AWS SDK calls. Here's how you can add it to your AWS SDK client:

import { S3Client } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "ap-northeast-1" });

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

Strange... As long as I run tests without middleware I had the issue. After adding it, I had no more timeout issue... I removed middleware it still works, resetting networking connection it still works. I will try after rebooting later on.

4. Setting up higher max attempts:
   You can increase the maximum number of retry attempts by configuring the retry strategy. Here's an example:

import { S3Client } from "@aws-sdk/client-s3";
import { NodeHttpHandler } from "@aws-sdk/node-http-handler";
import { RetryStrategy } from "@aws-sdk/util-retry";

const client = new S3Client({
  region: "ap-northeast-1",
  maxAttempts: 5,  // Increase this number as needed
  requestHandler: new NodeHttpHandler({
    connectionTimeout: 5000,  // 5 seconds
    socketTimeout: 5000,  // 5 seconds
  }),
  retryStrategy: new RetryStrategy(() => Promise.resolve(3000)),  // 3 seconds delay between retries
});

I will give it a try if I can reproduce error.

5. Sharing reproducible code:
   If the above steps don't resolve the issue, please provide:

   • The complete Terraform configuration file
   • Any relevant AWS SDK code you're using
   • Full error messages and stack traces
   • The exact CLI commands you're running
   • Any environment variables or AWS credentials setup

Here my test script:

const client = require('@aws-sdk/client-secrets-manager');

async function main() {
  const sm = new client.SecretsManager({
    region: 'ap-northeast-1',
    /*
    logger: console,
    maxAttempts: 3,
    retryMode: 'standard',
    requestTimeout: 100000,
    connectTimeout: 100000,
    */
  });

  // sm.middlewareStack.add(
  //   (next, context) => async (args) => {
  //     console.log("AWS SDK context", context.clientName, context.commandName);
  //     console.log("AWS SDK request input", args.input);
  //     const result = await next(args);
  //     console.log("AWS SDK request output:", result.output);
  //     return result;
  //   },
  //   {
  //     name: "MyMiddleware",
  //     step: "build",
  //     override: true,
  //   }
  // );

  const listSecrets = await sm.listSecrets();
  console.log(JSON.stringify(listSecrets.SecretList, undefined, 2));
}

main();