`Error: write ECONNABORTED` in node after blocking for long enough

[simonbuchan]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

When using a blocking call for long enough in node like (for windows) execSync("cmd /c timeout 35", { stdio: "inherit" }) seemingly prevents existing socket timeout code from triggering and instead of ECONNRESET you get ECONNABORTED from node and the AWS SDK simply fails immediately without attempting a retry.

Interestingly I can't trigger this with the STS service (when trying to avoid anything environment specific), so it may be specific to the cloudformation server configuration - I haven't tested with other services yet though.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-cloudformation@3.738.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

20.18.0

Reproduction Steps

import cp from "node:child_process";

import { CloudFormation } from "@aws-sdk/client-cloudformation";
import { fromIni } from "@aws-sdk/credential-providers";

const credentials = fromIni({ profile: "test" });
const client = new CloudFormation({ credentials, region: "us-east-1" });
await client.listStacks();
cp.execSync("cmd /c timeout 35", { stdio: "inherit" });
await client.listStacks();

Observed Behavior

node:internal/modules/run_main:129
    triggerUncaughtException(
    ^

Error: write ECONNABORTED
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:95:16)
    at writevGeneric (node:internal/stream_base_commons:139:26)
    at Socket._writeGeneric (node:net:952:11)
    at Socket._writev (node:net:961:8)
    at doWrite (node:internal/streams/writable:594:12)
    at clearBuffer (node:internal/streams/writable:773:5)
    at Writable.uncork (node:internal/streams/writable:529:7)
    at ClientRequest._flushOutput (node:_http_outgoing:1173:10)
    at ClientRequest._flush (node:_http_outgoing:1142:22)
    at onSocketNT (node:_http_client:907:9) {
  errno: -4079,
  code: 'ECONNABORTED',
  syscall: 'write',
  '$metadata': { attempts: 1, totalRetryDelay: 0 }
}

Expected Behavior

SDK should retry the request.

Possible Solution

Hacking in "ECONNABORTED" to NODEJS_TIMEOUT_ERROR_CODES in @smithy/node-http-handler seems to fix it.

Additional Information/Context

No response

[zshzbh]

Hey @simonbuchan ,

I can't reproduce this issue.

The code I have -

import cp from "node:child_process";

import { CloudFormation } from "@aws-sdk/client-cloudformation";
import { fromIni } from "@aws-sdk/credential-providers";

const credentials = fromIni({ profile: "dev" });
const client = new CloudFormation({ credentials, region: "us-east-1" });
await client.listStacks();
cp.execSync("sleep 35", { stdio: "inherit" }); // I'm using mac, here I want reproduce the same socket timeout interference issue using the sleep command
await client.listStacks();

I think the issue comes from node instead of aws js sdk.

I want to double check according to what you posted in this issue -

Hacking in "ECONNABORTED" to NODEJS_TIMEOUT_ERROR_CODES in @smithy/node-http-handler seems to fix it.

Which of these two scenarios is correct? -

1. The error was successfully fixed by this action
2. The JS SDK simply retried the request (as mentioned in the Expected Behavior section where it states 'SDK should retry the request')

[simonbuchan]

It's quite plausible it's only reproducible on Windows, I didn't have a quick way to check, sorry. It's entirely possible it's as simple as the OS timeout being different, but I haven't dug into exactly where the error comes from.

I'm not sure exactly what you mean by the question; that list is the list of errors from node that will get the SDK to retry, so both, really? The error is reported by node to the SDK which then correctly retries, so the caller doesn't see the error, which fixes the error.

[zshzbh]

Hey @simonbuchan ,

I just tried on windows and couldn't reproduce this issue -

import cp from "node:child_process";

import { CloudFormation } from "@aws-sdk/client-cloudformation";
import { fromIni } from "@aws-sdk/credential-providers";

const credentials = fromIni({ profile: "dev" });
const client = new CloudFormation({
  credentials,
  region: "us-east-1",
});
await client.listStacks();
cp.execSync("cmd /c timeout 35", { stdio: "inherit" });
await client.listStacks();
const res = await client.listStacks();
console.log(res);

{
  "dependencies": {
    "@aws-sdk/client-cloudformation": "^3.738.0",
    "@aws-sdk/credential-providers": "^3.743.0"
  }
}

And then the countdown started and no error got reported when then countdown is 0. I also tried to log the response, and the response httpStatusCode is 200

log

PS C:\Users\zshzb\OneDrive\Documents\Projects> node .\index.mjs

Waiting for  0 seconds, press a key to continue ...
{
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'f9eb71b4-a1ca-45be-9990-xxxx',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  StackSummaries: [
    {
      StackId: 'arn:aws:cloudformation:us-east-1:xxxxx:stack/amplify-test-staging-f2e17/xxxx-9e1e-11ef-bc16-xxxx',
      StackName: 'amplify-test-staging-f2e17',
      TemplateDescription: '{"createdOn":"Linux","createdBy":"AmplifyAdmin","createdWith":"12.13.0-rc.xxxxxx.0","stackType":"root","metadata":{"whyContinueWithGen1":"Prefer not to answer"}}',
      CreationTime: 2024-11-08T22:14:03.372Z,
      StackStatus: 'CREATE_COMPLETE',
      DriftInformation: [Object]
    }
  ]
}
PS C:\Users\zshzb\OneDrive\Documents\Projects>

[simonbuchan]

Hmm. Only things not self contained I can think of are the node version, the credentials and maybe the server being regional...

I'll double check when I'm back in the office

[zshzbh]

Sure!
Please check the client region, node version, credential and profile and try again.

If you want to apply custom retry strategy, you can add ECONNABORTED to retryStrategy 's retryDecider.

So then it will look like -

// Custom retry configuration
const clientWithCustomRetry = new CloudFormation({
  region: "us-east-1",
  maxAttempts: 5,
  retryStrategy: {
    maxAttempts: 5,
    retryMode: "adaptive",
    retryDecider: (error) => {
      // ECONNABORTED is retryable by default
      if (error.code === 'ECONNABORTED') {
        return true;
      }
      return false;
    },
    delayDecider: (delayBase, attempts) => {
      return Math.pow(2, attempts) * delayBase;
    }
  }
}); 

Let me know if that fixes the issue.

Thanks! : )

[simonbuchan]

Whelp, this is frustrating... I can't reproduce it now using the example code I posted, even in the package of the project this originally arose in, despite it doing so on thursday both in that script and in a clean project for reproduction - and yet it still reproduces in the actual project. Arrgh.

That retryStrategy workaround seems to be for some other version of the SDK, it doesn't match either the RetryStrategy or RetryStrategyV2 types or their related types, and indeed gives the error TypeError: retryStrategy.retry is not a function at runtime, but it did lead me to @smithy/util-retry, which seems to work:

class HackRetryStrategy extends AdaptiveRetryStrategy {
  override async refreshRetryTokenForRetry(
    tokenToRenew: AWS.StandardRetryToken,
    errorInfo: AWS.RetryErrorInfo,
  ): Promise<AWS.RetryToken> {
    if ((errorInfo.error as NodeJS.ErrnoException).code === "ECONNABORTED") {
      errorInfo = { ...errorInfo, errorType: "TRANSIENT" };
    }
    return await super.refreshRetryTokenForRetry(tokenToRenew, errorInfo);
  }
}

...
new CloudFormation({
  ...
  retryStrategy: new HackRetryStrategy(async () => 5),
})

I'll close this since I can't justify looking further into this, but hopefully at least the workaround helps someone else.