SDK attempts to parse a plain text AWS API response as JSON/XML throwing a misleading deserialization error

[hs-kgoriunov]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

There is an edge case when AWS API can respond in a plain text, regardless of the request's accept header. The SDK then attempts to parse the response as JSON or as XML, depending on what it expects, and fails with a misleading deserialization error.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-sns@3.840.0, @aws-sdk/client-sqs@3.840.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.17.0

Reproduction Steps

Example sending a large message to SNS, let's say it became this large due to a bug in a production software

import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";

const snsClient = new SNSClient({
    region: "us-west-2",
    // the example works with no credentials
    credentials: async () => ({
        accessKeyId: "dummy",
        secretAccessKey: "dummy",
    })
});
try {
    await snsClient.send(
        new PublishCommand({
            TopicArn: "arn:aws:sns:us-west-2:000000000000:topic",
            Message: "a".repeat(100 * 256 * 1024),
        }),
    );
} catch (error) {
    console.log(error);
    console.log("$response.headers.content-type:", error.$response?.headers?.["content-type"]);
    console.log("$response.statusCode:", error.$response?.statusCode);
    console.log("$response.reason:", error.$response?.reason);
    console.log("$response.body:", error.$response?.body);
}

A similar example sending a large SQS message

import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";

const sqsClient = new SQSClient({
    region: "us-west-2",
    // the example works with no credentials
    credentials: async () => ({
        accessKeyId: "dummy",
        secretAccessKey: "dummy",
    })
});
try {
    await sqsClient.send(
        new SendMessageCommand({
            QueueUrl: "https://sqs.us-west-2.amazonaws.com/000000000000/queue",
            MessageBody: "a".repeat(10 * 256 * 1024),
        }),
    );
} catch (error) {
    console.log(error);
    console.log("$response.headers.content-type:", error.$response?.headers?.["content-type"]);
    console.log("$response.statusCode:", error.$response?.statusCode);
    console.log("$response.reason:", error.$response?.reason);
    console.log("$response.body:", error.$response?.body);
}

Observed Behavior

SQS example outputs

Error: char 'H' is not expected.:1:1
  Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.
    at XMLParser.parse (/--redacted--/node_modules/fast-xml-parser/src/xmlparser/XMLParser.js:30:21)
    at /--redacted--/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1238:26
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async parseXmlErrorBody (/--redacted--/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1259:17)
    at async de_CommandError (/--redacted--/node_modules/@aws-sdk/client-sns/dist-cjs/index.js:1940:11)
    at async /--redacted--/node_modules/@smithy/middleware-serde/dist-cjs/index.js:36:20
    at async /--redacted--/node_modules/@smithy/core/dist-cjs/index.js:193:18
    at async /--redacted--/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /--redacted--/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
    at async file:///--redacted--/index.mjs:12:5 {
  '$metadata': {
    httpStatusCode: 413,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}
$response.headers.content-type: text; charset=utf-8
$response.statusCode: 413
$response.reason: Request Entity Too Large
$response.body: HTTP content length exceeded 3145728 bytes.

SNS example outputs

SyntaxError: Unexpected token 'H', "HTTP conte"... is not valid JSON
  Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.
    at JSON.parse (<anonymous>)
    at /--redacted--/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:154:19
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async parseJsonErrorBody (/--redacted--/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:167:17)
    at async de_CommandError (/--redacted--/node_modules/@aws-sdk/client-sqs/dist-cjs/index.js:1218:11)
    at async /--redacted--/node_modules/@smithy/middleware-serde/dist-cjs/index.js:36:20
    at async /--redacted--/node_modules/@smithy/core/dist-cjs/index.js:193:18
    at async /--redacted--/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /--redacted--/node_modules/@aws-sdk/middleware-sdk-sqs/dist-cjs/index.js:132:16
    at async /--redacted--/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22 {
  '$metadata': {
    httpStatusCode: 413,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}
$response.headers.content-type: text; charset=utf-8
$response.statusCode: 413
$response.reason: Request Entity Too Large
$response.body: HTTP content length exceeded 1662976 bytes.

Expected Behavior

I'd love to see an SDK error that conveys the actual API error being "413 Request Entity Too Large", not some misleading syntax/deserialization errors coming from within the SDK itself. Having a hint guiding to inspect the hidden $response field might be good for development environments, but bad for production environments where neither logs nor error reporting does not capture non-serializable fields. Furthermore, It is funny that before 3.723.0 there were no httpStatusCode in the error's $metadata field, making it impossible to figure out the root cause without updating production code to capture the hidden $response field and reproducing an issue.

Possible Solution

If the API response content-type is text; charset=utf-8, then maybe the SDK should not attempt parsing it, clearly stating in the error message that the response came out to have an unsupported/unexpected content-type. Ideally, the error message should also include the response status code and status text, instead of hiding it under the non-enumerable $response field.

Additional Information/Context

No response

[aBurmeseDev]

Hi @hs-kgoriunov - thanks for checking in.

The SDK is designed to work with the format that the service API returns through their defined models. We handle it this way to avoid accidentally logging sensitive information. For cases like these, we recommend users opt into printing $response. Hope that makes sense!

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.