DynamoDBDocumentClient: QueryCommand fails with ValidationException `operand type is incorrectly inferred as Map (M) `

[tas-oikawa]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

After upgrading @aws-sdk/client-dynamodb and @aws-sdk/lib-dynamodb to version 3.787.0, some of our test cases started throwing the following error:

Caught expected error: Invalid KeyConditionExpression: Incorrect operand type for operator or function; operator or function: begins_with, operand type: M

We did not encounter this issue when using version 3.767.0.

This error occurs when we reuse the same QueryCommand instance for pagination.
Specifically, we execute a query, get the LastEvaluatedKey from the response, assign it to the ExclusiveStartKey of the same QueryCommand instance, and then execute the query again.
At this point, the error occurs.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/package-name@version, ...

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.16.0

Reproduction Steps

I created some test cases to reproduce this issue.

package.json

{
  "name": "marshall_test_repo",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "vitest"
  },
  "private": true,
  "dependencies": {
    "@aws-sdk/client-dynamodb": "3.787.0",
    "@aws-sdk/lib-dynamodb": "3.787.0",
    "dynalite": "^3.2.2",
    "vitest": "^3.1.1"
  },
  "vitest": {
    "testMatch": [
      "**/*.test.ts"
    ]
  }
}

dynamodb.test.ts

import {
  DynamoDBClient,
  CreateTableCommand,
} from "@aws-sdk/client-dynamodb";
import {
  DynamoDBDocumentClient,
  BatchWriteCommand,
  QueryCommand,
} from "@aws-sdk/lib-dynamodb";
import dynalite from "dynalite";
import { beforeAll, afterAll, describe, it, expect } from "vitest";

const dynaliteServer = dynalite({ createTableMs: 0 });
const PORT = 4567;
const TABLE_NAME = "TestTable";

const client = new DynamoDBClient({
  endpoint: `http://localhost:${PORT}`,
  region: "us-east-1",
});
const ddbDocClient = DynamoDBDocumentClient.from(client);

beforeAll(async () => {
  await new Promise((resolve) => dynaliteServer.listen(PORT, resolve));
  await client.send(
    new CreateTableCommand({
      TableName: TABLE_NAME,
      KeySchema: [
        { AttributeName: "PK", KeyType: "HASH" },
        { AttributeName: "SK", KeyType: "RANGE" },
      ],
      AttributeDefinitions: [
        { AttributeName: "PK", AttributeType: "S" },
        { AttributeName: "SK", AttributeType: "S" },
        {
          AttributeName: "Data",
          AttributeType: "S",
        },
      ],
      GlobalSecondaryIndexes: [
        {
          IndexName: "GSI1",
          KeySchema: [
            {
              AttributeName: "SK",
              KeyType: "HASH",
            },
            {
              AttributeName: "Data",
              KeyType: "RANGE",
            },
          ],
          Projection: {
            ProjectionType: "ALL",
          },
        },
      ],
      BillingMode: "PAY_PER_REQUEST",
    })
  );
});

afterAll(async () => {
  await new Promise((resolve) => dynaliteServer.close(resolve));
});

describe("DynamoDB behavior tests with ExclusiveStartKey", () => {
  it("should error when use same query input", async () => {
    const items = [
      {
        PK: "a#123",
        SK: "b#123",
        Data: "c#123",
      },
      {
        PK: "a#987",
        SK: "b#123",
        Data: "c#123",
      },
    ];

    // Insert test data
    await ddbDocClient.send(
      new BatchWriteCommand({
        RequestItems: {
          [TABLE_NAME]: items.map((item) => ({ PutRequest: { Item: item } })),
        },
      })
    );

    const query = new QueryCommand({
      TableName: TABLE_NAME,
      IndexName: "GSI1",
      Select: "ALL_ATTRIBUTES",
      ExpressionAttributeNames: {
        "#GSI1PK": "SK",
        "#GSI1SK": "Data",
      },
      ExpressionAttributeValues: {
        ":GSI1PK": "b#123",
        ":GSI1SK": "c#",
      },
      KeyConditionExpression:
        "#GSI1PK = :GSI1PK and begins_with(#GSI1SK, :GSI1SK)",
      Limit: 1,
    });

    // First query to get LastEvaluatedKey
    const firstQuery = await ddbDocClient.send(
      query
    );

    const lastKey = firstQuery.LastEvaluatedKey;

    query.input.ExclusiveStartKey = lastKey;
    expect(lastKey).toBeDefined();

  
    let errorCaught = false;
    let count = 1;
    try {
      const result = await ddbDocClient.send(
        query
      );
    } catch (err: any) {
      errorCaught = true;
      console.log("Caught expected error:", err.message);
    }

    expect(errorCaught).toBe(false); // We expect this to be false
  });
});

Observed Behavior

Caught expected error: Invalid KeyConditionExpression: Incorrect operand type for operator or function; operator or function: begins_with, operand type: M

Expected Behavior

The test cases pass with no errors.

Possible Solution

No response

Additional Information/Context

No response

[kuhe]

There was a recent change in #6984 that changed how the input is read. This change was for making middleware attached to the DynamoDBDocumentClient behave more intuitively when reading any given request's contextual input.

This is what affected your test between the two versions.

We do not intend for Command inputs to be mutated after instantiation. Although we have set input as readonly, this did not prevent the modification of sub-properties. We should replace the readonly designation with a recursive version to correctly state our intention.

For your code, I suggest using one of the two solutions:

// use a new QueryCommand instance
const query = {
  TableName, IndexName, Select, ...
};
const firstQuery = await ddbDocClient.send(new QueryCommand({ ...query }));
query.ExclusiveStartKey = lastKey;
const secondQuery = await ddbDocClient.send(new QueryCommand({ ...query }));

// use the aggregated client
import { DynamoDBDocument } from "@aws-sdk/lib-dynamodb";

const client = new DynamoDBClient({
  region: "us-west-2",
});
const ddbDocClient = DynamoDBDocument.from(client);

const query = {
  TableName, IndexName, Select, ...
};
const firstQuery = await ddbDocClient.query(query);
query.ExclusiveStartKey = lastKey;
const secondQuery = await ddbDocClient.query(query);

[tas-oikawa]

Thanks a lot for the response and for sharing the workaround!
It fixed the issue in our code.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.