Custom middleware to mutate input does not work

[alfaproject]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

I did a custom DynamoDB client middleware to log used capacity units when a specific environment variable is set by mutating the input to the operations to include the ReturnConsumedCapacity when said env variable is configured, but the input that is mutated is ignored completely and only the original command input is serialized in the request

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-dynamodb@3.682.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.11.0

Reproduction Steps

import { ConsumedCapacity, ServiceInputTypes, ServiceOutputTypes } from '@aws-sdk/client-dynamodb';
import { InitializeMiddleware, Pluggable } from '@aws-sdk/types';
import { RegionResolvedConfig } from '@smithy/config-resolver';
import { logInfo } from '@tma/core-log';
import { getEnvNumber } from '@tma/node-env';

function createLogConsumedCapacityMiddleware(
  minConsumedCapacityUnits: number,
): InitializeMiddleware<ServiceInputTypes, ServiceOutputTypes> {
  return next => async args => {
    args.input = {
      ...args.input,
      ReturnConsumedCapacity: 'TOTAL',
    };

    const result = await next(args);

    const output = result.output as { ConsumedCapacity?: ConsumedCapacity | ConsumedCapacity[] };

    const consumedCapacities = output.ConsumedCapacity
      ? Array.isArray(output.ConsumedCapacity)
        ? output.ConsumedCapacity
        : [output.ConsumedCapacity]
      : [];

    if (consumedCapacities.some(consumedCapacity => (consumedCapacity.CapacityUnits ?? 0) >= minConsumedCapacityUnits)) {
      logInfo('ConsumedCapacity', {
        consumedCapacity: output.ConsumedCapacity,
        input: args.input,
      });
    }

    return result;
  };
}

export function createLogConsumedCapacityPlugin(_config: RegionResolvedConfig): Pluggable<ServiceInputTypes, ServiceOutputTypes> {
  return {
    applyToStack: stack => {
      const LOG_CONSUMED_CAPACITY = getEnvNumber('DYNAMODB_LOG_CONSUMED_CAPACITY', 0);
      if (LOG_CONSUMED_CAPACITY) {
        stack.add(createLogConsumedCapacityMiddleware(LOG_CONSUMED_CAPACITY), {
          name: 'logConsumedCapacityMiddleware',
          step: 'initialize',
        });
      }
    },
  };
}

Observed Behavior

ReturnConsumedCapacity is ignored in the new input

Expected Behavior

ReturnConsumedCapacity should not be ignored and the DynamoDB operations should return the CapacityUnits used

Possible Solution

No response

Additional Information/Context

No response

[aBurmeseDev]

Hi @alfaproject - thanks for reaching out.

Could you try this simpler version of code modifying ReturnConsumedCapacity in the middleware? It's working as expected for me.

import { DynamoDB } from "@aws-sdk/client-dynamodb";

const ddb = new DynamoDB();

ddb.middlewareStack.add(
  (next) => (args) => {
    args.input = {
      ...args.input,
      ReturnConsumedCapacity: "TOTAL",
    };

    return next(args);
  },
  {
    name: "logConsumedCapacityMiddleware",
    step: "initialize",
    override: true,
  }
);

await ddb.putItem({
  TableName: "test",
  Item: {
    id: {
      S: "1",
    },
    value: {
      S: "hello",
    },
  },
});

const item = await ddb.getItem({
  TableName: "test",
  Key: {
    id: {
      S: "1",
    },
  },
});

console.log(item);

[alfaproject]

Hello, thank you for your support, but that doesn't work either.

I suspect the problem is from DynamoDBDocumentClient doing something to the middleware stack and getting rid of or overriding the input from my custom middleware, because I'm using this client using DocumentClient like so:

// DocumentClient factory which is the one we're really using in our application
export function dynamoDbDocumentClient() {
  const ddb = new DynamoDBClient();
  ddb.middlewareStack.use(createLogConsumedCapacityPlugin(ddb.config));

  return DynamoDBDocumentClient.from(ddb);
}

[aBurmeseDev]

because I'm using this client using DocumentClient like so:

I may have misunderstood your workflow. In the code provided in the issue description, I saw that @aws-sdk/client-dynamodb was being imported, and it appeared that ReturnConsumedCapacity was not being used. However, now you mentioned @aws-sdk/lib-dynamodb. Could you please clarify which library you're using and provide more context about your setup? I'd be happy to assist further once I have a better understanding.

[alfaproject]

I have pasted the code above in my comment. Is it not clear? I'm adding the middleware to DynamoDBClient but I use that client for DynamoDBDocumentClient because that's how we need to use the DynamoDBDocumentClient, right? I'm guessing that DynamoDBDocumentClient is doing something to the middleware stack of DynamoDBClient?

My middleware has nothing specific to DynamoDBDocumentClient but somehow it's not getting respected when using it that way which is the problem I guess

[alfaproject]

While waiting for your feedback, I have tried to add the middleware to the DynamoDBDocumentClient directly but it doesn't work either:

// DocumentClient factory which is the one we're really using in our application
export function dynamoDbDocumentClient() {
  const ddb = new DynamoDBClient();

  const ddbDocumentClient = DynamoDBDocumentClient.from(ddb);
  ddbDocumentClient.middlewareStack.use(createLogConsumedCapacityPlugin(ddbDocumentClient.config));
  return ddbDocumentClient;
}

ConsumedCapacity is also not returned that way. The middleware code is the same from my first post.

[zshzbh]

Hey @alfaproject,

Sorry for hanging this issue for such a long time. I think I'm finally able to reproduce this issue. I think the root cause of this issue is this part -

      const LOG_CONSUMED_CAPACITY = getEnvNumber('DYNAMODB_LOG_CONSUMED_CAPACITY', 0);
      if (LOG_CONSUMED_CAPACITY) {
        stack.add(createLogConsumedCapacityMiddleware(LOG_CONSUMED_CAPACITY), {
          name: 'logConsumedCapacityMiddleware',
          step: 'initialize',
        });

(As the getEnvNumber function is not available for inspection, I simulated its behavior by manually setting LOG_CONSUMED_CAPACITY to different numbers. . )

1. When LOG_CONSUMED_CAPACITY = 0:

• In JavaScript, 0 is a "falsy" value
• So if(LOG_CONSUMED_CAPACITY) is equivalent to if(0) which evaluates to false

So this code can only add the consumed capacity logging middleware when LOG_CONSUMED_CAPACITY has any non-zero value (truthy value).

2. When LOG_CONSUMED_CAPACITY >0, the code block inside the if statement will execute, and I'm able to see it in both input and output

log -

 Plugin initialized with threshold: 0
Middleware added to stack
Middleware called with input: {
  TableName: 'test',
  Key: { testKey: { S: 'id' } },
  ReturnConsumedCapacity: 'TOTAL'
}
Received result: {
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'xxxx',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  ConsumedCapacity: { CapacityUnits: 0.5, TableName: 'test' }
}
High Capacity Usage Detected: {
  consumedCapacity: { CapacityUnits: 0.5, TableName: 'test' },
  input: {
    TableName: 'test',
    Key: { testKey: [Object] },
    ReturnConsumedCapacity: 'TOTAL'
  },
  threshold: 0
}
Operation completed

Could you please check what is the result of getEnvNumber('DYNAMODB_LOG_CONSUMED_CAPACITY', 0);? And if the result is 0, this verifies my investigation, and you could simply change the if condition to the following to cover the corner case -

  if (LOG_CONSUMED_CAPACITY!=null){
....
}