DynamoDB query subsegments show some of them take way too long - How to debug

[s1mrankaur]

I tried using the package to speed up the dynamodb response time when querying time-series data. Nothing fancy, uses a GSI for the query (id and timestamp) and uses them in the key expression. Not much to filter out in the filter expression so the index is optimized.

For some reason, if I use the queryOptimized method, some of the dynamodb query subsegments start taking way too long. In the range of 16seconds-25seconds which is killing the performance and resulting in API gateway timeouts.

The first 20-25 subsegments seem to start at approximately the same time and have decent performance i.e under 500ms, However, after 20-25 subsegments, It starts about 300ms later and the duration of most of these subsegments is really long.

How do I debug what is going on here? Doesn't seem to give any further info in traces. Retries are 0 for all of those subsegments, No failures or errors either. The table is configured to have on-demand capacity so do not imagine capacity to be the issue. Lambda function also has the max (10GB memory allocated) to it. Any ideas on how to figure out what's going on?

[vladholubiev]

See https://medium.com/shelf-io-engineering/how-to-scan-a-23-gb-dynamodb-table-in-1-minute-110730879e2b

[vladholubiev]

My article is about scan. But the same problem occurs for queries

[s1mrankaur]

@vladgolubev Thanks for your response. My hash key on the index is as follows:

PK: customerId#locationId
SK: timestamp

customerId is always the same in a given query while locationIds are different. Some locations may have more streamed time-series data than others for sure. QueryRegular seems to take only 10 seconds while queryOptimized times out at 29 seconds. Traces show subsegments od dynamoDB query taking 10-25 seconds. I was expecting queryOptimized to 2x the performance but it got multiple times slower.