Skip to content

DynamoDB writer should limit write batches to 25 items #992

@Archanian

Description

@Archanian

DynamoDB batch writes are limited to 25 items, as per the documentation here. The benthos dynamo writer currently falls back to one-by-one writes in the case of any errors (such as exceeding this 25 item limit), which has a huge impact on throughput in the case of large output batch sizes.

It would be nice if the dynamo writer would internally limit each batch call to 25 items, making this invisible to the user and avoiding the need to restrict output batch settings to 25 or fewer in benthos config.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bughancementIt's kind of a bug, but mostly an enhancement.outputsAny tasks or issues relating specifically to outputsux

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions