Skip to content

Conversation

@tak-amboss
Copy link
Contributor

Summary

Adds a bulkOperations.singleTransaction config option to handle database transaction limitations when processing large numbers of documents in bulk operations.

Problem

Some databases (e.g., DocumentDB) have transaction limitations when working with large datasets. DocumentDB throws an error when a transaction attempts to use a cursor for more than 100 documents: Feature not supported: cursor within transaction. Try increasing the batchsize.

Solution

  • Adds bulkOperations config with singleTransaction boolean option (default: false)
  • Available as global default in root config and per-collection override
  • When singleTransaction: true, bulk update and delete operations process documents sequentially in separate transactions
  • When singleTransaction: false (default), maintains current behavior with parallel processing in a single transaction
  • Fully backward compatible - no changes to existing behavior unless explicitly configured

Changes

  • Add config types for bulkOperations.singleTransaction (global and collection-level)
  • Modify updateOperation to support per-document transactions
  • Modify deleteOperation to support per-document transactions
  • Add documentation for the new config option
  • Add integration tests for both update and delete operations

Testing

Tested with both singleTransaction: true and false to ensure:

  • Documents are correctly updated/deleted in both modes
  • Sequential processing works without transaction conflicts
  • Parallel processing (default) maintains optimal performance

Add configuration to process bulk operations sequentially in separate transactions instead of all at once in a single transaction. This helps avoid transaction limitations with large datasets (e.g., DocumentDB cursor limits of 100 documents).

- Add bulkOperations.singleTransaction config (default: false)
- Supports global config default and per-collection override
- Update and delete operations process documents sequentially when enabled
- Maintains parallel processing for optimal performance when disabled
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant