Skip to content

feat: add timestamp-based CDC resume for MongoDB#294

Open
vaibhav-datazip wants to merge 1 commit intodatazip-inc:stagingfrom
vaibhav-datazip:stategen
Open

feat: add timestamp-based CDC resume for MongoDB#294
vaibhav-datazip wants to merge 1 commit intodatazip-inc:stagingfrom
vaibhav-datazip:stategen

Conversation

@vaibhav-datazip
Copy link
Collaborator

@vaibhav-datazip vaibhav-datazip commented May 22, 2025

Description

This PR adds the ability to resume CDC (Change Data Capture) operations from a specific timestamp in MongoDB. This is particularly useful when:

  • You need to start CDC from a known point in time
  • You want to skip the initial backfill process
  • You need to recover from a specific point after a failure or deletion of state file

Type of change

  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • Tested with MongoDB Atlas:

    • CDC sync with fresh state (no state file)
    • CDC sync with existing state file
    • CDC sync with timestamp flag and fresh state
    • CDC sync with timestamp flag and existing state
  • Tested with Local Replica Set:

    • CDC sync with fresh state (no state file)
    • CDC sync with existing state file
    • CDC sync with timestamp flag and fresh state
    • CDC sync with timestamp flag and existing state
  • Additional Test Cases:

    • Verified that backfill is skipped when using timestamp
    • Verified proper cursor handling
    • Verified that no resume tokens are skipped during CDC sync

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants