Add missing block polling in committer

[iuwqyir]

TL;DR

Added handling for missing staging data in the Committer and improved error handling.

What changed?

• Implemented handleMissingStagingData function in the Committer to detect and handle missing blocks in staging data.
• Updated getSequentialBlockDataToCommit to call handleMissingStagingData when no blocks are found in staging.
• Added new tests to cover the missing staging data handling functionality.
• Removed unnecessary assertion calls in test files.
• Removed a debug log message in the GetMaxBlockNumber function of ClickHouseConnector.

How to test?

1. Run the updated unit tests, particularly focusing on TestHandleMissingStagingData and TestHandleMissingStagingDataIsPolledWithCorrectBatchSize.
2. Simulate a scenario where staging data is missing and verify that the Committer correctly detects and handles the situation.
3. Check that the Committer polls for missing blocks with the correct batch size when staging data is missing.

Why make this change?

This change improves the robustness of the Committer by adding a mechanism to handle cases where expected staging data is missing. It ensures that the system can recover from potential data gaps and maintain data integrity. The additional tests provide better coverage for this new functionality, increasing overall system reliability.

[iuwqyir]

• Add missing block polling in committer #114 👈
• Poll missing blocks in gap handler #113
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @iuwqyir and the rest of your teammates on Graphite

[iuwqyir]

Merge activity

• Oct 25, 9:17 AM EDT: A user started a stack merge that includes this pull request via Graphite.
• Oct 25, 9:19 AM EDT: Graphite rebased this pull request as part of a merge.
• Oct 25, 9:20 AM EDT: A user merged this pull request with Graphite.