Skip to content

Conversation

@iuwqyir
Copy link
Contributor

@iuwqyir iuwqyir commented May 14, 2025

TL;DR

Added error handling for block data insertion that checks if data was actually inserted despite reported errors.

What changed?

  • Added a new getMaxBlockNumberConsistent method that retrieves the maximum block number with sequential consistency guarantees
  • Enhanced the InsertBlockData error handling to verify if data was actually inserted despite reported errors
  • When an error occurs during batch insertion, the code now:
    1. Determines the highest block number in the batch
    2. Waits briefly (500ms)
    3. Checks if the highest block from the batch matches the max consistent block in the database
    4. Only returns an error if the verification fails, otherwise logs the error but continues execution

How to test?

  1. Trigger a scenario where ClickHouse reports an error during batch insertion but actually inserts the data
  2. Verify that the process continues execution rather than failing
  3. Check logs for the message "Failure while inserting block data, but insert still succeeded"

Why make this change?

ClickHouse can sometimes report errors during batch insertions even when the data was successfully inserted in the background. This change prevents unnecessary failures by verifying the actual state of the database before deciding whether to propagate the error, improving the reliability of the indexer.

Copy link
Contributor Author

iuwqyir commented May 14, 2025

This stack of pull requests is managed by Graphite. Learn more about stacking.

@iuwqyir iuwqyir requested review from AmineAfia and catalyst17 May 14, 2025 12:46
@iuwqyir iuwqyir changed the title handle clickhouse errors during inserts handle false positive clickhouse errors during inserts May 14, 2025
@iuwqyir iuwqyir marked this pull request as ready for review May 14, 2025 12:47
@iuwqyir iuwqyir force-pushed the 05-14-handle_clickhouse_errors_during_inserts branch from 7ef6ba3 to 17e2e67 Compare May 14, 2025 12:50
@iuwqyir iuwqyir merged commit 90e34dd into main May 15, 2025
5 checks passed
@iuwqyir iuwqyir deleted the 05-14-handle_clickhouse_errors_during_inserts branch May 15, 2025 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants