Create storage functions for handling reorgs #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

iuwqyir merged 1 commit into main from 10-04-create_new_storage_functions_for_handling_reorgs

Oct 10, 2024

Contributor

iuwqyir commented Oct 4, 2024 •

edited

Loading

TL;DR

Added support for chain reorganization detection and handling in the storage layer.

What changed?

Introduced a new BlockHeader struct in block.go
Added methods to ClickHouseConnector for managing reorg checks and block headers
Implemented GetLastReorgCheckedBlockNumber and SetLastReorgCheckedBlockNumber in storage connectors
Added LookbackBlockHeaders and DeleteDataForBlocks methods to IMainStorage interface
Updated MemoryConnector to support new reorg-related operations
Added reorg check functionality to RedisConnector
Created a new SQL script for a cursors table in ClickHouse
Modified the staging table creation script to remove the primary key

iuwqyir mentioned this pull request

Refactor rpc for easier reuse #88

Merged

Contributor Author

iuwqyir commented Oct 4, 2024 •

edited

Loading

Test setup and reorg handling tests #95
Refactor RPC client to use interface #94
Add reorg handler functionality and configuration options #91
Create storage functions for handling reorgs #90 👈
Refactor data insertion logic #89
Refactor rpc for easier reuse #88
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @iuwqyir and the rest of your teammates on Graphite

This was referenced Oct 4, 2024

Refactor data insertion logic #89

Merged

Add reorg handler functionality and configuration options #91

Merged

iuwqyir changed the title ~~create new storage functions for handling reorgs~~ Create storage functions for handling reorgs

iuwqyir requested a review from AmineAfia

October 4, 2024 16:40

iuwqyir marked this pull request as ready for review

October 4, 2024 16:40

iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from d0e0f6f to 4773497 Compare

October 4, 2024 16:46

iuwqyir force-pushed the 10-04-create_new_storage_functions_for_handling_reorgs branch from 81f8a60 to 2e3c10b Compare

October 4, 2024 16:46

AmineAfia reviewed

View reviewed changes

internal/tools/clickhouse_create_staging_table.sql Show resolved Hide resolved

AmineAfia reviewed

View reviewed changes

internal/storage/clickhouse.go Show resolved Hide resolved

AmineAfia reviewed

View reviewed changes

internal/storage/clickhouse.go Show resolved Hide resolved

AmineAfia reviewed

View reviewed changes

internal/storage/clickhouse.go Show resolved Hide resolved

iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from 4773497 to d06c5d1 Compare

October 7, 2024 11:04

iuwqyir force-pushed the 10-04-create_new_storage_functions_for_handling_reorgs branch from 2e3c10b to 487685c Compare

October 7, 2024 11:05

AmineAfia reviewed

View reviewed changes

internal/storage/memory.go Outdated Show resolved Hide resolved

AmineAfia reviewed

View reviewed changes

internal/storage/memory.go Show resolved Hide resolved

AmineAfia reviewed

View reviewed changes

internal/storage/memory.go Show resolved Hide resolved

AmineAfia reviewed

View reviewed changes

internal/tools/clickhouse_create_cursors_table.sql Show resolved Hide resolved

iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from d06c5d1 to d0961cb Compare

October 7, 2024 11:44

iuwqyir force-pushed the 10-04-create_new_storage_functions_for_handling_reorgs branch 2 times, most recently from dab5d24 to 56646cf Compare

October 7, 2024 12:12

iuwqyir requested a review from AmineAfia

October 7, 2024 12:17

iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from d0961cb to b11b079 Compare

October 7, 2024 13:19

iuwqyir force-pushed the 10-04-create_new_storage_functions_for_handling_reorgs branch 2 times, most recently from a4e9d3e to b591820 Compare

October 7, 2024 13:22

AmineAfia approved these changes

View reviewed changes

iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from b11b079 to dd391a7 Compare

October 7, 2024 20:01

iuwqyir force-pushed the 10-04-create_new_storage_functions_for_handling_reorgs branch from b591820 to ddcfcbb Compare

October 7, 2024 20:01

iuwqyir mentioned this pull request

Refactor RPC client to use interface #94

Merged

iuwqyir mentioned this pull request

Test setup and reorg handling tests #95

Merged

iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from dd391a7 to ccac684 Compare

October 8, 2024 23:32

iuwqyir force-pushed the 10-04-create_new_storage_functions_for_handling_reorgs branch from ddcfcbb to f4c4f06 Compare

October 8, 2024 23:32

iuwqyir force-pushed the 10-04-change_main_storage_interface_to_handle_inserts_all_at_once branch from ccac684 to 6ead9a7 Compare

October 10, 2024 12:48

iuwqyir force-pushed the 10-04-create_new_storage_functions_for_handling_reorgs branch from f4c4f06 to d96a682 Compare

October 10, 2024 12:48

Contributor Author

iuwqyir commented Oct 10, 2024 •

edited

Loading

Merge activity

Oct 10, 10:01 AM EDT: A user started a stack merge that includes this pull request via Graphite.
Oct 10, 10:06 AM EDT: Graphite rebased this pull request as part of a merge.
Oct 10, 10:07 AM EDT: A user merged this pull request with Graphite.

iuwqyir changed the base branch from 10-04-change_main_storage_interface_to_handle_inserts_all_at_once to graphite-base/90

October 10, 2024 14:02

iuwqyir changed the base branch from graphite-base/90 to main

October 10, 2024 14:04


          create new storage functions for handling reorgs

0f70f2e

iuwqyir force-pushed the 10-04-create_new_storage_functions_for_handling_reorgs branch from d96a682 to 0f70f2e Compare

October 10, 2024 14:05

iuwqyir merged commit 80fcbb5 into main

4 checks passed

iuwqyir deleted the 10-04-create_new_storage_functions_for_handling_reorgs branch

October 10, 2024 14:07

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet