Skip to content

Conversation

@iuwqyir
Copy link
Contributor

@iuwqyir iuwqyir commented Feb 4, 2025

TL;DR

Migrated database tables from ReplacingMergeTree to VersionedCollapsingMergeTree engine and updated column names for better consistency.

What changed?

  • Replaced lightweight deletes with inserts for blocks, logs, transactions and traces
  • Changed database engine from ReplacingMergeTree to VersionedCollapsingMergeTree
  • Added sign and insert_timestamp columns for versioning support
  • Renamed block table columns for consistency:
    • number -> block_number
    • timestamp -> block_timestamp
  • Updated block timestamp type from UInt64 to DateTime
  • Optimized index granularity settings
  • Adjusted table ordering to use block_timestamp
  • Removed nullable constraints from certain columns
  • Updated corresponding Go structs to match new schema
  • Added option to force consistent data (uses FINAL with clickhouse)

How to test?

  1. Drop existing tables
  2. Run new table creation scripts
  3. Verify data insertion works with new schema
  4. Test data retrieval using updated column names
  5. Verify versioning works by inserting duplicate records
  6. Confirm proper handling of deleted records

Why make this change?

  • Better support for data versioning and deletion tracking
  • Improved data consistency with standardized column naming
  • Enhanced query performance through optimized indexing
  • More efficient handling of data updates and deletions
  • Better type handling for timestamp-related operations

Copy link
Contributor Author

iuwqyir commented Feb 4, 2025

@iuwqyir iuwqyir requested a review from a team February 4, 2025 19:53
@iuwqyir iuwqyir marked this pull request as ready for review February 4, 2025 19:53
@iuwqyir iuwqyir force-pushed the 02-04-use_versionedcollapsingmergetree_and_insert_based_deletes branch 5 times, most recently from c9504f7 to 4d85f84 Compare February 5, 2025 00:38
@iuwqyir iuwqyir force-pushed the 02-04-column_selection_and_api_serialization branch from c9db79f to 4fa7d41 Compare February 6, 2025 12:16
@iuwqyir iuwqyir force-pushed the 02-04-use_versionedcollapsingmergetree_and_insert_based_deletes branch from 4d85f84 to 7b87661 Compare February 6, 2025 12:16
@iuwqyir iuwqyir force-pushed the 02-04-column_selection_and_api_serialization branch from 4fa7d41 to 2733e5c Compare February 6, 2025 12:34
@iuwqyir iuwqyir force-pushed the 02-04-use_versionedcollapsingmergetree_and_insert_based_deletes branch from 7b87661 to 08bc042 Compare February 6, 2025 12:34
Base automatically changed from 02-04-column_selection_and_api_serialization to main February 6, 2025 12:36
…e_and_default_columns_configuration

Chain based table name and default columns configuration
@iuwqyir iuwqyir merged commit 30be13b into main Feb 6, 2025
5 checks passed
@iuwqyir iuwqyir deleted the 02-04-use_versionedcollapsingmergetree_and_insert_based_deletes branch February 6, 2025 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants