Refresh PG statistics after log replay on restart#398
Refresh PG statistics after log replay on restart#398JacksonYao287 wants to merge 2 commits intoeBay:mainfrom
Conversation
Add refresh_pg_statistics() to recalculate and update PG statistics (active_blob_count, tombstone_blob_count, total_occupied_blk_count) after log replay completes. This ensures statistics accuracy after system crashes or restarts. The function is called in on_log_replay_done() before the raft group joins, scanning the index table and chunks to recompute all three statistics from actual data. Statistics are persisted at the next periodic checkpoint.
|
according to the suggestion from @zhiteng , I made this change to force to refresh pg key metrics when restarting |
| de.total_occupied_blk_count.store(total_occupied, std::memory_order_relaxed); | ||
| }); | ||
|
|
||
| LOGI("Refreshed statistics for pg={}: active_blobs={}, tombstone_blobs={}, occupied_blocks={}", pg_id, active_count, |
There was a problem hiding this comment.
better to log out original value as well and a keyword like "corrected" for debugging.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #398 +/- ##
===========================================
- Coverage 63.15% 52.86% -10.30%
===========================================
Files 32 36 +4
Lines 1900 5272 +3372
Branches 204 656 +452
===========================================
+ Hits 1200 2787 +1587
- Misses 600 2194 +1594
- Partials 100 291 +191 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| EXPECT_GT(pg_stats.used_bytes, 0) << "Used bytes should be greater than 0"; | ||
|
|
||
| uint64_t used_bytes_after = pg_stats.used_bytes; | ||
|
|
There was a problem hiding this comment.
better to corrupt one more time here
| auto hs_pg = dynamic_cast< HSHomeObject::HS_PG* >(_obj_inst->_pg_map[pg_id].get()); | ||
| ASSERT_NE(hs_pg, nullptr); | ||
|
|
||
| // Manually corrupt statistics to simulate desync |
There was a problem hiding this comment.
s/desync/inconsistency
|
|
||
| LOGI("Refreshed statistics for pg={}: active_blobs={}, tombstone_blobs={}, occupied_blocks={}", pg_id, active_count, | ||
| tombstone_count, total_occupied); | ||
| LOGI("[corrected] Refreshed statistics for pg={}: active_blobs={} (original={}), tombstone_blobs={} (original={}), " |
There was a problem hiding this comment.
the [corrected] should not be printed if we didnt correct anything
| } else { | ||
| active_count++; | ||
| } | ||
| return false; // Continue scanning |
There was a problem hiding this comment.
It can hit OOM here....
Each blob takes 24B+, assuming we have 10GB mem , it is up to ~400M blobs, with 8KB minimal blob size , it can hit OOM.
Can we cleanup the dummy_output during pagination ?
| LOGINFO("Created shard {}", shard_info.id); | ||
|
|
||
| // Put some blobs to populate statistics using put_blobs | ||
| const uint32_t num_active_blobs = 10; |
There was a problem hiding this comment.
If we have to deal with pagination by ourselves, the number should be large enough to ensure b-tree do have multi pages.
3 participants
Add refresh_pg_statistics() to recalculate and update PG statistics (active_blob_count, tombstone_blob_count, total_occupied_blk_count) after log replay completes. This ensures statistics accuracy after system crashes or restarts.
The function is called in on_log_replay_done() before the raft group joins, scanning the index table and chunks to recompute all three statistics from actual data. Statistics are persisted at the next periodic checkpoint.