-
Notifications
You must be signed in to change notification settings - Fork 13
fix: improve performance of ApplyPowerTableDiffs in ImportSnapshotToDatastore #1047
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
fix: improve performance of ApplyPowerTableDiffs in ImportSnapshotToDatastore #1047
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1047 +/- ##
==========================================
- Coverage 64.77% 64.38% -0.40%
==========================================
Files 81 81
Lines 9896 9942 +46
==========================================
- Hits 6410 6401 -9
- Misses 2961 3005 +44
- Partials 525 536 +11
🚀 New features to boost your workflow:
|
5fbd9b1
to
af0db51
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR optimizes the performance of applying power table diffs during F3 snapshot import by introducing a new map-based approach that avoids expensive sorting operations on each iteration. The optimization shows a ~99% performance improvement when importing mainnet F3 snapshots.
- Introduced
ApplyPowerTableDiffsToMap
function to work with power table maps instead of sorted arrays - Added comprehensive validation for instance numbers, latest instance bounds, and power table CID verification
- Refactored
importSnapshotToDatastoreWithTestingPowerTableFrequency
to use the new map-based approach and include additional safety checks
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
File | Description |
---|---|
certstore/snapshot.go | Refactored snapshot import logic to use map-based power table operations and added validation checks |
certs/certs.go | Added new map-based power table diff functions and helper functions for array-map conversion |
return err | ||
} | ||
|
||
return cs.writeInstanceNumber(ctx, certStoreLatestKey, header.LatestInstance) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function writes header.LatestInstance
but should write the actual latest processed instance. If the loop breaks early due to EOF, this could write an incorrect latest instance number.
return cs.writeInstanceNumber(ctx, certStoreLatestKey, header.LatestInstance) | |
return cs.writeInstanceNumber(ctx, certStoreLatestKey, latestProcessedInstance) |
Copilot uses AI. Check for mistakes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This also seems valid.
Alternative is erroring out if header.Latest
is different from latestCert.GPBFTInstance
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed by returning err when latestCert.GPBFTInstance != header.LatestInstance
if err := cs.putPowerTable(ctx, cert.GPBFTInstance+1, pt); err != nil { | ||
return err | ||
} | ||
} | ||
} | ||
|
||
pt := certs.PowerTableMapToArray(ptm) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Potential nil pointer dereference if no certificates are processed (empty snapshot). latestCert
will be nil if the loop never executes due to immediate EOF.
pt := certs.PowerTableMapToArray(ptm) | |
pt := certs.PowerTableMapToArray(ptm) | |
if latestCert == nil { | |
return ErrUnknownLatestCertificate | |
} |
Copilot uses AI. Check for mistakes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be a valid issue. The suggested resolution could be better.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed by adding nil check
SGTM apart from the two issues highlighted by Copilot |
The
sort
operation inApplyPowerTableDiffs
turns out to be expensive.This PR introduces another
func ApplyPowerTableDiffsToMap(powerTableMap map[gpbft.ActorID]gpbft.PowerEntry, diffs ...PowerTableDiff) (map[gpbft.ActorID]gpbft.PowerEntry, error)
to avoid sorting.It also adds more validations against first instance, latest instance and powertable cid.
Perf stats of importing a mainnet F3 snapshot on my laptop show ~99% perf gain