Open
Conversation
b995ab1 to
094f2cf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adjust the amount of changes we try to process at once in real time based on the backlog of changes to sync.
The batch size increases exponentially with the amount of pending changes to process. In the default config:
This scaling can be configured via
apply_queue_min_batch_size,apply_queue_max_batch_size,apply_queue_step_basein the configuration file.This PR allows for processing of changes to be started early before accumulating a full batch. The decision for that is made by the
apply_queue_batch_threshold_ratioparameter which sets a ratio(0-1) of how full a batch needs to be to start processing it. By default that value is set to0.9For ex. with batch size 100 and 90 pending changes when we finish processing a batch we would instantly spawn another batch instead of waiting for changes to accumulate.The current batch size is available as the
corro.agent.changes.batch_sizemetric.