Skip to content

Burstable change processing#431

Open
gorbak25 wants to merge 3 commits intomainfrom
gorbak/burstable-change-processing
Open

Burstable change processing#431
gorbak25 wants to merge 3 commits intomainfrom
gorbak/burstable-change-processing

Conversation

@gorbak25
Copy link
Contributor

This PR adjust the amount of changes we try to process at once in real time based on the backlog of changes to sync.
The batch size increases exponentially with the amount of pending changes to process. In the default config:

changes -> batch_size
0-499 -> 100
500-999 -> 500
1000-1999 -> 1000
2000-3999 -> 2000
...
16000-Inf -> 16000

This scaling can be configured via apply_queue_min_batch_size, apply_queue_max_batch_size, apply_queue_step_base in the configuration file.

This PR allows for processing of changes to be started early before accumulating a full batch. The decision for that is made by the apply_queue_batch_threshold_ratio parameter which sets a ratio(0-1) of how full a batch needs to be to start processing it. By default that value is set to 0.9 For ex. with batch size 100 and 90 pending changes when we finish processing a batch we would instantly spawn another batch instead of waiting for changes to accumulate.

The current batch size is available as the corro.agent.changes.batch_size metric.

@gorbak25 gorbak25 requested a review from somtochiama February 27, 2026 02:21
Copy link
Contributor

@somtochiama somtochiama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@gorbak25 gorbak25 force-pushed the gorbak/burstable-change-processing branch from b995ab1 to 094f2cf Compare March 1, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants