[MySQL] BinLog Processing Queue backpressure handling #269
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…kage directly Added check for tablemap events
…tener class. Introduced a mechanism to limit the maximum size of the binlog processing queue, thus also limiting memory usage. This maximum processing queue size is configurable
# Conflicts: # modules/module-mysql/package.json
🦋 Changeset detectedLatest commit: cce00b9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 11 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Cleaned up BinLogStream logs a bit
Simplified BinLogListener stopping mechanism
Added a few more defensive stopped checks to the binlog listener
Rentacookie
commented
May 27, 2025
Rentacookie
commented
May 27, 2025
rkistner
reviewed
May 28, 2025
Rentacookie
changed the title
… memory usage rather than number of events
… memory usage rather than number of events. Introduced a maximum timeout that the binlog processing queue can be paused before auto-resuming. This is to prevent the replication connection timing out.
Rentacookie
commented
May 29, 2025
rkistner
previously approved these changes
Jun 2, 2025
rkistner
approved these changes
Jun 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sometimes when a large number of MySQL binlog events were received in a short timeframe, the bucket storage could not persist the events fast enough, causing a backlog in the processing queue. This could lead to out of memory errors because the queue size itself was unbounded.
We now monitor the queue memory footprint, and if it is found to have reached the configured maximum, we pause the binlog listener until all the events currently in the queue have been processed, or after a certain amount of time has passed. Streaming is then immediately resumed.
This effectively limits the number of in flight events that are in memory at any given point when replicating.
To accomplish the above a big chunk of the binlog listening logic was moved to its own class. This cleans up the BinLogStream a bit and made for easier testing.
Additionally, the interim types declared for the Zongji listener were moved to the Zongji repository in this pull request:
powersync-ja/powersync-mysql-zongji#3
The binlog listener config has been updated to include the port, which solves the issue reported here:
#238