PS-10245 feature: Implement receiving binlog events in GTID mode (part 4)#85
Merged
percona-ysorokin merged 1 commit intomainfrom Jan 12, 2026
Merged
Conversation
…t 4) https://perconadev.atlassian.net/browse/PS-10245 Implemented more sophisticated event data buffering and flushing at the 'binstv::storage' level. We now accumulate data from several events in the 'binstv::storage'internal memory buffer (16K initially but can grow) and perform actual writing only at transaction boundaries. This helps a lot with making sure that after a crash we can continue operations from a position that corresponds to a committed transaction. This is vitally important for GTID-based replication. 'binstv::storage::write_event()' method extended with an extra parameter that indicates whether the event data being written corresponds to the end of a transaction. Reworked 'binsrv::basic_storage_engine' interface: removed 'flush_stream()' method and all 'write_data_to_stream()' method overloads are now expected to immediately flush data. 'binsrv::filesystem_storage_backend' and 'binsrv::s3_storage_backend' concrete implementations updated to accommodate these changes. We also added "rdbuf()->pubsetbuf(nullptr, 0U)" calls before `std::fstream` read/writes inside 'binsrv::filesystem_storage_backend' to disable internal buffering. Fixed an issue with unnecessary data uploading upon stream closing / object destruction inside 'binsrv::s3_storage_backend'. 'binlog_streaming.binsrv' MTR test case adapted for new buffering behavior. Fixed problem with the 'checkpointing_both' combination in the 'binlog_streaming.checkpointing' MTR test case when checkpointing parameters were not set properly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://perconadev.atlassian.net/browse/PS-10245
Implemented more sophisticated event data buffering and flushing at the 'binstv::storage' level. We now accumulate data from several events in the 'binstv::storage'internal memory buffer (16K initially but can grow) and perform actual writing only at transaction boundaries. This helps a lot with making sure that after a crash we can continue operations from a position that corresponds to a committed transaction. This is vitally important for GTID-based replication. 'binstv::storage::write_event()' method extended with an extra parameter that indicates whether the event data being written corresponds to the end of a transaction.
Reworked 'binsrv::basic_storage_engine' interface: removed 'flush_stream()' method and all 'write_data_to_stream()' method overloads are now expected to immediately flush data. 'binsrv::filesystem_storage_backend' and 'binsrv::s3_storage_backend' concrete implementations updated to accommodate these changes.
We also added "rdbuf()->pubsetbuf(nullptr, 0U)" calls before
std::fstreamread/writes inside 'binsrv::filesystem_storage_backend' to disable internal buffering.Fixed an issue with unnecessary data uploading upon stream closing / object destruction inside 'binsrv::s3_storage_backend'.
'binlog_streaming.binsrv' MTR test case adapted for new buffering behavior.
Fixed problem with the 'checkpointing_both' combination in the 'binlog_streaming.checkpointing' MTR test case when checkpointing parameters were not set properly.