Skip to content

Commit 3c6e28d

Browse files
PS-10245 feature: Implement receiving binlog events in GTID mode (part 5) (#86)
https://perconadev.atlassian.net/browse/PS-10245 GTID-based replication mode is now supported. Along with writing data to binlog files 'binsrv::storage' class now also creates and maintains binlog file metadata - separate files that follows the '<binlog_file_name>.json' naming convention and store actual flushed file size (in both position-based and GTID-based replication modes) and accumulated GTID set (only in GTID-based replication mode). In addition, upon storage initialization, we now restore the GTID set from the latest binlog metadata file in order to continue GTID-based replication from this set. Implemented additional logic that discards events from the latest incomplete transaction from the storage buffer upon MySQL server disconnect. Main application extended with additional diagnostic messages informing about binlog files creation / opening / closing / reusing. In GTID mode 'binsrv::storage' class now also keeps track of the GTIDs associated with processed events that helps with updating binlog metadata files. 'binsrv::storage' constructor now performs additional checks to ensure that each file in the binlog index has a corresponding binlog file metadata ('<binlog_file_name>.json') and the size from that metadata file matches the actual binlog file size. 'binsrv::basic_storage_backend::open_stream()' method now returns the size of the opened file or 0 if the file was reated. Concreate implementations ('binsrv::filesystem_storage_backend' and 'binsrv::s3_storage_backend') updated correspondingly. 'binsrv::storage::open_binlog()' method now returns extra info about file creating / opening: - created - opened_empty - opened_at_magic_paylod_offset - opened_with_data_present This helps with making a decision which events 'binsrv::event::reader_context' should expect next. 'binsrv::event::reader_context' extended with additional logic that helps to determine if the processed event is "info-only" and should not be written to the binlog files (this concerns artificial RORATE, FORMAT_DESCRIPTION and PREVIOUS_GTID_LOG events). Improved 'util::nv_tuple_to_json()' function - we now create keys for optional objects only when they are non-empty. Added new 'binlog_streaming.resume_streaming' MTR test case that checks various continuation scenarios bot the Binlog Server utility (starting from an empty storage, starting from a file that has just magic payload in it, receiving ROTATE as the last event, etc.). The testcase has 4 combinations for position-based / GTID-based replication in buffered / unbuffered (immediately flushing) mode. It is now possible to specify to set '$binsrv_replication_mode' (to either 'position' or 'gtid') before including the 'set_up_binsrv_environment.inc' MTR include file to set desired Binlog Server utility replication mode.
1 parent 5f7b48b commit 3c6e28d

28 files changed

Lines changed: 1016 additions & 157 deletions

CMakeLists.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -224,6 +224,10 @@ set(source_files
224224
src/binsrv/basic_storage_backend.hpp
225225
src/binsrv/basic_storage_backend.cpp
226226

227+
src/binsrv/binlog_file_metadata_fwd.hpp
228+
src/binsrv/binlog_file_metadata.hpp
229+
src/binsrv/binlog_file_metadata.cpp
230+
227231
src/binsrv/cout_logger.hpp
228232
src/binsrv/cout_logger.cpp
229233

mtr/binlog_streaming/include/set_up_binsrv_environment.inc

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
# --let $binsrv_tls_version = TLSv1.2 (optional)
1515
# --let $binsrv_idle_time = 10
1616
# --let $binsrv_verify_checksum = TRUE | FALSE
17+
# --let $binsrv_replication_mode = position | gtid
1718
# --let $binsrv_checkpoint_size = 2M (optional)
1819
# --let $binsrv_checkpoint_interval = 30s (optional)
1920
# --source set_up_binsrv_environment.inc
@@ -82,7 +83,7 @@ eval SET @binsrv_config_json = JSON_OBJECT(
8283
'server_id', @@server_id + 1,
8384
'idle_time', $binsrv_idle_time,
8485
'verify_checksum', $binsrv_verify_checksum,
85-
'mode', 'position'
86+
'mode', '$binsrv_replication_mode'
8687
),
8788
'storage', JSON_OBJECT(
8889
'backend', '$storage_backend',
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
*** Resetting replication at the very beginning of the test.
2+
3+
*** Generating a configuration file in JSON format for the Binlog
4+
*** Server utility.
5+
6+
*** Determining binlog file directory from the server.
7+
8+
*** Creating a temporary directory <BINSRV_STORAGE_PATH> for storing
9+
*** binlog files downloaded via the Binlog Server utility.
10+
11+
*** 1. Executing the Binlog Server utility to for the very first
12+
*** time on an empty storage and receive no real events.
13+
14+
*** 1a. Executing the Binlog Server utility on a storage that has
15+
*** only one binlog file that has only magic payload in it and
16+
*** receive no events.
17+
18+
*** Creating a simple table
19+
CREATE TABLE t1(id INT UNSIGNED NOT NULL AUTO_INCREMENT, PRIMARY KEY(id)) ENGINE=InnoDB;
20+
21+
*** 2. Executing the Binlog Server utility on a storage that has
22+
*** only one binlog file that has only magic payload in it and
23+
*** receive events from the CREATE TABLE transaction.
24+
25+
*** 2a. Executing the Binlog Server utility on a storage that has
26+
*** only one binlog file that has CREATE TABLE transaction and
27+
*** receive no events.
28+
29+
*** Filling the table with some data (one transaction with one
30+
*** insert and another one with two inserts).
31+
INSERT INTO t1 VALUES(DEFAULT);
32+
START TRANSACTION;
33+
INSERT INTO t1 VALUES(DEFAULT);
34+
INSERT INTO t1 VALUES(DEFAULT);
35+
COMMIT;
36+
37+
*** 3. Executing the Binlog Server utility on a storage that has
38+
*** only one binlog file with some data and receive events from
39+
*** the INSERT transactions.
40+
41+
*** 3a. Executing the Binlog Server utility on a storage that has
42+
*** only one binlog file with some data and receive no events.
43+
44+
*** Flushing the first binary log and switching to the second one.
45+
FLUSH BINARY LOGS;
46+
47+
*** 4. Executing the Binlog Server utility on a storage that has
48+
*** only one binlog file with some data and receive a single
49+
*** ROTATE event.
50+
51+
*** 4a. Executing the Binlog Server utility on a storage that has
52+
*** one binlog file with some data and another one with just
53+
*** magic payload and receive no events.
54+
55+
*** Filling the table with some more data (one transaction with one
56+
*** insert and another one with two inserts).
57+
INSERT INTO t1 VALUES(DEFAULT);
58+
START TRANSACTION;
59+
INSERT INTO t1 VALUES(DEFAULT);
60+
INSERT INTO t1 VALUES(DEFAULT);
61+
COMMIT;
62+
63+
*** 5. Executing the Binlog Server utility on a storage that has
64+
*** one binlog file with some data and another one with just
65+
*** magic payload and receive events from the second group of
66+
*** INSERT transactions.
67+
68+
*** 5a. Executing the Binlog Server utility on a storage that has
69+
*** two binlog files with some data and receive no events
70+
71+
*** Flushing the second binary log and switching to the third one.
72+
*** Immediately after that updating data inserted previously.
73+
FLUSH BINARY LOGS;
74+
UPDATE t1 SET id = id + 100;
75+
76+
*** 6. Executing the Binlog Server utility on a storage that has
77+
*** two binlog files with some data and receive a ROTATE event
78+
*** followed by events from the UPDATE transaction
79+
80+
*** 6a. Executing the Binlog Server utility on a storage that has
81+
*** three binlog files with some data and receive no events.
82+
83+
*** Flushing the third binary log and switching to the fourth one.
84+
*** Immediately after that deleting some data updated previously and
85+
*** flushing one more time switching to the fifth binary log file.
86+
FLUSH BINARY LOGS;
87+
DELETE FROM t1 WHERE id <= 103;
88+
FLUSH BINARY LOGS;
89+
90+
*** 7. Executing the Binlog Server utility on a storage that has
91+
*** three binlog files with some data and receive a ROTATE
92+
*** event followed by a events from the DELETE transaction
93+
*** followed by another ROTATE event
94+
95+
*** 7a. Executing the Binlog Server utility on a storage that has
96+
*** five binlog files with some data and receive no events.
97+
98+
*** Dropping the table.
99+
DROP TABLE t1;
100+
101+
*** Removing the Binlog Server utility storage directory.
102+
103+
*** Removing the Binlog Server utility log file.
104+
105+
*** Removing the Binlog Server utility configuration file.

mtr/binlog_streaming/t/binsrv.test

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ INSERT INTO t1 VALUES(DEFAULT);
4242
--let $binsrv_read_timeout = 60
4343
--let $binsrv_idle_time = 10
4444
--let $binsrv_verify_checksum = $extracted_init_connect_variable_value
45+
--let $binsrv_replication_mode = position
4546
--let $binsrv_checkpoint_size = 1
4647
--source ../include/set_up_binsrv_environment.inc
4748

mtr/binlog_streaming/t/checkpointing.test

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,8 @@ DROP TABLE t1;
5858
--let $binsrv_read_timeout = 60
5959
--let $binsrv_idle_time = 10
6060
--let $binsrv_verify_checksum = TRUE
61-
# Enabling checkointing at about 40% of expected single binlog file size, so that it
61+
--let $binsrv_replication_mode = position
62+
# Enabling checkpointing at about 40% of expected single binlog file size, so that it
6263
# will happen twice before rotation.
6364
# Like wise, from the time point of view, make interval pretty low so that
6465
# at lease a few interval checkpointing events will happen.

mtr/binlog_streaming/t/kill_and_restart.test

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,7 @@ INSERT INTO t1 VALUES(DEFAULT);
130130
--let $binsrv_read_timeout = 60
131131
--let $binsrv_idle_time = 10
132132
--let $binsrv_verify_checksum = TRUE
133+
--let $binsrv_replication_mode = position
133134
--source ../include/set_up_binsrv_environment.inc
134135

135136
--echo

mtr/binlog_streaming/t/pull_mode.test

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ eval $stmt_reset_binary_logs_and_gtids;
1919
--let $binsrv_read_timeout = 3
2020
--let $binsrv_idle_time = 1
2121
--let $binsrv_verify_checksum = TRUE
22+
--let $binsrv_replication_mode = position
2223
--source ../include/set_up_binsrv_environment.inc
2324

2425
--echo
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
[position_buffered]
2+
init-connect = SET @binsrv_buffering_mode = 'buffered'
3+
4+
[position_unbuffered]
5+
init-connect = SET @binsrv_buffering_mode = 'unbuffered'
6+
7+
[gtid_buffered]
8+
gtid-mode=on
9+
enforce-gtid-consistency
10+
init-connect = SET @binsrv_buffering_mode = 'buffered'
11+
12+
[gtid_unbuffered]
13+
gtid-mode=on
14+
enforce-gtid-consistency
15+
init-connect = SET @binsrv_buffering_mode = 'unbuffered'
Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
--source ../include/have_binsrv.inc
2+
3+
--source ../include/v80_v84_compatibility_defines.inc
4+
5+
# in case of --repeat=N, we need to start from a fresh binary log to make
6+
# this test deterministic
7+
--echo *** Resetting replication at the very beginning of the test.
8+
--disable_query_log
9+
eval $stmt_reset_binary_logs_and_gtids;
10+
--enable_query_log
11+
12+
# identifying backend storage type ('file' or 's3')
13+
--source ../include/identify_storage_backend.inc
14+
15+
# identifying utility buffering mode from the conbination
16+
--let $extracted_init_connect_variable_name = binsrv_buffering_mode
17+
--source ../include/extract_init_connect_variable_value.inc
18+
19+
# creating data directory, configuration file, etc.
20+
--let $binsrv_connect_timeout = 20
21+
--let $binsrv_read_timeout = 60
22+
--let $binsrv_idle_time = 10
23+
--let $binsrv_verify_checksum = TRUE
24+
--let $binsrv_replication_mode = `SELECT IF(@@global.gtid_mode, 'gtid', 'position')`
25+
if ($extracted_init_connect_variable_value == 'buffered')
26+
{
27+
--let $binsrv_checkpoint_size = 1G
28+
}
29+
if ($extracted_init_connect_variable_value == 'unbuffered')
30+
{
31+
--let $binsrv_checkpoint_size = 1
32+
}
33+
--source ../include/set_up_binsrv_environment.inc
34+
35+
--echo
36+
--echo *** 1. Executing the Binlog Server utility to for the very first
37+
--echo *** time on an empty storage and receive no real events.
38+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
39+
40+
--echo
41+
--echo *** 1a. Executing the Binlog Server utility on a storage that has
42+
--echo *** only one binlog file that has only magic payload in it and
43+
--echo *** receive no events.
44+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
45+
46+
--echo
47+
--echo *** Creating a simple table
48+
CREATE TABLE t1(id INT UNSIGNED NOT NULL AUTO_INCREMENT, PRIMARY KEY(id)) ENGINE=InnoDB;
49+
50+
--echo
51+
--echo *** 2. Executing the Binlog Server utility on a storage that has
52+
--echo *** only one binlog file that has only magic payload in it and
53+
--echo *** receive events from the CREATE TABLE transaction.
54+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
55+
56+
--echo
57+
--echo *** 2a. Executing the Binlog Server utility on a storage that has
58+
--echo *** only one binlog file that has CREATE TABLE transaction and
59+
--echo *** receive no events.
60+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
61+
62+
--echo
63+
--echo *** Filling the table with some data (one transaction with one
64+
--echo *** insert and another one with two inserts).
65+
INSERT INTO t1 VALUES(DEFAULT);
66+
START TRANSACTION;
67+
INSERT INTO t1 VALUES(DEFAULT);
68+
INSERT INTO t1 VALUES(DEFAULT);
69+
COMMIT;
70+
71+
--echo
72+
--echo *** 3. Executing the Binlog Server utility on a storage that has
73+
--echo *** only one binlog file with some data and receive events from
74+
--echo *** the INSERT transactions.
75+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
76+
77+
--echo
78+
--echo *** 3a. Executing the Binlog Server utility on a storage that has
79+
--echo *** only one binlog file with some data and receive no events.
80+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
81+
82+
--echo
83+
--echo *** Flushing the first binary log and switching to the second one.
84+
FLUSH BINARY LOGS;
85+
86+
--echo
87+
--echo *** 4. Executing the Binlog Server utility on a storage that has
88+
--echo *** only one binlog file with some data and receive a single
89+
--echo *** ROTATE event.
90+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
91+
92+
--echo
93+
--echo *** 4a. Executing the Binlog Server utility on a storage that has
94+
--echo *** one binlog file with some data and another one with just
95+
--echo *** magic payload and receive no events.
96+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
97+
98+
--echo
99+
--echo *** Filling the table with some more data (one transaction with one
100+
--echo *** insert and another one with two inserts).
101+
INSERT INTO t1 VALUES(DEFAULT);
102+
START TRANSACTION;
103+
INSERT INTO t1 VALUES(DEFAULT);
104+
INSERT INTO t1 VALUES(DEFAULT);
105+
COMMIT;
106+
107+
--echo
108+
--echo *** 5. Executing the Binlog Server utility on a storage that has
109+
--echo *** one binlog file with some data and another one with just
110+
--echo *** magic payload and receive events from the second group of
111+
--echo *** INSERT transactions.
112+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
113+
114+
--echo
115+
--echo *** 5a. Executing the Binlog Server utility on a storage that has
116+
--echo *** two binlog files with some data and receive no events
117+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
118+
119+
--echo
120+
--echo *** Flushing the second binary log and switching to the third one.
121+
--echo *** Immediately after that updating data inserted previously.
122+
FLUSH BINARY LOGS;
123+
UPDATE t1 SET id = id + 100;
124+
125+
--echo
126+
--echo *** 6. Executing the Binlog Server utility on a storage that has
127+
--echo *** two binlog files with some data and receive a ROTATE event
128+
--echo *** followed by events from the UPDATE transaction
129+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
130+
131+
--echo
132+
--echo *** 6a. Executing the Binlog Server utility on a storage that has
133+
--echo *** three binlog files with some data and receive no events.
134+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
135+
136+
--echo
137+
--echo *** Flushing the third binary log and switching to the fourth one.
138+
--echo *** Immediately after that deleting some data updated previously and
139+
--echo *** flushing one more time switching to the fifth binary log file.
140+
FLUSH BINARY LOGS;
141+
DELETE FROM t1 WHERE id <= 103;
142+
FLUSH BINARY LOGS;
143+
144+
--echo
145+
--echo *** 7. Executing the Binlog Server utility on a storage that has
146+
--echo *** three binlog files with some data and receive a ROTATE
147+
--echo *** event followed by a events from the DELETE transaction
148+
--echo *** followed by another ROTATE event
149+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
150+
151+
--echo
152+
--echo *** 7a. Executing the Binlog Server utility on a storage that has
153+
--echo *** five binlog files with some data and receive no events.
154+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
155+
156+
--echo
157+
--echo *** Dropping the table.
158+
DROP TABLE t1;
159+
160+
# cleaning up
161+
--source ../include/tear_down_binsrv_environment.inc

mtr/binlog_streaming/t/ssl_connection.test

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ eval GRANT REPLICATION SLAVE ON *.* TO '$x509_user_name'@'$localhost_ip';
3636
--let $binsrv_read_timeout = 60
3737
--let $binsrv_idle_time = 10
3838
--let $binsrv_verify_checksum = TRUE
39+
--let $binsrv_replication_mode = position
3940

4041
# a query that checks SSL connection status
4142
--let $ssl_status_query = SELECT IF(VARIABLE_VALUE = '', '<unspecified>', VARIABLE_VALUE) AS SSL_status FROM performance_schema.session_status WHERE VARIABLE_NAME = 'Ssl_version'

0 commit comments

Comments
 (0)