Skip to content

Commit 1b79789

Browse files
PS-10280 feature: Implement logic for search_by_timestamp mode (#89)
https://perconadev.atlassian.net/browse/PS-10280 Main executable can now be run in one more mode - 'search_by_timestamp'. Execute './binlog_server search_by_timestamp <config_file> <timestamp>' and get the list of all binlog files that have at least one event generated not later than the provided '<timestamp>'. <timestamp> is expected to be in ISO format, e.g. '2026-02-10T17:14:42'. In this mode, we simply initialize an instance of the 'binsrv::storage' class in 'read_only' mode, perform storage object listing and return binlog records that match the specified criteria in JSON format. The output is written to the standard output. The application immediately terminates after performing this operation. In addition, the application exit code can be used to determine whether the operation was successful. Refactored 'binsrv::storage' class - upon initialization it now stores the full list of known binlog files along with their metadata (file name, size, associated GTID set and min/max timestamp). Its constructor now takes one more parameter 'construction_mode' (which can be either 'querying_only' or 'streaming'). 'querying_only' is used in 'search_by_timestamp' mode. Class internal machinery now also keeps track of min/max timestamps found in the events stored in particular binlog files. They are now also being stored in binlog file metadata JSON file. Added new namespace 'binsrv::models' with the following classes: - 'binlog_record' - 'response_status' - 'error_response' - 'search_by_timestamp_response' which are used to form JSON responses for query operations (currently for 'search_by_timestamp' mode). Added 'binsrv::ctime_timestamp' class to represent a timestamp based on 'std::time_t' that can be easily converted to / from a string. Added 'binsrv::ctime_timestamp_range' class that represents a range (min and max values) of 'binsrv::ctime_timestamp'. 'binsrv::basic_storage_backend' interface extended with additional method 'get_object_uri()' that can generate storage URI for the given binlog file name (this functionality is used during generating JSON response for the 'search_by_timestamp' command). Both 'binsrv::filesystem_storage_backend' and 'binsrv::s3_storage_backend' give concrete implementation to this function. Added new 'binlog_streaming.search_by_timestamp' MTR test case with two combinations ('position' and 'gtid') that checks various scenarios of running Binlog Server utility in 'search_by_timestamp' mode. 'README.md' documentation file updated with another section describing the 'search_by_timestamp' mode.
1 parent a65581d commit 1b79789

38 files changed

Lines changed: 1453 additions & 129 deletions

CMakeLists.txt

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -215,6 +215,21 @@ set(source_files
215215
src/binsrv/gtids/uuid.hpp
216216
src/binsrv/gtids/uuid.cpp
217217

218+
# models files
219+
src/binsrv/models/binlog_file_record_fwd.hpp
220+
src/binsrv/models/binlog_file_record.hpp
221+
222+
src/binsrv/models/error_response_fwd.hpp
223+
src/binsrv/models/error_response.hpp
224+
src/binsrv/models/error_response.cpp
225+
226+
src/binsrv/models/response_status_type_fwd.hpp
227+
src/binsrv/models/response_status_type.hpp
228+
229+
src/binsrv/models/search_by_timestamp_response_fwd.hpp
230+
src/binsrv/models/search_by_timestamp_response.hpp
231+
src/binsrv/models/search_by_timestamp_response.cpp
232+
218233
# binlog files
219234
src/binsrv/basic_logger_fwd.hpp
220235
src/binsrv/basic_logger.hpp
@@ -231,6 +246,14 @@ set(source_files
231246
src/binsrv/cout_logger.hpp
232247
src/binsrv/cout_logger.cpp
233248

249+
src/binsrv/ctime_timestamp_fwd.hpp
250+
src/binsrv/ctime_timestamp.hpp
251+
src/binsrv/ctime_timestamp.cpp
252+
253+
src/binsrv/ctime_timestamp_range_fwd.hpp
254+
src/binsrv/ctime_timestamp_range.hpp
255+
src/binsrv/ctime_timestamp_range.cpp
256+
234257
src/binsrv/exception_handling_helpers.hpp
235258
src/binsrv/exception_handling_helpers.cpp
236259

README.md

Lines changed: 47 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -146,17 +146,18 @@ The result binary can be found under the following path `ws/percona-binlog-serve
146146

147147
Please run
148148
```bash
149-
./binlog_server <operation_mode> [ <json_config_file> ]
149+
./binlog_server <operation_mode> [ <json_config_file> [ <subcommand_parameter> ] ]
150150
```
151151
where
152-
`<operation_mode>` can be either `version`, `fetch`, or `pull`
153-
and
154-
`<json_config_file>` is an optional parameter (required only when `<operation_mode>` is not `version`) that represents a path to a JSON configuration file (described below).
152+
`<operation_mode>` can be either `version`, `search_by_timestamp`, `fetch`, or `pull`,
153+
`<json_config_file>` is an optional parameter (required when `<operation_mode>` is not `version`) that represents a path to a JSON configuration file (described below),
154+
and `<subcommand_parameter>` is an optional parameter (required only when `<operation_mode>` is `search_by_timestamp`), that represents a valid timestamp in ISO format (e.g. `2026-02-10T14:30:00`)
155155

156156
### Operation modes
157157

158158
Percona Binary Log Server utility can operate in three modes:
159159
- 'version'
160+
- 'search_by_timestamp'
160161
- 'fetch'
161162
- 'pull'
162163

@@ -172,6 +173,48 @@ may print
172173
0.1.0
173174
```
174175

176+
#### 'search_by_timestamp' operation mode
177+
178+
In this mode the utility requires one additional command line parameter `<iso_timestamp>` and will print to the standard output the list of binlog files stored in the Binary Log Server data directory that have at least one event whose timestamp is less or equal to the provided `<iso_timestamp>`.
179+
Along with the file name the output will also return its current size in bytes, timestamps and URI.
180+
For instance,
181+
```bash
182+
./binlog_server search_by_timestamp config.json 2026-02-10T14:30:00
183+
```
184+
may print
185+
```json
186+
{
187+
"status": "success",
188+
"result": [
189+
{
190+
"name": "binlog.000001",
191+
"size": 134217728,
192+
"uri": "s3://binsrv-bucket/storage/binlog.000001",
193+
"min_timestamp":"2026-02-09T17:22:01",
194+
"max_timestamp":"2026-02-09T17:22:08"
195+
},
196+
{
197+
"name": "binlog.000002",
198+
"size": 134217728,
199+
"uri": "s3://binsrv-bucket/storage/binlog.000002",
200+
"min_timestamp":"2026-02-09T17:22:08",
201+
"max_timestamp":"2026-02-09T17:22:09"
202+
}
203+
]
204+
}
205+
```
206+
If an error occurs,
207+
```json
208+
{
209+
"status": "error",
210+
"message": "<reason>"
211+
}
212+
```
213+
The `<reason>` may be one of the following (but not limited to):
214+
- `Invalid timestamp format`
215+
- `Binlog storage is empty`
216+
- `Timestamp is too old`
217+
175218
#### 'fetch' operation mode
176219

177220
In this mode the utility tries to connect to a remote MySQL server, switch connection to replication mode and read events from all available binary logs already stored on the server. After reading the very last event, the utility gracefully disconnects and exits.
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
*** Resetting replication at the very beginning of the test.
2+
3+
*** Generating a configuration file in JSON format for the Binlog
4+
*** Server utility.
5+
6+
*** Determining binlog file directory from the server.
7+
8+
*** Creating a temporary directory <BINSRV_STORAGE_PATH> for storing
9+
*** binlog files downloaded via the Binlog Server utility.
10+
11+
*** 1. Executing the Binlog Server utility in the 'search_by_timestamp'
12+
*** mode with an invalid timestamp
13+
include/read_file_to_var.inc
14+
15+
*** 2. Executing the Binlog Server utility in the 'search_by_timestamp'
16+
*** mode with the beginning of Epoch (Jan 1, 1970, 00:00:00) on an
17+
*** empty storage
18+
include/read_file_to_var.inc
19+
20+
*** Creating a simple table.
21+
CREATE TABLE t1(id INT UNSIGNED NOT NULL AUTO_INCREMENT, PRIMARY KEY(id)) ENGINE=InnoDB;
22+
23+
*** Filling the table with some data.
24+
INSERT INTO t1 VALUES();
25+
26+
*** Flushing the first binary log and switching to the second one.
27+
FLUSH BINARY LOGS;
28+
29+
*** Filling the table with more data.
30+
INSERT INTO t1 VALUES();
31+
32+
*** Executing the Binlog Server utility and fetching all events.
33+
34+
*** 3. Executing the Binlog Server utility in the 'search_by_timestamp'
35+
*** mode with the beginning of Epoch (Jan 1, 1970, 00:00:00) on an
36+
*** non-empty storage
37+
include/read_file_to_var.inc
38+
39+
*** 4. Executing the Binlog Server utility in the 'search_by_timestamp'
40+
*** mode with the <before_activity_timestamp>
41+
include/read_file_to_var.inc
42+
43+
*** 5. Executing the Binlog Server utility in the 'search_by_timestamp'
44+
*** mode with the <inside_activity_timestamp> and expecting one binlog
45+
*** file to be returned
46+
include/read_file_to_var.inc
47+
48+
*** 6. Executing the Binlog Server utility in the 'search_by_timestamp'
49+
*** mode with the <after_activity_timestamp> and expecting two binlog
50+
*** files to be returned
51+
include/read_file_to_var.inc
52+
53+
*** Removing the search result file.
54+
55+
*** Dropping the table.
56+
DROP TABLE t1;
57+
58+
*** Removing the Binlog Server utility storage directory.
59+
60+
*** Removing the Binlog Server utility log file.
61+
62+
*** Removing the Binlog Server utility configuration file.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
[position]
2+
3+
[gtid]
4+
gtid-mode=on
5+
enforce-gtid-consistency
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
--source ../include/have_binsrv.inc
2+
3+
--source ../include/v80_v84_compatibility_defines.inc
4+
5+
# in case of --repeat=N, we need to start from a fresh binary log to make
6+
# this test deterministic
7+
--echo *** Resetting replication at the very beginning of the test.
8+
--disable_query_log
9+
eval $stmt_reset_binary_logs_and_gtids;
10+
--enable_query_log
11+
12+
# identifying backend storage type ('file' or 's3')
13+
--source ../include/identify_storage_backend.inc
14+
15+
# creating data directory, configuration file, etc.
16+
--let $binsrv_connect_timeout = 20
17+
--let $binsrv_read_timeout = 60
18+
--let $binsrv_idle_time = 10
19+
--let $binsrv_verify_checksum = TRUE
20+
--let $binsrv_replication_mode = `SELECT IF(@@global.gtid_mode, 'gtid', 'position')`
21+
--let $binsrv_checkpoint_size = 1
22+
--source ../include/set_up_binsrv_environment.inc
23+
24+
--let $timestamp_query = SELECT DATE_FORMAT(CONVERT_TZ(NOW(), @@session.time_zone, '+00:00'),'%Y-%m-%dT%H:%i:%s')
25+
26+
--let $read_from_file = $MYSQL_TMP_DIR/search_result.json
27+
28+
--echo
29+
--echo *** 1. Executing the Binlog Server utility in the 'search_by_timestamp'
30+
--echo *** mode with an invalid timestamp
31+
--error 1
32+
--exec $BINSRV search_by_timestamp $binsrv_config_file_path 20000101T000000 > $read_from_file
33+
--source include/read_file_to_var.inc
34+
--assert(`SELECT JSON_EXTRACT('$result', '$.status') = 'error'`)
35+
--assert(`SELECT JSON_EXTRACT('$result', '$.message') = 'Invalid timestamp format'`)
36+
37+
--echo
38+
--echo *** 2. Executing the Binlog Server utility in the 'search_by_timestamp'
39+
--echo *** mode with the beginning of Epoch (Jan 1, 1970, 00:00:00) on an
40+
--echo *** empty storage
41+
--error 1
42+
--exec $BINSRV search_by_timestamp $binsrv_config_file_path 1970-01-01T00:00:00 > $read_from_file
43+
--source include/read_file_to_var.inc
44+
--assert(`SELECT JSON_EXTRACT('$result', '$.status') = 'error'`)
45+
--assert(`SELECT JSON_EXTRACT('$result', '$.message') = 'Binlog storage is empty'`)
46+
47+
--let $before_activity_timestamp = 2000-01-01T00:00:00
48+
49+
--echo
50+
--echo *** Creating a simple table.
51+
CREATE TABLE t1(id INT UNSIGNED NOT NULL AUTO_INCREMENT, PRIMARY KEY(id)) ENGINE=InnoDB;
52+
53+
--echo
54+
--echo *** Filling the table with some data.
55+
INSERT INTO t1 VALUES();
56+
57+
--let $inside_activity_timestamp = `$timestamp_query`
58+
--sleep 2
59+
60+
--echo
61+
--echo *** Flushing the first binary log and switching to the second one.
62+
FLUSH BINARY LOGS;
63+
64+
--echo
65+
--echo *** Filling the table with more data.
66+
INSERT INTO t1 VALUES();
67+
68+
--let $after_activity_timestamp = `$timestamp_query`
69+
70+
--echo
71+
--echo *** Executing the Binlog Server utility and fetching all events.
72+
--exec $BINSRV fetch $binsrv_config_file_path > /dev/null
73+
74+
--echo
75+
--echo *** 3. Executing the Binlog Server utility in the 'search_by_timestamp'
76+
--echo *** mode with the beginning of Epoch (Jan 1, 1970, 00:00:00) on an
77+
--echo *** non-empty storage
78+
--error 1
79+
--exec $BINSRV search_by_timestamp $binsrv_config_file_path 1970-01-01T00:00:00 > $read_from_file
80+
--source include/read_file_to_var.inc
81+
--assert(`SELECT JSON_EXTRACT('$result', '$.status') = 'error'`)
82+
--assert(`SELECT JSON_EXTRACT('$result', '$.message') = 'Timestamp is too old'`)
83+
84+
--echo
85+
--echo *** 4. Executing the Binlog Server utility in the 'search_by_timestamp'
86+
--echo *** mode with the <before_activity_timestamp>
87+
--error 1
88+
--exec $BINSRV search_by_timestamp $binsrv_config_file_path $before_activity_timestamp > $read_from_file
89+
--source include/read_file_to_var.inc
90+
--assert(`SELECT JSON_EXTRACT('$result', '$.status') = 'error'`)
91+
--assert(`SELECT JSON_EXTRACT('$result', '$.message') = 'Timestamp is too old'`)
92+
93+
--echo
94+
--echo *** 5. Executing the Binlog Server utility in the 'search_by_timestamp'
95+
--echo *** mode with the <inside_activity_timestamp> and expecting one binlog
96+
--echo *** file to be returned
97+
--exec $BINSRV search_by_timestamp $binsrv_config_file_path $inside_activity_timestamp > $read_from_file
98+
--source include/read_file_to_var.inc
99+
--assert(`SELECT JSON_EXTRACT('$result', '$.status') = 'success'`)
100+
--assert(`SELECT JSON_LENGTH(JSON_EXTRACT('$result', '$.result')) = 1`)
101+
102+
--echo
103+
--echo *** 6. Executing the Binlog Server utility in the 'search_by_timestamp'
104+
--echo *** mode with the <after_activity_timestamp> and expecting two binlog
105+
--echo *** files to be returned
106+
--exec $BINSRV search_by_timestamp $binsrv_config_file_path $after_activity_timestamp > $read_from_file
107+
--source include/read_file_to_var.inc
108+
--assert(`SELECT JSON_EXTRACT('$result', '$.status') = 'success'`)
109+
--assert(`SELECT JSON_LENGTH(JSON_EXTRACT('$result', '$.result')) = 2`)
110+
111+
--echo
112+
--echo *** Removing the search result file.
113+
--remove_file $read_from_file
114+
115+
--echo
116+
--echo *** Dropping the table.
117+
DROP TABLE t1;
118+
119+
# cleaning up
120+
--source ../include/tear_down_binsrv_environment.inc

0 commit comments

Comments
 (0)