Skip to content

Commit 69265fc

Browse files
committed
feat: Add SYNC command for manual table synchronization
Add SYNC and SYNC_STATUS commands to allow manual triggering of table synchronization from MySQL binlog. This addresses the need for on-demand synchronization without relying solely on automatic replication. Changes: - Add SyncHandler for SYNC and SYNC_STATUS commands - Add sync state tracking (IN_PROGRESS, COMPLETED, FAILED) - Add syncing_tables set to prevent concurrent syncs - Update query parser to support SYNC commands - Add comprehensive tests for sync functionality - Add documentation for SYNC command usage - Fix clang-tidy warnings: - Add NOLINT comments for intentional reference members - Fix else-after-return in sync handler - Replace magic number with named constant - Add explicit static_cast for narrowing conversions - Add deleted copy/move for RAII guard - Use explicit nullptr comparison Closes #2
1 parent eeaebee commit 69265fc

32 files changed

+1523
-216
lines changed

docs/en/configuration.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ build:
5151

5252
replication:
5353
enable: true
54+
auto_initial_snapshot: false
5455
server_id: 12345
5556
start_from: "snapshot"
5657
queue_size: 10000
@@ -326,6 +327,11 @@ Index build configuration:
326327
MySQL binlog replication settings:
327328

328329
- **enable**: Enable binlog replication (default: `true`)
330+
- **auto_initial_snapshot**: Automatically build snapshot on startup (default: `false`)
331+
- `false`: Manual snapshot synchronization using `SYNC` command (recommended, safe by default)
332+
- `true`: Legacy behavior - automatically build snapshot on startup
333+
- Setting to `false` prevents unexpected MySQL load on startup
334+
- See [SYNC Command Guide](sync_command.md) for manual synchronization
329335
- **server_id**: MySQL server ID (required, must be non-zero and unique in replication topology)
330336
- Generate a random number or use a unique value for your environment
331337
- Example: `12345`

docs/en/protocol.md

Lines changed: 122 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ SEARCH articles hello
4141

4242
With filters and pagination:
4343
```
44-
SEARCH articles tech FILTER status=1 LIMIT 10
44+
SEARCH articles tech FILTER status = 1 LIMIT 10
4545
```
4646

4747
### Response
@@ -67,7 +67,7 @@ COUNT <table> <text> [OPTIONS]
6767
### Example
6868

6969
```
70-
COUNT articles tech AND AI FILTER status=1
70+
COUNT articles tech AND AI FILTER status = 1
7171
```
7272

7373
### Response
@@ -209,40 +209,6 @@ Returns a YAML-style formatted configuration showing:
209209

210210
---
211211

212-
## CONFIG VERIFY
213-
214-
Validate current configuration and check system status.
215-
216-
### Syntax
217-
218-
```
219-
CONFIG VERIFY
220-
```
221-
222-
### Response
223-
224-
```
225-
OK CONFIG VERIFIED
226-
tables: <count>
227-
228-
table: <table_name>
229-
primary_key: <column>
230-
text_source: <source>
231-
ngram_size: <size>
232-
filters: <count>
233-
required_filters: <count>
234-
status: loaded|not_loaded
235-
documents: <count>
236-
terms: <count>
237-
238-
replication:
239-
status: running|stopped
240-
gtid: <gtid>
241-
242-
END
243-
```
244-
245-
---
246212

247213
## DUMP Commands
248214

@@ -254,12 +220,12 @@ Save complete snapshot to single binary file (`.dmp`).
254220

255221
**Syntax:**
256222
```
257-
DUMP SAVE [<filepath>] [WITH STATISTICS]
223+
DUMP SAVE [<filepath>] [--with-stats]
258224
```
259225

260226
**Example:**
261227
```
262-
DUMP SAVE /backup/mygramdb.dmp WITH STATISTICS
228+
DUMP SAVE /backup/mygramdb.dmp --with-stats
263229
```
264230

265231
### DUMP LOAD
@@ -367,6 +333,123 @@ OK REPLICATION STARTED
367333

368334
---
369335

336+
## SYNC
337+
338+
Manually trigger snapshot synchronization from MySQL to MygramDB for a specific table.
339+
340+
### Syntax
341+
342+
```
343+
SYNC <table_name>
344+
```
345+
346+
### Parameters
347+
348+
- **table_name**: Name of the table to synchronize (must be configured in config file)
349+
350+
### Response (Success)
351+
352+
```
353+
OK SYNC STARTED table=<table_name> job_id=1
354+
```
355+
356+
### Response (Error)
357+
358+
```
359+
ERROR SYNC already in progress for table '<table_name>'
360+
ERROR Memory critically low. Cannot start SYNC. Check system memory.
361+
ERROR Table '<table_name>' not found in configuration
362+
```
363+
364+
### Behavior
365+
366+
- Runs asynchronously in the background
367+
- Returns immediately after starting
368+
- Builds snapshot from MySQL using SELECT query
369+
- Captures GTID at snapshot time
370+
- Automatically starts binlog replication with captured GTID when complete
371+
372+
### Conflicts
373+
374+
- **DUMP LOAD**: Blocked during SYNC (prevents data corruption)
375+
- **REPLICATION START**: Blocked during SYNC (SYNC auto-starts replication)
376+
- **SYNC** (same table): Blocked if already in progress
377+
378+
### Example
379+
380+
```bash
381+
# Using CLI
382+
mygram-cli SYNC articles
383+
384+
# Using telnet
385+
echo "SYNC articles" | nc localhost 11016
386+
```
387+
388+
---
389+
390+
## SYNC STATUS
391+
392+
Check the progress and status of SYNC operations.
393+
394+
### Syntax
395+
396+
```
397+
SYNC STATUS
398+
```
399+
400+
### Response Examples
401+
402+
**In Progress:**
403+
```
404+
table=articles status=IN_PROGRESS progress=10000/25000 rows (40.0%) rate=5000 rows/s
405+
```
406+
407+
**Completed:**
408+
```
409+
table=articles status=COMPLETED rows=25000 time=5.2s gtid=uuid:123 replication=STARTED
410+
```
411+
412+
**Failed:**
413+
```
414+
table=articles status=FAILED rows=5000 error="MySQL connection lost"
415+
```
416+
417+
**Idle:**
418+
```
419+
status=IDLE message="No sync operation performed"
420+
```
421+
422+
### Status Fields
423+
424+
| Field | Description |
425+
|-------|-------------|
426+
| `table` | Table name being synced |
427+
| `status` | `IN_PROGRESS`, `COMPLETED`, `FAILED`, `IDLE`, `CANCELLED` |
428+
| `progress` | Current/total rows processed |
429+
| `rate` | Processing rate (rows/s) |
430+
| `rows` | Total rows processed |
431+
| `time` | Total processing time |
432+
| `gtid` | Captured snapshot GTID |
433+
| `replication` | Replication status: `STARTED`, `ALREADY_RUNNING`, `DISABLED`, `FAILED` |
434+
| `error` | Error message (if failed) |
435+
436+
### Example
437+
438+
```bash
439+
# Using CLI
440+
mygram-cli SYNC STATUS
441+
442+
# Using telnet
443+
echo "SYNC STATUS" | nc localhost 11016
444+
```
445+
446+
### See Also
447+
448+
- [SYNC Command Guide](sync_command.md) - Detailed usage guide
449+
- [Replication Guide](replication.md) - Manual snapshot synchronization setup
450+
451+
---
452+
370453
## OPTIMIZE Command
371454

372455
Optimize index posting lists (convert Delta Encoding to Roaring Bitmap based on density).
@@ -567,7 +650,7 @@ In interactive mode, type `help` to see available commands:
567650
> help
568651
Available commands:
569652
SEARCH, COUNT, GET - Search and retrieval
570-
INFO, CONFIG, CONFIG VERIFY - Server information and validation
653+
INFO, CONFIG - Server information and configuration
571654
DUMP SAVE/LOAD/VERIFY/INFO - Snapshot management
572655
REPLICATION STATUS/STOP/START - Replication control
573656
OPTIMIZE - Index optimization

0 commit comments

Comments
 (0)