Skip to content

Latest commit

 

History

History
349 lines (219 loc) · 10.1 KB

v1.3.7.md

File metadata and controls

349 lines (219 loc) · 10.1 KB

MygramDB v1.3.7 Release Notes

Release Date: 2025-12-02 Type: Feature Release Previous Version: v1.3.6

Overview

Version 1.3.7 introduces asynchronous DUMP SAVE operations and the new DUMP STATUS command for improved operational usability. This release also includes critical bug fixes and logging standardization.

Highlights:

  • Async DUMP SAVE - Non-blocking dump operations with immediate response
  • DUMP STATUS command - Real-time progress monitoring for dump operations
  • Critical SIGSEGV fix - VariableHandler ownership issue resolved
  • Auto-dump/manual dump mutual exclusion - Prevents concurrent dump execution

New Features

1. Async DUMP SAVE

Type: Feature

DUMP SAVE now executes asynchronously, returning immediately while the dump operation runs in the background.

Before: DUMP SAVE blocked until the entire dump operation completed, potentially taking minutes for large datasets.

After: DUMP SAVE returns immediately with OK DUMP_STARTED <filepath> and performs the dump in a background worker thread.

Key Changes:

  • Background worker thread (DumpSaveWorker) performs actual dump operation
  • Replication pausing/resuming handled within worker thread for proper async flow
  • Synchronous fallback when dump_progress is unavailable (e.g., unit tests)
  • Error messages suggest using DUMP STATUS for progress monitoring

Files Changed:

  • src/server/handlers/dump_handler.cpp - Async implementation with worker thread
  • src/server/handlers/dump_handler.h - DumpSaveWorker declaration
  • src/server/server_types.h - DumpProgress struct

2. DUMP STATUS Command

Type: Feature

New command to query dump operation progress in real-time.

Syntax:

DUMP STATUS

Response Fields:

Field Description
save_in_progress Whether a save operation is running
load_in_progress Whether a load operation is running
replication_paused_for_dump Whether replication was paused for the dump
status Current status: IDLE, SAVING, LOADING, COMPLETED, FAILED
filepath Path of the dump file being processed
tables_processed Number of tables completed
tables_total Total number of tables
current_table Name of the table currently being processed
elapsed_seconds Time elapsed since operation started
result_filepath Final filepath on completion
error Error message on failure

Files Changed:

  • src/server/handlers/dump_handler.cpp - DUMP STATUS handler
  • src/server/server_lifecycle_manager.cpp - DUMP_STATUS registration
  • src/query/query_parser.cpp - DUMP_STATUS query type

3. DumpProgress Structure

Type: Internal

Thread-safe progress tracking structure for dump operations.

Features:

  • Mutex-protected state updates
  • DumpStatus enum: IDLE, SAVING, LOADING, COMPLETED, FAILED
  • Methods: Reset(), UpdateTable(), Complete(), Fail(), JoinWorker()
  • Tracks start_time/end_time for elapsed time calculation
  • Manages worker thread lifecycle

Files Changed:

  • src/server/server_types.h - DumpProgress struct and DumpStatus enum

Bug Fixes

1. Critical: Use-After-Free in RemoveDocument

Type: Critical Severity: Critical

Fix use-after-free crash in DocumentStore::RemoveDocument during concurrent delete operations.

Root Cause: The function stored a reference to the primary key string from the map, then erased the map entry (invalidating the reference), and finally used the dangling reference in the log statement. This bug existed since the initial commit but was exposed by Linux CI's memory allocator behavior during concurrent tests.

Symptom: Segmentation fault in DocumentStoreTest.ConcurrentDeletes test on Linux CI.

Solution:

  • Copy the primary key string before erasing the map entry
  • Add stress tests (DocumentStoreStressTest) with SLOW label for regression detection

Files Changed:

  • src/storage/document_store.cpp - Copy string before erase
  • tests/storage/document_store_stress_test.cpp - New stress test file
  • tests/storage/CMakeLists.txt - Add stress test with SLOW label

2. Critical: SIGSEGV on SET/SHOW VARIABLES

Type: Critical Severity: Critical

Fix segmentation fault when executing SET or SHOW VARIABLES commands.

Root Cause: VariableHandler and RuntimeVariableManager ownership was not transferred from InitializedComponents to TcpServer. When InitializedComponents went out of scope, both objects were destroyed, leaving HandlerContext with a dangling pointer.

Solution:

  • Add variable_manager_ and variable_handler_ members to TcpServer
  • Transfer ownership via std::move in TcpServer::Start()
  • Update HandlerContext to point to TcpServer-owned variable_manager

Files Changed:

  • src/server/tcp_server.h - Add member declarations
  • src/server/tcp_server.cpp - Transfer ownership and update GetVariableManager()

3. Medium: Auto-dump/Manual DUMP SAVE Mutual Exclusion

Type: Bug Fix Severity: Medium

Implement proper synchronization between SnapshotScheduler (auto-dump) and manual DUMP SAVE operations to prevent concurrent dump execution.

Problem: Auto-dump and manual DUMP SAVE could execute concurrently, potentially causing data corruption.

Solution:

  • Pass dump_save_in_progress flag to SnapshotScheduler constructor
  • SnapshotScheduler checks and sets flag before taking automatic snapshots
  • Auto-dump is skipped when manual DUMP SAVE is in progress
  • Manual DUMP SAVE is blocked when auto-dump is running

Flag Renaming:

  • read_only_ -> dump_save_in_progress_
  • loading_ -> dump_load_in_progress_
  • GetLoadingFlag() -> GetDumpLoadInProgressFlag()

Files Changed:

  • src/server/snapshot_scheduler.cpp - Flag check logic
  • src/server/snapshot_scheduler.h - Constructor parameter
  • src/server/handlers/dump_handler.cpp - Updated flag names
  • Multiple test files - Updated flag names

4. Low: DUMP LOAD GTID Restoration

Type: Bug Fix Severity: Low

GTID is now restored even when replication was not running before DUMP LOAD, enabling manual REPLICATION START after loading a dump file.

Files Changed:

  • src/server/handlers/dump_handler.cpp - GTID restoration logic

5. Low: BinlogReader::IsRunning() Flag Check

Type: Bug Fix Severity: Low

Fix BinlogReader::IsRunning() to check both running_ and should_stop_ flags. Also reset should_stop_ flag in Stop() for proper restart behavior.

Files Changed:

  • src/mysql/binlog_reader.cpp - IsRunning() and Stop() logic

Refactoring

1. Structured Logging Migration

Type: Refactoring

Standardize logging across the codebase by converting spdlog calls to StructuredLog API.

Changes:

  • Convert all spdlog::debug() to StructuredLog().Debug()
  • Convert spdlog::error() in binlog_event_parser.cpp to StructuredLog().Error()
  • Add context fields: doc_id, text_length, ngrams counts, batch_size
  • Add connection info: host, port for binlog_reader and server_ready
  • Simplify verbose logs by removing redundant "reason" fields
  • Update tests to check new log message format

Files Changed:

  • src/mysql/binlog_event_parser.cpp
  • src/mysql/binlog_reader.cpp
  • src/mysql/connection.cpp
  • src/mysql/rows_parser.cpp
  • src/index/index.cpp
  • src/index/posting_list.cpp
  • src/storage/document_store.cpp
  • And 20+ more files

2. IBinlogReader Interface

Type: Refactoring

Introduce IBinlogReader interface for better testability.

Changes:

  • Create IBinlogReader interface with pure virtual methods
  • BinlogReader implements IBinlogReader
  • Handlers use IBinlogReader instead of concrete class
  • Add deleted copy/move constructors and protected default constructor
  • Add final keyword and override on destructor

Files Changed:

  • src/mysql/binlog_reader_interface.h - New interface file
  • src/mysql/binlog_reader.h - Implement interface
  • src/server/handlers/* - Use interface type

Documentation

1. DUMP SAVE Async Behavior

Updated protocol documentation to reflect new async behavior:

  • Document asynchronous execution behavior
  • Add response format showing OK DUMP_STARTED message
  • Note to use DUMP STATUS for progress monitoring

Files Changed:

  • docs/en/protocol.md
  • docs/ja/protocol.md

2. DUMP STATUS Command

Added complete documentation for the new command:

  • Syntax and response format
  • All response field descriptions
  • Status value meanings
  • Example responses

Files Changed:

  • docs/en/protocol.md
  • docs/ja/protocol.md

Testing

New Tests

  • 5 new DUMP STATUS tests (idle, save in progress, load in progress, replication paused, GTID preservation)
  • GTID restoration test with MockBinlogReader
  • Variable handler ownership test
  • 2 new stress tests for RemoveDocument use-after-free regression (DocumentStoreStressTest)

Test Infrastructure

  • Add DumpHandlerGtidTest fixture with MockBinlogReader
  • Add document_store_stress_test.cpp with SLOW label for CI exclusion
  • Update 10+ test files with new flag names

Files Changed Summary

Category Files
Core Features dump_handler.cpp/h, server_types.h, query_parser.cpp
Bug Fixes document_store.cpp, tcp_server.cpp/h, snapshot_scheduler.cpp/h, binlog_reader.cpp
Refactoring 25+ files (StructuredLog migration)
Interface binlog_reader_interface.h
Documentation docs/en/protocol.md, docs/ja/protocol.md
Tests document_store_stress_test.cpp, dump_handler_test.cpp, variable_handler_test.cpp, 10+ updated

Total: 68 files changed

Upgrade Instructions

From v1.3.6 or earlier

  1. Stop MygramDB
  2. Upgrade to v1.3.7
  3. Restart MygramDB

Important: After upgrade, DUMP SAVE commands will behave asynchronously. Update any scripts that depend on synchronous DUMP SAVE completion to use DUMP STATUS for progress monitoring.

Known Issues

None.

Migration Notes

No migration required. This is a backward-compatible release.

Behavioral Change: DUMP SAVE is now asynchronous. Scripts waiting for DUMP SAVE completion should be updated to poll DUMP STATUS instead.