feat(cluster): Introduce lease mechanism for Kvrocks cluster to mitigate the brain-split issue by redwood9 · Pull Request #3397 · apache/kvrocks

redwood9 · 2026-03-20T02:04:49Z

Based on the ideas I proposed in issue #3380
#3380

this is the first step of the solution:
introducing a lease mechanism for the Kvrocks master node.
This mechanism ensures that the maximum split-brain window is bounded by the lease TTL. It also comes with flexible configuration options:

No config / disabled — Behaves exactly like the current system. If the lease feature is not enabled, there is zero impact on existing behavior.
Reject writes — When the lease expires, the node rejects all write operations to prevent any data conflicts / divergence.
Log only — On lease expiration, the node only logs an error (no write rejection). This makes it easy to catch via monitoring / alerting systems without disrupting traffic.

To minimize invasion into the existing codebase structure, the implementation follows these strategies:

Controller probing — Add a new HEARTBEAT command (non-conflicting with existing standard Redis commands). The response format remains unchanged to keep controller integration simple and backward-compatible.
Unified lease check — Add the lease validation logic inside Storage::writeToDB(), so all write paths are guarded in one central place.

I will also submit the corresponding HEARTBEAT implementation in the controller at the same time / in parallel.

Introduce a master lease mechanism that allows a master node to detect when it may have lost cluster ownership and optionally block writes to prevent split-brain data corruption. - Add `master_lease_mode` config option: disabled / log-only / block-write - Add lease atomics (`lease_deadline_`, `lease_owner_`) to `Storage` - Add `UpdateLease()` and `ResetLease()` methods on `Storage` - Check lease expiry in `Storage::Write()` and `writeToDB()` to cover both direct writes and `CommitTxn()` paths - Add `CLUSTERX HEARTBEAT <election_version> <ttl_ms>` command for controller-driven lease renewal - Reset master lease automatically on role transition to slave - Add C++ unit tests (`tests/cppunit/lease_test.cc`) - Add Go integration tests (`tests/gocase/integration/cluster/lease_test.go`)

git-hulk · 2026-03-23T12:24:42Z

src/commands/cmd_cluster.cc

+      if (args.size() != 5) return {Status::RedisParseErr, errWrongNumOfArguments};
+      master_node_id_ = args[2];
+
+      auto parse_lease_ms = ParseInt<uint64_t>(args[3], 10);


Should have a proper range for the lease timeout?

git-hulk · 2026-03-23T12:31:15Z

src/storage/storage.cc

                                   rocksdb::WriteBatch *updates) {
+  // Master lease check: applied here so it covers both Storage::Write() and CommitTxn().
+  // Only active when master_lease_mode != disabled. Read mode once to avoid TOCTOU.
+  auto lease_mode = config_->master_lease_mode;


The lease state should be maintained inside the server instead of the storage. And you can refuse the write operation like the slave's read-only mode. And the lease mode should only take effect if the cluster mode was enabled.

git-hulk · 2026-03-23T12:33:46Z

src/config/config.cc


+const std::vector<ConfigEnum<MasterLeaseMode>> master_lease_modes{
+    {"disabled", MasterLeaseMode::kDisabled},
+    {"log-only", MasterLeaseMode::kLogOnly},


log-only could be dry-run?

git-hulk · 2026-03-23T12:39:05Z

@redwood9 My another concern is that the Kvrocks will forcefully depends on the availability of the controller once the lease mode was enabled. And it cannot fully resolve the network partition issue, the network partition duration is totally depended on the lease timeout.

jihuayu · 2026-03-24T01:21:33Z

@git-hulk I agree with your point. I think it’s better to make this an optional feature, especially since AP and CP are mutually exclusive. Users can then choose based on their specific requirements.

jihuayu

For me, the main issue is that write-protection should be handled at the command level rather than the storage layer.

git-hulk · 2026-03-24T01:48:23Z

@git-hulk I agree with your point. I think it’s better to make this an optional feature, especially since AP and CP are mutually exclusive. Users can then choose based on their specific requirements.

Kvrocks is now supporting the WAIT command to achieve the majority writes behavior.

jihuayu · 2026-03-24T05:13:20Z

Kvrocks is now supporting the WAIT command to achieve the majority writes behavior.Kvrocks 现在支持等待命令，以实现大多数写入行为。

Thanks for the reminder. I now think that the core value of this PR is introducing a lease mechanism to the server, which helps define the upper bound of the write window for a stale master.
However, depended on a single controller poses a significant risk; I believe we should complete all feature only after supporting multiple controllers.

Or perhaps we should wait until the full design is finalized before considering this PR?

redwood9 · 2026-03-29T04:47:58Z

@redwood9 My another concern is that the Kvrocks will forcefully depends on the availability of the controller once the lease mode was enabled. And it cannot fully resolve the network partition issue, the network partition duration is totally depended on the lease timeout.

@git-hulk @jihuayu
This is a really good question and definitely worth discussing in depth. Here's my take:

The core of the lease mechanism is to allow the master to safely stop accepting writes during network partitions. Different use cases may call for different solutions:
1. In Redis Sentinel mode, min-replicas-to-write is the primary mechanism. However, if there's only one replica and the master/replica are spread across two regions, it's easy to trigger failures even when min-replicas-to-write==1.
2. For scenarios that demand extremely strong consistency and have low concurrency, we need to verify the master's identity on every write. This is exactly what step 4 in the proposal aims to solve.
3. For cases that can tolerate a short partition but not a prolonged one, leases are a great fit and can be flexibly configured based on actual needs. This is the main scenario this PR is targeting.
4. For situations where partitions are acceptable but we want alerts when they occur so humans can handle them, this PR also covers it. In this mode, the system behavior remains identical to the current implementation — we only add an alert when a lease anomaly is detected.
5. For the scenario mentioned in Travis ci #1.3, once the lease expires we can fall back to the approach in Travis ci #1.2 (verifying master status on every write). This fallback can be implemented as part of the Travis ci #1.2 work.
The controller is already a strongly consistent, multi-region system with high availability.
Overall, since users deploy Kvrocks in a wide variety of scenarios, I believe the best approach is to provide rich configuration options so they can make the right trade-offs for their specific use case.
Maybe we can dive deeper into the details. Once we settle on a concrete solution, I’d be happy to push a PR implementing it.

redwood9 added 5 commits March 19, 2026 17:20

Optimize lease determination logic

0022c9a

Merge branch 'unstable' of github.com:redwood9/kvrocks into unstable

9114bd8

Merge branch 'unstable' into unstable

222e277

This was referenced Mar 22, 2026

feat(split-brain): Introduce lease mechanism for Kvrocks master apache/kvrocks-controller#386

Closed

feat(controller): add lease config and HEARTBEAT support for master l… apache/kvrocks-controller#387

Open

git-hulk changed the title ~~feat(split-brain): Introduce lease mechanism for Kvrocks master~~ feat(cluster): Introduce lease mechanism for Kvrocks cluster to mitigate the brain-split issue Mar 23, 2026

git-hulk reviewed Mar 23, 2026

View reviewed changes

jihuayu reviewed Mar 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cluster): Introduce lease mechanism for Kvrocks cluster to mitigate the brain-split issue#3397

feat(cluster): Introduce lease mechanism for Kvrocks cluster to mitigate the brain-split issue#3397
redwood9 wants to merge 5 commits intoapache:unstablefrom
redwood9:unstable

redwood9 commented Mar 20, 2026

Uh oh!

git-hulk Mar 23, 2026

Uh oh!

git-hulk Mar 23, 2026

Uh oh!

git-hulk Mar 23, 2026

Uh oh!

git-hulk commented Mar 23, 2026

Uh oh!

jihuayu commented Mar 24, 2026

Uh oh!

jihuayu left a comment

Uh oh!

git-hulk commented Mar 24, 2026

Uh oh!

jihuayu commented Mar 24, 2026 •

edited

Loading

Uh oh!

redwood9 commented Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

redwood9 commented Mar 20, 2026

Uh oh!

git-hulk Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

git-hulk Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

git-hulk Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

git-hulk commented Mar 23, 2026

Uh oh!

jihuayu commented Mar 24, 2026

Uh oh!

jihuayu left a comment

Choose a reason for hiding this comment

Uh oh!

git-hulk commented Mar 24, 2026

Uh oh!

jihuayu commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

redwood9 commented Mar 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jihuayu commented Mar 24, 2026 •

edited

Loading