feat: add server-side `KvTransaction` gRPC API (client gated by feature flag) by drmingdrmer · Pull Request #55 · databendlabs/databend-meta

drmingdrmer · 2026-02-17T15:43:38Z

Changelog

feat: add server-side `KvTransaction` gRPC API (client gated by feature flag)

The existing Transaction endpoint stores protobuf TxnRequest directly in the
raft log, coupling wire format to persistence. Any proto schema change risks
breaking raft log deserialization. The new KvTransaction endpoint introduces a
three-layer architecture: protobuf transport types for the wire, Rust-native
serde types (kv_transaction::Transaction) for raft log storage, and conversions
between them. This lets the proto schema evolve freely without affecting stored
data.

The new API uses a cleaner branch-based model where each branch has an optional
predicate and a list of operations, evaluated in order. This replaces the legacy
dual execution paths, stringly-typed execution_path, and misleading success
field of TxnRequest.

The server now exposes the KvTransaction RPC endpoint, but the client does not
use it by default — ClientHandle::transaction_v2() is gated behind the
transaction-v2 feature flag and only enabled in tests. Production client code
continues to use the old Transaction API. The new endpoint will be adopted on
the client side in a future change once the server has been deployed widely
enough.

The applier is rewritten to operate on kv_transaction::* types. The old
Cmd::Transaction(TxnRequest) path converts to kv_transaction::Transaction
internally and delegates to the same code. Both paths return
AppliedState::KvTxnReply(pb::KvTransactionReply).

Changes:

Add kv_transaction module with Transaction, Branch, Predicate,
Condition, Operation, Operand, CompareOperator storage types
Add Cmd::KvTransaction(Transaction) variant
Replace AppliedState::TxnReply with AppliedState::KvTxnReply
Add KvTransaction RPC using redirect pattern (not forward), returning
leader endpoint in gRPC metadata for client-side redirect
Gate ClientHandle::transaction_v2() behind transaction-v2 feature flag
Register Feature::KvTransaction at version 260217.0.0

chore: Bump ver: 260217.0.0

refactor: reorganize crates into common/client/server categories

Group the 11 flat crates under crates/ into three category
subdirectories that reflect the dependency architecture: common/
(types, runtime-api, version, test-harness), client/ (kvapi,
kvapi-test-suite, kvapi-tests, client), and server/ (sled-store,
raft-store, service).

The client crate now re-exports kvapi and kvapi_test_suite, and
the service crate re-exports raft_store and sled_store, so
downstream consumers only need one dependency per category.

Changes:

Add pub extern crate ... as ... re-exports in client and service
Add databend-meta-kvapi-test-suite as a dependency of the client crate

Improvement

Group the 11 flat crates under `crates/` into three category subdirectories that reflect the dependency architecture: `common/` (types, runtime-api, version, test-harness), `client/` (kvapi, kvapi-test-suite, kvapi-tests, client), and `server/` (sled-store, raft-store, service). The client crate now re-exports `kvapi` and `kvapi_test_suite`, and the service crate re-exports `raft_store` and `sled_store`, so downstream consumers only need one dependency per category. Changes: - Add `pub extern crate ... as ...` re-exports in client and service - Add `databend-meta-kvapi-test-suite` as a dependency of the client crate

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 877839b8f3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

crates/server/service/src/meta_node/meta_node.rs

crates/common/types/src/kv_transaction/convert.rs

xp-trumpet

@xp-trumpet partially reviewed 325 files and all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on drmingdrmer).

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

xp-trumpet

@xp-trumpet partially reviewed 34 files and all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on drmingdrmer).

Copilot

Pull request overview

Copilot reviewed 33 out of 325 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-17T16:48:33Z

crates/common/types/src/kv_transaction/convert.rs

+impl From<pb::TxnOp> for Operation {
+    fn from(op: pb::TxnOp) -> Self {
+        use crate::txn_op;
+
+        match op.request {
+            Some(txn_op::Request::Get(g)) => Operation::get(g.key),
+            Some(txn_op::Request::Put(p)) => Operation::Put(operation::Put {
+                key: p.key,
+                value: p.value,
+                expire_at_ms: p.expire_at,
+                ttl_ms: p.ttl_ms,
+            }),
+            Some(txn_op::Request::Delete(d)) => Operation::Delete(operation::Delete {
+                key: d.key,
+                match_seq: d.match_seq,
+            }),
+            Some(txn_op::Request::DeleteByPrefix(d)) => Operation::delete_by_prefix(d.prefix),
+            Some(txn_op::Request::FetchIncreaseU64(f)) => {
+                Operation::FetchIncreaseU64(operation::FetchIncreaseU64 {
+                    key: f.key,
+                    match_seq: f.match_seq,
+                    delta: f.delta,
+                    floor: f.max_value,
+                })
+            }
+            Some(txn_op::Request::PutSequential(p)) => {
+                Operation::PutSequential(operation::PutSequential {
+                    prefix: p.prefix,
+                    sequence_key: p.sequence_key,
+                    value: p.value,
+                    expire_at_ms: p.expires_at_ms,
+                    ttl_ms: p.ttl_ms,
+                })
+            }
+            None => Operation::get(""),
+        }
+    }
+}


TxnOp { request: None } used to be skipped by the legacy evaluator (if let Some(request) { ... }). Converting it to Operation::get(\"\") changes behavior and can cause unintended reads of the empty key. Consider representing this as Option<Operation> (return None when request is missing) and filter_map when building operations, or introduce an explicit Noop variant that results in no response/no-op execution.

Copilot · 2026-02-17T16:48:34Z

crates/common/types/src/kv_transaction/convert.rs

+impl CompareOperator {
+    fn from_condition_result(expected: i32) -> Self {
+        use crate::ConditionResult;
+
+        match <ConditionResult as num_traits::FromPrimitive>::from_i32(expected) {
+            Some(ConditionResult::Eq) => CompareOperator::Eq,
+            Some(ConditionResult::Gt) => CompareOperator::Gt,
+            Some(ConditionResult::Ge) => CompareOperator::Ge,
+            Some(ConditionResult::Lt) => CompareOperator::Lt,
+            Some(ConditionResult::Le) => CompareOperator::Le,
+            Some(ConditionResult::Ne) => CompareOperator::Ne,
+            None => CompareOperator::Eq,
+        }
+    }
+}


For invalid expected values, legacy code logged a warning and treated the condition as false. Defaulting to CompareOperator::Eq can incorrectly make a previously-false condition pass, changing transaction semantics. To preserve backward compatibility, treat invalid operators as always-false (e.g., return an Option<CompareOperator> and have the caller construct an always-false condition, or add an explicit Invalid operator handled as false).

Copilot · 2026-02-17T16:48:34Z

crates/server/service/src/meta_node/meta_node.rs

+            Err(e) => return Err(Status::internal(e.to_string())),
+        };
+
+        let reply: KvTransactionReply = applied.try_into().expect("expect KvTransactionReply");


Using expect() in the gRPC request path can panic the server process if AppliedState is not the expected variant (e.g., a regression in applier wiring). Prefer converting with error handling and returning Status::internal(...) on mismatch, so a single bad state does not crash the meta service.

Suggested change

let reply: KvTransactionReply = applied.try_into().expect("expect KvTransactionReply");

let reply: KvTransactionReply = match applied.try_into() {

Ok(reply) => reply,

Err(e) => {

return Err(Status::internal(format!(

"failed to convert AppliedState to KvTransactionReply: {}",

e

)));

}

};

Copilot · 2026-02-17T16:48:35Z

crates/server/service/src/meta_node/meta_handle.rs

+                    let kv_reply: pb::KvTransactionReply =
+                        applied_state.try_into().expect("expect KvTransactionReply");
+                    let txn_reply: TxnReply = kv_reply.into_txn_reply(&txn);


Same concern as server-side: expect() here can panic the handler thread on an unexpected AppliedState, turning a recoverable server response mismatch into a crash. Consider mapping the conversion failure into a Status::internal(...) (or whatever error type is used in this forwarding path) and include enough context to debug the unexpected applied state.

crates/server/raft-store/src/applier/mod.rs

crates/common/types/proto/meta.proto

crates/common/types/src/proto_ext/kv_transaction_reply_ext.rs

xp-trumpet

@xp-trumpet partially reviewed 24 files.
Reviewable status: 318 of 325 files reviewed, 7 unresolved discussions (waiting on drmingdrmer).

…re flag) The existing `Transaction` endpoint stores protobuf `TxnRequest` directly in the raft log, coupling wire format to persistence. Any proto schema change risks breaking raft log deserialization. The new `KvTransaction` endpoint introduces a three-layer architecture: protobuf transport types for the wire, Rust-native serde types (`kv_transaction::Transaction`) for raft log storage, and conversions between them. This lets the proto schema evolve freely without affecting stored data. The new API uses a cleaner branch-based model where each branch has an optional predicate and a list of operations, evaluated in order. This replaces the legacy dual execution paths, stringly-typed `execution_path`, and misleading `success` field of `TxnRequest`. The server now exposes the `KvTransaction` RPC endpoint, but the client does not use it by default — `ClientHandle::transaction_v2()` is gated behind the `transaction-v2` feature flag and only enabled in tests. Production client code continues to use the old `Transaction` API. The new endpoint will be adopted on the client side in a future change once the server has been deployed widely enough. The applier is rewritten to operate on `kv_transaction::*` types. The old `Cmd::Transaction(TxnRequest)` path converts to `kv_transaction::Transaction` internally and delegates to the same code. Both paths return `AppliedState::KvTxnReply(pb::KvTransactionReply)`. Changes: - Add `kv_transaction` module with `Transaction`, `Branch`, `Predicate`, `Condition`, `Operation`, `Operand`, `CompareOperator` storage types - Add `Cmd::KvTransaction(Transaction)` variant - Replace `AppliedState::TxnReply` with `AppliedState::KvTxnReply` - Add `KvTransaction` RPC using redirect pattern (not forward), returning leader endpoint in gRPC metadata for client-side redirect - Gate `ClientHandle::transaction_v2()` behind `transaction-v2` feature flag - Register `Feature::KvTransaction` at version 260217.0.0

Copilot

Pull request overview

Copilot reviewed 33 out of 325 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/common/types/src/kv_transaction/operation.rs

xp-trumpet

@xp-trumpet partially reviewed 8 files and all commit messages.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on drmingdrmer).

Zhang Yanpo added 2 commits February 17, 2026 11:31

chore: Bump ver: 260217.0.0

9fffb30

Copilot AI review requested due to automatic review settings February 17, 2026 15:43

chatgpt-codex-connector bot reviewed Feb 17, 2026

View reviewed changes

crates/server/service/src/meta_node/meta_node.rs Outdated Show resolved Hide resolved

crates/common/types/src/kv_transaction/convert.rs Outdated Show resolved Hide resolved

crates/common/types/src/kv_transaction/convert.rs Outdated Show resolved Hide resolved

xp-trumpet previously approved these changes Feb 17, 2026

View reviewed changes

Copilot AI reviewed Feb 17, 2026

View reviewed changes

drmingdrmer dismissed xp-trumpet’s stale review via fe90657 February 17, 2026 16:39

drmingdrmer force-pushed the 019-xx branch from 877839b to fe90657 Compare February 17, 2026 16:39

Copilot AI review requested due to automatic review settings February 17, 2026 16:42

drmingdrmer force-pushed the 019-xx branch from fe90657 to 8409133 Compare February 17, 2026 16:42

xp-trumpet previously approved these changes Feb 17, 2026

View reviewed changes

Copilot AI reviewed Feb 17, 2026

View reviewed changes

drmingdrmer dismissed xp-trumpet’s stale review via 389aa2f February 17, 2026 16:51

drmingdrmer force-pushed the 019-xx branch from 8409133 to 389aa2f Compare February 17, 2026 16:51

drmingdrmer enabled auto-merge February 17, 2026 17:01

xp-trumpet previously approved these changes Feb 17, 2026

View reviewed changes

drmingdrmer added this pull request to the merge queue Feb 17, 2026

drmingdrmer removed this pull request from the merge queue due to a manual request Feb 17, 2026

Copilot AI review requested due to automatic review settings February 17, 2026 17:02

drmingdrmer dismissed xp-trumpet’s stale review via 55b6c5a February 17, 2026 17:02

drmingdrmer force-pushed the 019-xx branch from 389aa2f to 55b6c5a Compare February 17, 2026 17:02

Copilot AI reviewed Feb 17, 2026

View reviewed changes

crates/common/types/src/kv_transaction/operation.rs Show resolved Hide resolved

xp-trumpet approved these changes Feb 17, 2026

View reviewed changes

drmingdrmer added this pull request to the merge queue Feb 17, 2026

Merged via the queue into databendlabs:main with commit a84adc3 Feb 17, 2026
3 checks passed

drmingdrmer deleted the 019-xx branch February 17, 2026 17:10

-        let reply: KvTransactionReply = applied.try_into().expect("expect KvTransactionReply");
+        let reply: KvTransactionReply = match applied.try_into() {
+            Ok(reply) => reply,
+            Err(e) => {
+                return Err(Status::internal(format!(
+                    "failed to convert AppliedState to KvTransactionReply: {}",
+                    e
+                )));
+            }
+        };

Conversation

drmingdrmer commented Feb 17, 2026

Changelog

feat: add server-side KvTransaction gRPC API (client gated by feature flag)

chore: Bump ver: 260217.0.0

refactor: reorganize crates into common/client/server categories

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xp-trumpet left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

xp-trumpet left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xp-trumpet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

xp-trumpet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: add server-side `KvTransaction` gRPC API (client gated by feature flag)