Skip to content

[ISSUE #6393]🚀Implement HAStatus Command for High Availability Runtime Monitoring#6481

Merged
mxsm merged 1 commit intomxsm:mainfrom
WaterWhisperer:feat-6393
Feb 24, 2026
Merged

[ISSUE #6393]🚀Implement HAStatus Command for High Availability Runtime Monitoring#6481
mxsm merged 1 commit intomxsm:mainfrom
WaterWhisperer:feat-6393

Conversation

@WaterWhisperer
Copy link
Contributor

@WaterWhisperer WaterWhisperer commented Feb 23, 2026

Which Issue(s) This PR Fixes(Closes)

Brief Description

How Did You Test This Change?

Summary by CodeRabbit

  • New Features
    • Added broker high-availability (HA) status retrieval capability to query runtime HA information
    • New admin tool command to fetch and display detailed HA status data with support for individual brokers or cluster-wide queries
    • Added optional interval-based periodic status monitoring for continuous HA metrics tracking

@rocketmq-rust-robot rocketmq-rust-robot added the feature🚀 Suggest an idea for this project. label Feb 23, 2026
@rocketmq-rust-bot
Copy link
Collaborator

🔊@WaterWhisperer 🚀Thanks for your contribution🎉!

💡CodeRabbit(AI) will review your code first🔥!

Note

🚨The code review suggestions from CodeRabbit are to be used as a reference only, and the PR submitter can decide whether to make changes based on their own judgment. Ultimately, the project management personnel will conduct the final code review💥.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 23, 2026

Walkthrough

This pull request implements the haStatus command for the RocketMQ admin tool, enabling operators to retrieve and display High Availability runtime status information from brokers. Changes span the broker, client, and admin tool layers to support querying master-slave replication metrics and HA health across single brokers or entire clusters.

Changes

Cohort / File(s) Summary
Broker HA Handler
rocketmq-broker/src/processor/admin_broker_processor.rs, rocketmq-broker/src/processor/admin_broker_processor/get_broker_ha_status_handler.rs
Introduces new GetBrokerHaStatusHandler to handle incoming HA status requests; retrieves message store and HA service, formats runtime info as JSON response, and wires handler into AdminBrokerProcessor.
Client API Layer
rocketmq-client/src/admin/default_mq_admin_ext_impl.rs, rocketmq-client/src/implementation/mq_client_api_impl.rs
Implements functional get_broker_ha_status methods in both DefaultMQAdminExtImpl and MQClientAPIImpl; previously unimplemented stubs now delegate to broker via remoting API and decode HARuntimeInfo responses.
Admin Tool Command
rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/admin/default_mq_admin_ext.rs, rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands.rs, rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/ha_commands.rs
Adds new haStatus command entry and wires HaStatus variant into HACommands enum; updates DefaultMQAdminExt to delegate to implementation layer.
HA Status Sub-Command
rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/ha_commands/ha_status_sub_command.rs
Implements new HAStatusSubCommand supporting broker-specific or cluster-wide HA status queries with optional polling interval; handles master/slave status formatting, transfer speed calculation, and synchronization offset display.

Sequence Diagram

sequenceDiagram
    participant Client as Admin Client
    participant AdminAPI as DefaultMQAdminExt
    participant ClientAPI as MQClientAPIImpl
    participant Broker as AdminBrokerProcessor
    participant Handler as GetBrokerHaStatusHandler
    participant Store as MessageStore & HA Service

    Client->>AdminAPI: get_broker_ha_status(broker_addr)
    AdminAPI->>ClientAPI: get_broker_ha_status(broker_addr, timeout)
    ClientAPI->>Broker: GetBrokerHaStatus request
    Broker->>Handler: get_broker_ha_status(request)
    Handler->>Store: Retrieve MessageStore
    Handler->>Store: Get HA Service
    Handler->>Store: Get max offset & HA runtime info
    Handler->>Handler: Serialize HARuntimeInfo to JSON
    Handler-->>Broker: Response with HA data
    Broker-->>ClientAPI: RemotingCommand response
    ClientAPI->>ClientAPI: Decode HARuntimeInfo from body
    ClientAPI-->>AdminAPI: HARuntimeInfo
    AdminAPI-->>Client: HARuntimeInfo
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

feature🚀, AI review first

Suggested reviewers

  • mxsm
  • TeslaRustor
  • rocketmq-rust-bot

Poem

🐰 Hops through HA land so bright,
Master-slave replication in sight,
Transfer speeds and offsets clear,
Cluster health displayed without fear,
HAStatus commands now take flight! 🚀

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title clearly identifies the main change: implementing the HAStatus command for HA runtime monitoring, which directly aligns with the primary objective of adding HA visibility tooling.
Linked Issues check ✅ Passed The PR implements all core coding requirements from issue #6393: the haStatus command with broker/cluster parameters, HA status retrieval via MQAdminExt, master/slave-specific formatting, and admin tool integration.
Out of Scope Changes check ✅ Passed All changes are directly related to implementing the HAStatus command functionality as specified in issue #6393. No unrelated modifications are present.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In
`@rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/ha_commands/ha_status_sub_command.rs`:
- Around line 31-108: The HAStatusSubCommand currently allows both or neither of
-b/--brokerAddr and -c/--clusterName and silently prints an error instead of
failing; add an ArgGroup to the HAStatusSubCommand struct to require exactly one
of broker_addr and cluster_name (match patterns used in
BrokerStatusSubCommand/GetBrokerConfigSubCommand), remove the manual println
error in inner_exec, and return a RocketMQError (or any appropriate Err from
inner_exec) when neither is provided; update the struct annotation to include
#[group(...)] (or the clap ArgGroup equivalent) referencing "broker_addr" and
"cluster_name" and adjust inner_exec and execute error handling so the command
fails fast when arg validation fails.

ℹ️ Review info

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 609817f and 2f9cac4.

📒 Files selected for processing (8)
  • rocketmq-broker/src/processor/admin_broker_processor.rs
  • rocketmq-broker/src/processor/admin_broker_processor/get_broker_ha_status_handler.rs
  • rocketmq-client/src/admin/default_mq_admin_ext_impl.rs
  • rocketmq-client/src/implementation/mq_client_api_impl.rs
  • rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/admin/default_mq_admin_ext.rs
  • rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands.rs
  • rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/ha_commands.rs
  • rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/ha_commands/ha_status_sub_command.rs

Comment on lines +31 to +108
#[derive(Debug, Clone, Parser)]
pub struct HAStatusSubCommand {
#[arg(
short = 'b',
long = "brokerAddr",
value_name = "HOST:PORT",
required = false,
help = "which broker to fetch"
)]
broker_addr: Option<String>,

#[arg(
short = 'c',
long = "clusterName",
value_name = "CLUSTER",
required = false,
help = "which cluster"
)]
cluster_name: Option<String>,

#[arg(
short = 'i',
long = "interval",
value_name = "SECONDS",
required = false,
help = "the interval(second) of get info"
)]
interval: Option<u64>,
}

impl CommandExecute for HAStatusSubCommand {
async fn execute(&self, _rpc_hook: Option<Arc<dyn RPCHook>>) -> RocketMQResult<()> {
let mut default_mqadmin_ext = DefaultMQAdminExt::new();
default_mqadmin_ext
.client_config_mut()
.set_instance_name(get_current_millis().to_string().into());

let operation_result = async {
MQAdminExt::start(&mut default_mqadmin_ext).await.map_err(|e| {
RocketMQError::Internal(format!("HAStatusSubCommand: Failed to start MQAdminExt: {}", e))
})?;

if let Some(interval) = self.interval {
let flush_second = if interval > 0 { interval } else { 3 };
loop {
self.inner_exec(&default_mqadmin_ext).await?;
tokio::time::sleep(tokio::time::Duration::from_secs(flush_second)).await;
}
} else {
self.inner_exec(&default_mqadmin_ext).await?;
}

Ok(())
}
.await;

MQAdminExt::shutdown(&mut default_mqadmin_ext).await;
operation_result
}
}

impl HAStatusSubCommand {
async fn inner_exec(&self, default_mqadmin_ext: &DefaultMQAdminExt) -> RocketMQResult<()> {
if let Some(ref broker_addr) = self.broker_addr {
Self::print_status(broker_addr.trim(), default_mqadmin_ext).await?;
} else if let Some(ref cluster_name) = self.cluster_name {
let cluster_info = default_mqadmin_ext.examine_broker_cluster_info().await.map_err(|e| {
RocketMQError::Internal(format!("HAStatusSubCommand: Failed to get cluster info: {}", e))
})?;
let master_addrs = CommandUtil::fetch_master_addr_by_cluster_name(&cluster_info, cluster_name.trim())?;
for addr in master_addrs {
Self::print_status(&addr, default_mqadmin_ext).await?;
}
} else {
println!("Error: either -b (brokerAddr) or -c (clusterName) must be specified");
}

Ok(())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find Cargo.toml files and check clap version
fd -n "Cargo.toml" -x grep -l "clap" {} \; | head -5

Repository: mxsm/rocketmq-rust

Length of output: 292


🏁 Script executed:

#!/bin/bash
# Read the main Cargo.toml to find clap version
find . -name "Cargo.toml" -type f | head -1 | xargs cat | grep -A 2 "clap"

Repository: mxsm/rocketmq-rust

Length of output: 44


🏁 Script executed:

#!/bin/bash
# Get the specific file and read lines 31-108
cat -n "rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/ha_commands/ha_status_sub_command.rs" | sed -n '31,108p'

Repository: mxsm/rocketmq-rust

Length of output: 3337


🏁 Script executed:

#!/bin/bash
# Check if there's a Cargo.lock and look for clap version there
if [ -f "Cargo.lock" ]; then
  grep -A 2 "name = \"clap\"" Cargo.lock | head -10
fi

Repository: mxsm/rocketmq-rust

Length of output: 159


🏁 Script executed:

#!/bin/bash
# Search for ArgGroup usage in the codebase
rg "ArgGroup" --type rust

Repository: mxsm/rocketmq-rust

Length of output: 6687


🏁 Script executed:

#!/bin/bash
# Look for other command implementations to see patterns
fd "sub_command.rs" -x grep -l "impl CommandExecute" {} \; | head -3

Repository: mxsm/rocketmq-rust

Length of output: 375


🏁 Script executed:

#!/bin/bash
# Check the main Cargo.toml for the clap dependency definition
cat rocketmq-tools/rocketmq-admin/rocketmq-admin-core/Cargo.toml | grep -A 3 "clap"

Repository: mxsm/rocketmq-rust

Length of output: 259


🏁 Script executed:

#!/bin/bash
# Check an example ArgGroup implementation to see the exact pattern
cat rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/broker_commands/broker_status_sub_command.rs | head -100

Repository: mxsm/rocketmq-rust

Length of output: 3847


🏁 Script executed:

#!/bin/bash
# Also check get_broker_config_sub_command.rs for comparison
cat rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/broker_commands/get_broker_config_sub_command.rs | head -100

Repository: mxsm/rocketmq-rust

Length of output: 3433


🏁 Script executed:

#!/bin/bash
# Check the inner_exec error handling in broker_status_sub_command.rs
grep -A 20 "async fn inner_exec" rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/broker_commands/broker_status_sub_command.rs

Repository: mxsm/rocketmq-rust

Length of output: 44


Fix argument validation: use ArgGroup to enforce one-of-two requirement for -b or -c.

The struct accepts both flags (silently preferring -b when both provided) and prints an error without failing when neither is set. This violates the CLI contract. Use an ArgGroup like other similar commands in the codebase (e.g., BrokerStatusSubCommand, GetBrokerConfigSubCommand) and return a proper error instead of printing and returning Ok(()).

✅ Suggested fix
+use clap::ArgGroup;

 #[derive(Debug, Clone, Parser)]
+#[command(group(
+    clap::ArgGroup::new("target")
+        .required(true)
+        .args(&["broker_addr", "cluster_name"])
+))]
 pub struct HAStatusSubCommand {
     #[arg(
         short = 'b',
         long = "brokerAddr",
         value_name = "HOST:PORT",
         required = false,
         help = "which broker to fetch"
     )]
     broker_addr: Option<String>,

     #[arg(
         short = 'c',
         long = "clusterName",
         value_name = "CLUSTER",
         required = false,
         help = "which cluster"
     )]
     cluster_name: Option<String>,
@@
         } else {
-            println!("Error: either -b (brokerAddr) or -c (clusterName) must be specified");
+            return Err(RocketMQError::IllegalArgument(
+                "Either -b (brokerAddr) or -c (clusterName) must be specified".into(),
+            ));
         }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
#[derive(Debug, Clone, Parser)]
pub struct HAStatusSubCommand {
#[arg(
short = 'b',
long = "brokerAddr",
value_name = "HOST:PORT",
required = false,
help = "which broker to fetch"
)]
broker_addr: Option<String>,
#[arg(
short = 'c',
long = "clusterName",
value_name = "CLUSTER",
required = false,
help = "which cluster"
)]
cluster_name: Option<String>,
#[arg(
short = 'i',
long = "interval",
value_name = "SECONDS",
required = false,
help = "the interval(second) of get info"
)]
interval: Option<u64>,
}
impl CommandExecute for HAStatusSubCommand {
async fn execute(&self, _rpc_hook: Option<Arc<dyn RPCHook>>) -> RocketMQResult<()> {
let mut default_mqadmin_ext = DefaultMQAdminExt::new();
default_mqadmin_ext
.client_config_mut()
.set_instance_name(get_current_millis().to_string().into());
let operation_result = async {
MQAdminExt::start(&mut default_mqadmin_ext).await.map_err(|e| {
RocketMQError::Internal(format!("HAStatusSubCommand: Failed to start MQAdminExt: {}", e))
})?;
if let Some(interval) = self.interval {
let flush_second = if interval > 0 { interval } else { 3 };
loop {
self.inner_exec(&default_mqadmin_ext).await?;
tokio::time::sleep(tokio::time::Duration::from_secs(flush_second)).await;
}
} else {
self.inner_exec(&default_mqadmin_ext).await?;
}
Ok(())
}
.await;
MQAdminExt::shutdown(&mut default_mqadmin_ext).await;
operation_result
}
}
impl HAStatusSubCommand {
async fn inner_exec(&self, default_mqadmin_ext: &DefaultMQAdminExt) -> RocketMQResult<()> {
if let Some(ref broker_addr) = self.broker_addr {
Self::print_status(broker_addr.trim(), default_mqadmin_ext).await?;
} else if let Some(ref cluster_name) = self.cluster_name {
let cluster_info = default_mqadmin_ext.examine_broker_cluster_info().await.map_err(|e| {
RocketMQError::Internal(format!("HAStatusSubCommand: Failed to get cluster info: {}", e))
})?;
let master_addrs = CommandUtil::fetch_master_addr_by_cluster_name(&cluster_info, cluster_name.trim())?;
for addr in master_addrs {
Self::print_status(&addr, default_mqadmin_ext).await?;
}
} else {
println!("Error: either -b (brokerAddr) or -c (clusterName) must be specified");
}
Ok(())
use clap::ArgGroup;
#[derive(Debug, Clone, Parser)]
#[command(group(
clap::ArgGroup::new("target")
.required(true)
.args(&["broker_addr", "cluster_name"])
))]
pub struct HAStatusSubCommand {
#[arg(
short = 'b',
long = "brokerAddr",
value_name = "HOST:PORT",
required = false,
help = "which broker to fetch"
)]
broker_addr: Option<String>,
#[arg(
short = 'c',
long = "clusterName",
value_name = "CLUSTER",
required = false,
help = "which cluster"
)]
cluster_name: Option<String>,
#[arg(
short = 'i',
long = "interval",
value_name = "SECONDS",
required = false,
help = "the interval(second) of get info"
)]
interval: Option<u64>,
}
impl CommandExecute for HAStatusSubCommand {
async fn execute(&self, _rpc_hook: Option<Arc<dyn RPCHook>>) -> RocketMQResult<()> {
let mut default_mqadmin_ext = DefaultMQAdminExt::new();
default_mqadmin_ext
.client_config_mut()
.set_instance_name(get_current_millis().to_string().into());
let operation_result = async {
MQAdminExt::start(&mut default_mqadmin_ext).await.map_err(|e| {
RocketMQError::Internal(format!("HAStatusSubCommand: Failed to start MQAdminExt: {}", e))
})?;
if let Some(interval) = self.interval {
let flush_second = if interval > 0 { interval } else { 3 };
loop {
self.inner_exec(&default_mqadmin_ext).await?;
tokio::time::sleep(tokio::time::Duration::from_secs(flush_second)).await;
}
} else {
self.inner_exec(&default_mqadmin_ext).await?;
}
Ok(())
}
.await;
MQAdminExt::shutdown(&mut default_mqadmin_ext).await;
operation_result
}
}
impl HAStatusSubCommand {
async fn inner_exec(&self, default_mqadmin_ext: &DefaultMQAdminExt) -> RocketMQResult<()> {
if let Some(ref broker_addr) = self.broker_addr {
Self::print_status(broker_addr.trim(), default_mqadmin_ext).await?;
} else if let Some(ref cluster_name) = self.cluster_name {
let cluster_info = default_mqadmin_ext.examine_broker_cluster_info().await.map_err(|e| {
RocketMQError::Internal(format!("HAStatusSubCommand: Failed to get cluster info: {}", e))
})?;
let master_addrs = CommandUtil::fetch_master_addr_by_cluster_name(&cluster_info, cluster_name.trim())?;
for addr in master_addrs {
Self::print_status(&addr, default_mqadmin_ext).await?;
}
} else {
return Err(RocketMQError::IllegalArgument(
"Either -b (brokerAddr) or -c (clusterName) must be specified".into(),
));
}
Ok(())
}
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@rocketmq-tools/rocketmq-admin/rocketmq-admin-core/src/commands/ha_commands/ha_status_sub_command.rs`
around lines 31 - 108, The HAStatusSubCommand currently allows both or neither
of -b/--brokerAddr and -c/--clusterName and silently prints an error instead of
failing; add an ArgGroup to the HAStatusSubCommand struct to require exactly one
of broker_addr and cluster_name (match patterns used in
BrokerStatusSubCommand/GetBrokerConfigSubCommand), remove the manual println
error in inner_exec, and return a RocketMQError (or any appropriate Err from
inner_exec) when neither is provided; update the struct annotation to include
#[group(...)] (or the clap ArgGroup equivalent) referencing "broker_addr" and
"cluster_name" and adjust inner_exec and execute error handling so the command
fails fast when arg validation fails.

Copy link
Collaborator

@rocketmq-rust-bot rocketmq-rust-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - All CI checks passed ✅

@codecov
Copy link

codecov bot commented Feb 23, 2026

Codecov Report

❌ Patch coverage is 0% with 151 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.19%. Comparing base (f5035ef) to head (2f9cac4).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
.../src/commands/ha_commands/ha_status_sub_command.rs 0.00% 76 Missing ⚠️
...n_broker_processor/get_broker_ha_status_handler.rs 0.00% 36 Missing ⚠️
...mq-client/src/implementation/mq_client_api_impl.rs 0.00% 19 Missing ⚠️
...etmq-client/src/admin/default_mq_admin_ext_impl.rs 0.00% 7 Missing ⚠️
...tmq-broker/src/processor/admin_broker_processor.rs 0.00% 5 Missing ⚠️
...rocketmq-admin/rocketmq-admin-core/src/commands.rs 0.00% 5 Missing ⚠️
...ketmq-admin-core/src/admin/default_mq_admin_ext.rs 0.00% 2 Missing ⚠️
...in/rocketmq-admin-core/src/commands/ha_commands.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6481      +/-   ##
==========================================
- Coverage   42.24%   42.19%   -0.05%     
==========================================
  Files         940      942       +2     
  Lines      131560   131708     +148     
==========================================
- Hits        55579    55577       -2     
- Misses      75981    76131     +150     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mxsm mxsm merged commit b4f7d94 into mxsm:main Feb 24, 2026
10 of 20 checks passed
@rocketmq-rust-bot rocketmq-rust-bot added approved PR has approved and removed ready to review waiting-review waiting review this PR labels Feb 24, 2026
@WaterWhisperer WaterWhisperer deleted the feat-6393 branch February 24, 2026 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI review first Ai review pr first approved PR has approved auto merge feature🚀 Suggest an idea for this project.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature🚀] Implement HAStatus Command for High Availability Runtime Monitoring

4 participants