Skip to content

DocDB: Add the lag column to list_all_masters yb-admin output#30349

Open
keisku wants to merge 1 commit intoyugabyte:masterfrom
keisku:issue-28675
Open

DocDB: Add the lag column to list_all_masters yb-admin output#30349
keisku wants to merge 1 commit intoyugabyte:masterfrom
keisku:issue-28675

Conversation

@keisku
Copy link
Contributor

@keisku keisku commented Feb 17, 2026

Closes #28675


There is a big number on in the Heartbeat Lag clumn (this PR added) since yb-master on the node3 stopped.

./bin/yb-ctl status
----------------------------------------------------------------------------------------------------
| Node Count: 3 | Replication Factor: 3                                                            |
----------------------------------------------------------------------------------------------------
| JDBC                : jdbc:postgresql://127.0.0.1:5433/yugabyte                                  |
| YSQL Shell          : bin/ysqlsh                                                                 |
| YCQL Shell          : bin/ycqlsh                                                                 |
| YEDIS Shell         : bin/redis-cli                                                              |
| Web UI              : http://127.0.0.1:7000/                                                     |
| Cluster Data        : /Users/keisukeumegaki/yugabyte-data                                        |
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
| Node 1: yb-tserver (pid 93159), yb-master (pid 58555)                                            |
----------------------------------------------------------------------------------------------------
| JDBC                : jdbc:postgresql://127.0.0.1:5433/yugabyte                                  |
| YSQL Shell          : bin/ysqlsh                                                                 |
| YCQL Shell          : bin/ycqlsh                                                                 |
| YEDIS Shell         : bin/redis-cli                                                              |
| data-dir[0]         : /Users/keisukeumegaki/yugabyte-data/node-1/disk-1/yb-data                  |
| yb-tserver Logs     : /Users/keisukeumegaki/yugabyte-data/node-1/disk-1/yb-data/tserver/logs     |
| yb-master Logs      : /Users/keisukeumegaki/yugabyte-data/node-1/disk-1/yb-data/master/logs      |
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
| Node 2: yb-tserver (Stopped), yb-master (pid 91341)                                              |
----------------------------------------------------------------------------------------------------
| JDBC                : jdbc:postgresql://127.0.0.2:5433/yugabyte                                  |
| YSQL Shell          : bin/ysqlsh -h 127.0.0.2                                                    |
| YCQL Shell          : bin/ycqlsh 127.0.0.2                                                       |
| YEDIS Shell         : bin/redis-cli -h 127.0.0.2                                                 |
| data-dir[0]         : /Users/keisukeumegaki/yugabyte-data/node-2/disk-1/yb-data                  |
| yb-tserver Logs     : /Users/keisukeumegaki/yugabyte-data/node-2/disk-1/yb-data/tserver/logs     |
| yb-master Logs      : /Users/keisukeumegaki/yugabyte-data/node-2/disk-1/yb-data/master/logs      |
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
| Node 3: yb-tserver (pid 9970), yb-master (Stopped)                                               |
----------------------------------------------------------------------------------------------------
| JDBC                : jdbc:postgresql://127.0.0.3:5433/yugabyte                                  |
| YSQL Shell          : bin/ysqlsh -h 127.0.0.3                                                    |
| YCQL Shell          : bin/ycqlsh 127.0.0.3                                                       |
| YEDIS Shell         : bin/redis-cli -h 127.0.0.3                                                 |
| data-dir[0]         : /Users/keisukeumegaki/yugabyte-data/node-3/disk-1/yb-data                  |
| yb-tserver Logs     : /Users/keisukeumegaki/yugabyte-data/node-3/disk-1/yb-data/tserver/logs     |
| yb-master Logs      : /Users/keisukeumegaki/yugabyte-data/node-3/disk-1/yb-data/master/logs      |
----------------------------------------------------------------------------------------------------

./build/latest/bin/yb-admin --master_addresses 127.0.0.1:7100,127.0.0.2:7100,127.0.0.3:7100 list_all_masters
Master UUID                             RPC Host/Port           State           Role            Broadcast Host/Port     Heartbeat Lag
9c26c847c2c24691a4ef839ccbb82c15        127.0.0.1:7100          ALIVE           LEADER          N/A                     0ms
577f32bb523847249e338b3f89981e1a        127.0.0.2:7100          ALIVE           FOLLOWER        N/A                     178ms
8214242e58ac478595d15d9f5244dc6a        127.0.0.3:7100          NETWORK_ERROR   UNKNOWN         N/A                     678092ms

Note

Low Risk
Change is limited to CLI output formatting and best-effort RPC data retrieval; primary risk is minor output/compatibility impact for scripts parsing list_all_masters.

Overview
Adds a Heartbeat Lag column to yb-admin list_all_masters output by best-effort querying the leader via GetMasterHeartbeatDelays, showing 0ms for the leader, per-follower lag in ms when available, and N/A otherwise.

Updates multi-master integration tests to accommodate the new non-deterministic lag values by normalizing \d+ms before comparing outputs, and adds a new test asserting the column/header and basic lag formatting are present.

Written by Cursor Bugbot for commit 3234178. This will update automatically on new commits. Configure here.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

This is the final PR Bugbot will review for you during this billing cycle

Your free Bugbot reviews will reset on February 25

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

<< "Leader should show 0ms lag: " << line;
} else if (line.find("FOLLOWER") != std::string::npos) {
ASSERT_NE(line.find("ms"), std::string::npos)
<< "Follower should show lag in ms: " << line;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test assumes followers always have lag data

Medium Severity

The test requires all follower lines to contain ms, but the implementation outputs N/A for followers when has_lag_data is true but their UUID isn't found in the heartbeat map (line 1509 of yb-admin_client.cc). A follower missing from the heartbeat response would fail this assertion despite being valid output from the implementation.

Fix in Cursor Fix in Web

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DocDB] Add the lag column to list_all_masters yb-admin output

1 participant