Implement server_stat command based on the new metrics framework #2382
Description
The server_stat command in Pegasus shell is used to list several important metrics of a meta or replica server, allowing system administrators to check the current status of the node.
>>> server_stat
COMMAND: server-stat
CALL [meta-server] [192.168.0.1:34601] succeed:
replica*app.pegasus*manual.compact.enqueue.count=not_found, replica*app.pegasus*manual.compact.running.count=not_found, replica*app.pegasus*rdb.block_cache.memory_usage=not_found, replica*eon.replica_stub*closing.replica(Count)=not_found, replica*eon.replica_stub*disk.available.max.ratio=not_found, replica*eon.replica_stub*disk.available.min.ratio=not_found, replica*eon.replica_stub*disk.available.total.ratio=not_found, replica*eon.replica_stub*disk.capacity.total(MB)=not_found, replica*eon.replica_stub*opening.replica(Count)=not_found, replica*eon.replica_stub*replica(Count)=not_found, replica*eon.replica_stub*replicas.commit.qps=not_found, replica*eon.replica_stub*replicas.learning.count=not_found, replica*eon.replica_stub*shared.log.size(MB)=not_found, replica*server*memused.res(MB)=not_found, replica*server*memused.virt(MB)=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.qps=not_found, zion*profiler*RPC_RRDB_RRDB_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_GET.qps=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_GET.qps=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_PUT.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_PUT.qps=not_found, zion*profiler*RPC_RRDB_RRDB_PUT.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_PUT.qps=not_found
CALL [meta-server] [192.168.0.2:34601] succeed:
replica*app.pegasus*manual.compact.enqueue.count=not_found, replica*app.pegasus*manual.compact.running.count=not_found, replica*app.pegasus*rdb.block_cache.memory_usage=not_found, replica*eon.replica_stub*closing.replica(Count)=not_found, replica*eon.replica_stub*disk.available.max.ratio=not_found, replica*eon.replica_stub*disk.available.min.ratio=not_found, replica*eon.replica_stub*disk.available.total.ratio=not_found, replica*eon.replica_stub*disk.capacity.total(MB)=not_found, replica*eon.replica_stub*opening.replica(Count)=not_found, replica*eon.replica_stub*replica(Count)=not_found, replica*eon.replica_stub*replicas.commit.qps=not_found, replica*eon.replica_stub*replicas.learning.count=not_found, replica*eon.replica_stub*shared.log.size(MB)=not_found, replica*server*memused.res(MB)=not_found, replica*server*memused.virt(MB)=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.qps=not_found, zion*profiler*RPC_RRDB_RRDB_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_GET.qps=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_GET.qps=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_PUT.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_PUT.qps=not_found, zion*profiler*RPC_RRDB_RRDB_PUT.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_PUT.qps=not_found
CALL [meta-server] [192.168.0.3:34601] succeed:
memused_res(MB)=36, memused_virt(MB)=622, replica*app.pegasus*manual.compact.enqueue.count=not_found, replica*app.pegasus*manual.compact.running.count=not_found, replica*app.pegasus*rdb.block_cache.memory_usage=not_found, replica*eon.replica_stub*closing.replica(Count)=not_found, replica*eon.replica_stub*disk.available.max.ratio=not_found, replica*eon.replica_stub*disk.available.min.ratio=not_found, replica*eon.replica_stub*disk.available.total.ratio=not_found, replica*eon.replica_stub*disk.capacity.total(MB)=not_found, replica*eon.replica_stub*opening.replica(Count)=not_found, replica*eon.replica_stub*replica(Count)=not_found, replica*eon.replica_stub*replicas.commit.qps=not_found, replica*eon.replica_stub*replicas.learning.count=not_found, replica*eon.replica_stub*shared.log.size(MB)=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.qps=not_found, zion*profiler*RPC_RRDB_RRDB_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_GET.qps=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_GET.qps=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_PUT.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_MULTI_PUT.qps=not_found, zion*profiler*RPC_RRDB_RRDB_PUT.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_PUT.qps=not_found
CALL [replica-server] [192.168.0.1:34801] succeed:
manual_compact_enqueue_count=0, manual_compact_running_count=0, rdb_block_cache_memory_usage=140318640, closing_replica_count=0, disk_available_max_ratio=51, disk_available_min_ratio=51, disk_available_total_ratio=51, disk_capacity_total(MB)=2558750, opening_replica_count=0, serving_replica_count=199, commit_throughput=0, learning_count=0, shared_log_size(MB)=0, memused_res(MB)=1216, memused_virt(MB)=3469, get_p99(ns)=667825, get_qps=0, multi_get_p99(ns)=642897, multi_get_qps=0, multi_put_p99(ns)=0, multi_put_qps=0, put_p99(ns)=1630531, put_qps=0, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.qps=not_found
CALL [replica-server] [192.168.0.2:34801] succeed:
manual_compact_enqueue_count=0, manual_compact_running_count=0, rdb_block_cache_memory_usage=151393632, closing_replica_count=0, disk_available_max_ratio=52, disk_available_min_ratio=52, disk_available_total_ratio=52, disk_capacity_total(MB)=2558750, opening_replica_count=0, serving_replica_count=199, commit_throughput=0, learning_count=0, shared_log_size(MB)=0, memused_res(MB)=1279, memused_virt(MB)=3320, get_p99(ns)=458820, get_qps=0, multi_get_p99(ns)=749831, multi_get_qps=0, multi_put_p99(ns)=719222, multi_put_qps=0, put_p99(ns)=3358819, put_qps=0, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.qps=not_found
CALL [replica-server] [192.168.0.3:34801] succeed:
manual_compact_enqueue_count=0, manual_compact_running_count=0, rdb_block_cache_memory_usage=139694624, closing_replica_count=0, disk_available_max_ratio=51, disk_available_min_ratio=51, disk_available_total_ratio=51, disk_capacity_total(MB)=2558750, opening_replica_count=0, serving_replica_count=199, commit_throughput=0, learning_count=0, shared_log_size(MB)=0, memused_res(MB)=1291, memused_virt(MB)=3325, get_p99(ns)=1047973, get_qps=0, multi_get_p99(ns)=1169620, multi_get_qps=0, multi_put_p99(ns)=762679, multi_put_qps=0, put_p99(ns)=8026349, put_qps=0, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.qps=not_found
CALL [replica-server] [192.168.0.4:34801] succeed:
manual_compact_enqueue_count=0, manual_compact_running_count=0, rdb_block_cache_memory_usage=139586208, closing_replica_count=0, disk_available_max_ratio=48, disk_available_min_ratio=48, disk_available_total_ratio=48, disk_capacity_total(MB)=2558750, opening_replica_count=0, serving_replica_count=195, commit_throughput=0, learning_count=0, shared_log_size(MB)=0, memused_res(MB)=1262, memused_virt(MB)=3295, get_p99(ns)=1269360, get_qps=0, multi_get_p99(ns)=1015725, multi_get_qps=0, multi_put_p99(ns)=699263, multi_put_qps=0, put_p99(ns)=1851956, put_qps=0, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.latency.server=not_found, zion*profiler*RPC_RRDB_RRDB_BATCH_GET.qps=not_found
Succeed count: 7
Failed count: 0
In Pegasus 2.5 and earlier, the server_stat command retrieves metrics based on perf_counter. Since a new metrics framework was introduced starting from 2.6, the implementation of server_stat needs to be migrated from perf_counter to the new metrics framework.
Task list:
- feat(new_metrics): support server_stat command showing some important server-level metrics (part 1) #2085
- feat(new_metrics): support
server_statcommand showing some important server-level metrics (part 2) #2223 - feat(new_metrics): support
server_statcommand showing some important server-level metrics (part 3) #2385