[feat] support fe/be metrics system table #54221
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
yiguolei
reviewed
Aug 4, 2025
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 33714 ms |
TPC-DS: Total hot run time: 160081 ms |
ClickBench: Total hot run time: 34.91 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
zhaorongsheng
force-pushed
the
feature_support_fe_be_metrics_tables
branch
2 times, most recently
from
875d6d5 to
1cfbc7d
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 33853 ms |
TPC-DS: Total hot run time: 171123 ms |
ClickBench: Total hot run time: 34.89 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
zhaorongsheng
changed the title
yiguolei
previously approved these changes
Dec 18, 2025
yiguolei
force-pushed
the
feature_support_fe_be_metrics_tables
branch
from
1b4f560 to
2b2e7a1
Compare
Contributor
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
zhaorongsheng
force-pushed
the
feature_support_fe_be_metrics_tables
branch
from
2b2e7a1 to
45774dd
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds frontend (FE) and backend (BE) metrics system tables to the information_schema database, enabling SQL-based querying of metrics data from both FE and BE nodes.
Key changes:
- Added new Thrift RPC interface
fetchFrontendMetricsfor retrieving FE metrics - Implemented
ListMetricVisitorto convert metrics into list format for system table consumption - Created schema scanners for both
backend_metricsandfrontend_metricstables
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
| gensrc/thrift/FrontendService.thrift | Adds Thrift request/response structures and RPC method for fetching FE metrics |
| gensrc/thrift/Descriptors.thrift | Adds enum values for BE and FE metrics schema table types |
| fe/fe-core/src/main/java/org/apache/doris/metric/ListMetricVisitor.java | New visitor implementation that converts metrics to list format with JSON-like tags |
| fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java | Implements RPC endpoint to fetch FE metrics using ListMetricVisitor |
| fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java | Defines schema for backend_metrics and frontend_metrics tables |
| fe/fe-core/src/main/java/org/apache/doris/analysis/SchemaTableType.java | Adds enum entries for the new metrics schema tables |
| fe/fe-core/src/main/java/org/apache/doris/planner/BackendPartitionedSchemaScanNode.java | Registers backend_metrics as a backend-partitioned table |
| fe/fe-core/src/test/java/org/apache/doris/metric/MetricsTest.java | Adds test for ListMetricVisitor and updates existing tests to use update() instead of increase() |
| be/src/util/metrics.h | Declares new method to populate metrics into a Block for system table |
| be/src/util/metrics.cpp | Implements metrics extraction and conversion to Block format with JSON tags |
| be/src/exec/schema_scanner/schema_helper.h | Declares helper method for fetching FE metrics via Thrift RPC |
| be/src/exec/schema_scanner/schema_helper.cpp | Implements Thrift RPC call to fetch FE metrics |
| be/src/exec/schema_scanner/schema_frontend_metrics_scanner.h | Header for FE metrics schema scanner |
| be/src/exec/schema_scanner/schema_frontend_metrics_scanner.cpp | Implements scanner that fetches and returns FE metrics from all FE nodes |
| be/src/exec/schema_scanner/schema_backend_metrics_scanner.h | Header for BE metrics schema scanner |
| be/src/exec/schema_scanner/schema_backend_metrics_scanner.cpp | Implements scanner that returns local BE metrics |
| be/src/exec/schema_scanner.cpp | Registers the new BE and FE metrics scanners in the factory |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
fe/fe-core/src/main/java/org/apache/doris/metric/ListMetricVisitor.java
Outdated
Show resolved
Hide resolved
Comment on lines
121
to
135
| threads.getCount(), "\"type\"=\"count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getPeakCount(), "\"type\"=\"peak_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsNewCount(), "\"type\"=\"new_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsRunnableCount(), "\"type\"=\"runnable_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsBlockedCount(), "\"type\"=\"blocked_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsWaitingCount(), "\"type\"=\"waiting_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsTimedWaitingCount(), "\"type\"=\"timed_waiting_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsTerminatedCount(), "\"type\"=\"terminated_count\"}")); |
There was a problem hiding this comment.
Missing opening brace in the tag string. The tag should be "{"type"="count"}" but it's currently ""type"="count"}", which is missing the opening brace. This inconsistency with other tag strings in this file will cause malformed JSON-like tags.
Suggested change
| threads.getCount(), "\"type\"=\"count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getPeakCount(), "\"type\"=\"peak_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsNewCount(), "\"type\"=\"new_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsRunnableCount(), "\"type\"=\"runnable_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsBlockedCount(), "\"type\"=\"blocked_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsWaitingCount(), "\"type\"=\"waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTimedWaitingCount(), "\"type\"=\"timed_waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTerminatedCount(), "\"type\"=\"terminated_count\"}")); | |
| threads.getCount(), "{\"type\"=\"count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getPeakCount(), "{\"type\"=\"peak_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsNewCount(), "{\"type\"=\"new_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsRunnableCount(), "{\"type\"=\"runnable_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsBlockedCount(), "{\"type\"=\"blocked_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsWaitingCount(), "{\"type\"=\"waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTimedWaitingCount(), "{\"type\"=\"timed_waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTerminatedCount(), "{\"type\"=\"terminated_count\"}")); |
Comment on lines
121
to
135
| threads.getCount(), "\"type\"=\"count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getPeakCount(), "\"type\"=\"peak_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsNewCount(), "\"type\"=\"new_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsRunnableCount(), "\"type\"=\"runnable_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsBlockedCount(), "\"type\"=\"blocked_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsWaitingCount(), "\"type\"=\"waiting_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsTimedWaitingCount(), "\"type\"=\"timed_waiting_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsTerminatedCount(), "\"type\"=\"terminated_count\"}")); |
There was a problem hiding this comment.
Missing opening brace in the tag string. The tag should be "{"type"="peak_count"}" but it's currently ""type"="peak_count"}", which is missing the opening brace. This inconsistency with other tag strings in this file will cause malformed JSON-like tags.
Suggested change
| threads.getCount(), "\"type\"=\"count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getPeakCount(), "\"type\"=\"peak_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsNewCount(), "\"type\"=\"new_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsRunnableCount(), "\"type\"=\"runnable_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsBlockedCount(), "\"type\"=\"blocked_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsWaitingCount(), "\"type\"=\"waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTimedWaitingCount(), "\"type\"=\"timed_waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTerminatedCount(), "\"type\"=\"terminated_count\"}")); | |
| threads.getCount(), "{\"type\"=\"count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getPeakCount(), "{\"type\"=\"peak_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsNewCount(), "{\"type\"=\"new_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsRunnableCount(), "{\"type\"=\"runnable_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsBlockedCount(), "{\"type\"=\"blocked_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsWaitingCount(), "{\"type\"=\"waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTimedWaitingCount(), "{\"type\"=\"timed_waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTerminatedCount(), "{\"type\"=\"terminated_count\"}")); |
Comment on lines
121
to
135
| threads.getCount(), "\"type\"=\"count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getPeakCount(), "\"type\"=\"peak_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsNewCount(), "\"type\"=\"new_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsRunnableCount(), "\"type\"=\"runnable_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsBlockedCount(), "\"type\"=\"blocked_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsWaitingCount(), "\"type\"=\"waiting_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsTimedWaitingCount(), "\"type\"=\"timed_waiting_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsTerminatedCount(), "\"type\"=\"terminated_count\"}")); |
There was a problem hiding this comment.
Missing opening brace in the tag string. The tag should be "{"type"="new_count"}" but it's currently ""type"="new_count"}", which is missing the opening brace. This inconsistency with other tag strings in this file will cause malformed JSON-like tags.
Suggested change
| threads.getCount(), "\"type\"=\"count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getPeakCount(), "\"type\"=\"peak_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsNewCount(), "\"type\"=\"new_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsRunnableCount(), "\"type\"=\"runnable_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsBlockedCount(), "\"type\"=\"blocked_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsWaitingCount(), "\"type\"=\"waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTimedWaitingCount(), "\"type\"=\"timed_waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTerminatedCount(), "\"type\"=\"terminated_count\"}")); | |
| threads.getCount(), "{\"type\"=\"count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getPeakCount(), "{\"type\"=\"peak_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsNewCount(), "{\"type\"=\"new_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsRunnableCount(), "{\"type\"=\"runnable_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsBlockedCount(), "{\"type\"=\"blocked_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsWaitingCount(), "{\"type\"=\"waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTimedWaitingCount(), "{\"type\"=\"timed_waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTerminatedCount(), "{\"type\"=\"terminated_count\"}")); |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsNewCount(), "\"type\"=\"new_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsRunnableCount(), "\"type\"=\"runnable_count\"}")); |
There was a problem hiding this comment.
Missing opening brace in the tag string. The tag should be "{"type"="runnable_count"}" but it's currently ""type"="runnable_count"}", which is missing the opening brace. This inconsistency with other tag strings in this file will cause malformed JSON-like tags.
Comment on lines
121
to
135
| threads.getCount(), "\"type\"=\"count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getPeakCount(), "\"type\"=\"peak_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsNewCount(), "\"type\"=\"new_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsRunnableCount(), "\"type\"=\"runnable_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsBlockedCount(), "\"type\"=\"blocked_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsWaitingCount(), "\"type\"=\"waiting_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsTimedWaitingCount(), "\"type\"=\"timed_waiting_count\"}")); | ||
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | ||
| threads.getThreadsTerminatedCount(), "\"type\"=\"terminated_count\"}")); |
There was a problem hiding this comment.
Missing opening brace in the tag string. The tag should be "{"type"="blocked_count"}" but it's currently ""type"="blocked_count"}", which is missing the opening brace. This inconsistency with other tag strings in this file will cause malformed JSON-like tags.
Suggested change
| threads.getCount(), "\"type\"=\"count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getPeakCount(), "\"type\"=\"peak_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsNewCount(), "\"type\"=\"new_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsRunnableCount(), "\"type\"=\"runnable_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsBlockedCount(), "\"type\"=\"blocked_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsWaitingCount(), "\"type\"=\"waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTimedWaitingCount(), "\"type\"=\"timed_waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTerminatedCount(), "\"type\"=\"terminated_count\"}")); | |
| threads.getCount(), "{\"type\"=\"count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getPeakCount(), "{\"type\"=\"peak_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsNewCount(), "{\"type\"=\"new_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsRunnableCount(), "{\"type\"=\"runnable_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsBlockedCount(), "{\"type\"=\"blocked_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsWaitingCount(), "{\"type\"=\"waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTimedWaitingCount(), "{\"type\"=\"timed_waiting_count\"}")); | |
| this.metricsList.add(newGaugeSubList(JVM_THREAD, | |
| threads.getThreadsTerminatedCount(), "{\"type\"=\"terminated_count\"}")); |
Comment on lines
+191
to
+192
| this.metricsList.add(newHistogramSubList(fullName + "_count", | ||
| histogram.getCount(), "{\"quantile\"=\"0.75\"" + delimiter + fullTag + "}")); |
There was a problem hiding this comment.
The tag used for the "_count" metric should not include the "quantile" label, as count is not associated with a quantile. Additionally, using "{"quantile"="0.75"" is misleading since this represents the total count, not the 75th percentile. Consider removing the quantile label entirely or using an empty tag set for the count metric.
…itor.java Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
zhaorongsheng
force-pushed
the
feature_support_fe_be_metrics_tables
branch
from
6cb6e30 to
8a957dd
Compare
Contributor
|
run buildall |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
add fe/be metrics system table in db
information_schemaRelease note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)