Commit 9a88328
authored
[enhance](paimon)Doris Paimon Scan Metrics Integration (#59281)
### What problem does this PR solve?
#### Overview
This change pipes Apache Paimon scan metrics directly into Doris query
profiles so operators can inspect per-scan
statistics (manifests, file counts, scan durations, cache hit/miss) from
the FE profile UI. The integration consists
of three pieces:
- Summary Profile Slot - SummaryProfile now includes a Paimon Scan
Metrics entry in the execution summary list so the
FE profile table reserves space for the Paimon telemetry.
- Metric Registry + Reporter - a new PaimonMetricRegistry collects
Paimon MetricGroups, and PaimonScanMetricsReporter
formats ScanMetrics values and appends them to the summary entry
whenever a Paimon scan runs.
- Scan Integration - PaimonScanNode attaches a registry to the TableScan
(when InnerTableScan supports
withMetricsRegistry), plans splits, and reports metrics after planning.
All metrics remain scoped to Paimon scans; other table formats are
untouched and still populate their own
runtime-profile sections as before.
#### Implementation Details
1. SummaryProfile
`fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java`
- Added the constant `PAIMON_SCAN_METRICS = "Paimon Scan Metrics"`.
- Inserted the key into `EXECUTION_SUMMARY_KEYS` (with indentation level
3) so the runtime profile tree displays it
under the scheduling block when present.
- No default text is shown unless metrics are actually reported; the
entry stays N/A otherwise.
2. PaimonMetricRegistry
`fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonMetricRegistry.java`
- Extends `MetricRegistry` and caches `MetricGroup` instances keyed by
`name:table`.
- Uses the `table` tag to disambiguate groups
(`MetricRegistry.KEY_TABLE` is private in Paimon).
- Exposes `getGroup/removeGroup/clear` helpers to manage registry
lifecycle.
3. PaimonScanMetricsReporter
`fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonScanMetricsReporter.java`
- Reads `ScanMetrics.GROUP_NAME` from the registry and appends a
per-scan entry under "Paimon Scan Metrics" in the
SummaryProfile.
- Metrics covered: `last_scan_duration`, `scan_duration`
(count/mean/p95/max), `last_scanned_manifests`,
`last_scan_skipped_table_files`, `last_scan_resulted_table_files`,
`manifest_hit_cache`, `manifest_missed_cache`.
- If the direct lookup fails, falls back to the single ScanMetrics group
in the registry (and skips reporting if
multiple groups exist).
- Cleans up the registry group after reporting.
4. PaimonScanNode
`fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java`
- Creates a `TableScan` via
`newReadBuilder().withFilter().withProjection().newScan()`.
- When scan is an `InnerTableScan`, attaches `PaimonMetricRegistry` with
`withMetricsRegistry`.
- Plans splits, reports metrics, and clears the registry to avoid leaks.
5. Regression test
`regression-test/suites/external_table_p0/paimon/test_paimon_scan_metrics_profile.groovy`
- Adds a Paimon scan metrics profile case that asserts "Paimon Scan
Metrics" appears in the query profile.
#### Example Profile Output
After running a query against `test_paimon_scan_metrics.db1.all_table`,
the FE profile shows:
```
Paimon Scan Metrics:
Table Scan (test_paimon_scan_metrics.db1.all_table):
- last_scan_duration: 7ms
- scan_duration: count=1, mean=7ms, p95=7ms, max=7ms
- last_scanned_manifests: 1
- last_scan_skipped_table_files: 0
- last_scan_resulted_table_files: 3
- manifest_hit_cache: 0
- manifest_missed_cache: 1
```1 parent b1303c3 commit 9a88328
File tree
4 files changed
+243
-3
lines changed- fe/fe-core/src/main/java/org/apache/doris
- common/profile
- datasource/paimon
- profile
- source
4 files changed
+243
-3
lines changedLines changed: 3 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
| 136 | + | |
136 | 137 | | |
137 | 138 | | |
138 | 139 | | |
| |||
166 | 167 | | |
167 | 168 | | |
168 | 169 | | |
| 170 | + | |
169 | 171 | | |
170 | 172 | | |
171 | 173 | | |
| |||
218 | 220 | | |
219 | 221 | | |
220 | 222 | | |
| 223 | + | |
221 | 224 | | |
222 | 225 | | |
223 | 226 | | |
| |||
Lines changed: 72 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/profile/PaimonScanMetricsReporter.java
Lines changed: 152 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
Lines changed: 16 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
| 38 | + | |
37 | 39 | | |
38 | 40 | | |
39 | 41 | | |
| |||
55 | 57 | | |
56 | 58 | | |
57 | 59 | | |
| 60 | + | |
58 | 61 | | |
59 | 62 | | |
| 63 | + | |
60 | 64 | | |
61 | 65 | | |
62 | 66 | | |
| |||
413 | 417 | | |
414 | 418 | | |
415 | 419 | | |
416 | | - | |
| 420 | + | |
417 | 421 | | |
418 | | - | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
419 | 433 | | |
420 | 434 | | |
421 | 435 | | |
| |||
699 | 713 | | |
700 | 714 | | |
701 | 715 | | |
702 | | - | |
| |||
0 commit comments