[improvement](metrics) Export file cache remote index byte metrics#65398
Open
hoshinojyunn wants to merge 1 commit into
Open
[improvement](metrics) Export file cache remote index byte metrics#65398hoshinojyunn wants to merge 1 commit into
hoshinojyunn wants to merge 1 commit into
Conversation
hoshinojyunn
requested review from
Gabriel39,
gavinchou,
liaoxin01 and
morningman
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Issue Number: None
Related PR: None
Problem Summary: File cache profile statistics already track `inverted_index_bytes_read_from_remote` and `segment_footer_index_bytes_read_from_remote`, but BE Doris metrics only exported aggregate cache, remote, and peer read bytes. That made these index-specific remote read costs visible in query profile only, not in the BE metrics endpoint. This change registers two new Doris metrics, extends the existing `FileCacheMetrics` aggregation path to accumulate them, and exports them through the normal metrics hook. The update also extends the existing file cache profile unit test helpers so segment footer index statistics are covered and adds a test for the new metric export path.
Expose `inverted_index_bytes_read_from_remote` and `segment_footer_index_bytes_read_from_remote` in BE Doris metrics.
- Test: Manual compile verification
- Started `./run-be-ut.sh --run --filter=FileCacheProfileReporterTest.*`, but it triggered a large `doris_be_test` rebuild and was interrupted after the changed objects compiled successfully
- Recompiled changed objects with `ninja -C be/ut_build_ASAN src/common/CMakeFiles/Common.dir/metrics/doris_metrics.cpp.o src/io/CMakeFiles/IO.dir/cache/block_file_cache_profile.cpp.o test/CMakeFiles/doris_be_test.dir/io/cache/block_file_cache_profile_reporter_test.cpp.o`
- Behavior changed: Yes (adds two new BE metrics for file cache remote index bytes)
- Does this need documentation: No
hoshinojyunn
force-pushed
the
feat/profile
branch
from
f128c74 to
da72c65
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29891 ms |
Contributor
TPC-DS: Total hot run time: 181241 ms |
Contributor
ClickBench: Total hot run time: 25.24 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Number: None
Related PR: None
Problem Summary:
File cache statistics already track the remote-read bytes for inverted index and segment footer
index data, but BE Doris metrics only export the aggregate cache/local/remote/peer byte counters.
As a result, these two index-specific remote read costs are visible in query profile internals but
not exposed from the BE
/metricsendpoint, which makes them harder to monitor and alert on incloud/file-cache scenarios.
This PR exposes the following two BE metrics:
doris_be_inverted_index_bytes_read_from_remotedoris_be_segment_footer_index_bytes_read_from_remoteThe change keeps the existing file cache metrics aggregation path and extends it in three places:
DorisMetricsFileCacheMetrics::AtomicStatisticsand the aggregation/update path to accumulate themRelease note
Add two new BE metrics for file cache remote index read bytes:
doris_be_inverted_index_bytes_read_from_remotedoris_be_segment_footer_index_bytes_read_from_remoteCheck List (For Author)
Test
./run-regression-test.sh --run -d cloud_p0 -s test_file_cache_remote_index_byte_metrics -runMode=cloud./run-be-ut.sh --run --filter=DorisMetricsTest.*:FileCacheProfileReporterTest.*Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)