[feature](cache) support file cache admission control #59065
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ache_admission_control
wenzhenghu
requested review from
CalvinKirs,
dataroaring,
gavinchou and
morningman
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 35116 ms |
TPC-DS: Total hot run time: 178276 ms |
ClickBench: Total hot run time: 27.17 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 36571 ms |
TPC-DS: Total hot run time: 178175 ms |
ClickBench: Total hot run time: 27.96 s |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 32051 ms |
TPC-DS: Total hot run time: 174278 ms |
ClickBench: Total hot run time: 27.6 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
xuchenhao
force-pushed
the
file_cache_admission_control
branch
from
f48b520 to
bc0ef08
Compare
Contributor
|
run buildall |
Contributor
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 31840 ms |
TPC-DS: Total hot run time: 174775 ms |
ClickBench: Total hot run time: 27.32 s |
Contributor
|
run buildall |
wenzhenghu
commented
Jan 15, 2026
| } | ||
| } | ||
|
|
||
| public static class ConcurrentRuleManager { |
Contributor
Author
There was a problem hiding this comment.
ConcurrentRuleManager 这个是否也直接改成 RuleManager
wenzhenghu
commented
Jan 15, 2026
| List<AdmissionRule> allRules = new ArrayList<>(); | ||
|
|
||
| for (File jsonFile : jsonFiles) { | ||
| List<AdmissionRule> loadedRules = RuleLoader.loadRulesFromFile(jsonFile.getPath()); |
Contributor
Author
There was a problem hiding this comment.
如果出现重复的rule，会如何处理，行为是确定的吗？
wenzhenghu
commented
Jan 15, 2026
| return this.scanParams; | ||
| } | ||
|
|
||
| protected boolean fileCacheAdmissionCheck() throws UserException { |
Contributor
Author
There was a problem hiding this comment.
存算分离场景的内表scan是走filequeryscannode吗？
Contributor
Author
There was a problem hiding this comment.
哪些类型catalog的scan会走filequeryscannode？
wenzhenghu
commented
Jan 15, 2026
| protected FileSplitter fileSplitter; | ||
|
|
||
| private static final Set<String> CACHEABLE_CATALOGS = new HashSet<>( | ||
| Arrays.asList("hms", "iceberg", "paimon", "lakesoul") |
Contributor
Author
There was a problem hiding this comment.
如何判断是这几种可以启用filecache？
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 31828 ms |
TPC-DS: Total hot run time: 174466 ms |
ClickBench: Total hot run time: 27.12 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To fully understand the implementation of the PR, please refer to the following link（It is a Chinese document）: https://www.notion.so/V3-1-2c31293e1081807ca476dd5c87efb28e
1. PR Function Overview
The core function of this PR is the implementation of File Cache Admission Control.
2. Implementation Scheme Analysis
The implementation consists of the following key components:
2.1. FE Side: Admission Decision
The primary logic is located in the
createScanRangeLocationsmethod ofFileQueryScanNode.java.Config.enable_file_cache_admission_controlswitch.FileCacheAdmissionManager(Singleton) to execute the specific admission judgment logic.FileQueryScanNoderetrieves the current User Identity (userIdentity), Catalog, Database, and Table information.FileCacheAdmissionManager.getInstance().isAllowed(...)to obtain a boolean resultfileCacheAdmission.2.2. FE Side: Decision Propagation
The decision result needs to be propagated from the
FileQueryScanNodedown to the underlying split assignment logic.SplitAssignment Modification:
SplitAssignmentclass (located inorg.apache.doris.datasource) is modified to accept a newboolean fileCacheAdmissionparameter.SplitToScanRange Modification:
splitToScanRangemethod (or its corresponding Lambda expression) is updated to receive thefileCacheAdmissionparameter.2.3. Communication Protocol: Thrift Definition Update
To pass the FE's decision to the BE, the Thrift definition (likely
TFileRangeDescorTFileScanRangeParamsinPlanNodes.thrift) requires a new field.optional bool file_cache_admission, is added to theTFileRangeDescstruct.2.4. BE Side: Enforcement
Although the analysis focuses on the FE, the complete loop requires enforcement on the BE side:
FileReader(e.g.,HdfsFileReaderorS3FileReader) checks thefile_cache_admissionflag in the incomingTFileRangeDescduring initialization or reading.file_cache_admissionis true (default): It uses the standardFileCachePolicy, where data not found in the cache is written to the Block File Cache after reading.file_cache_admissionis false: It sets theFileCachePolicytoNO_CACHE, skips the cache writing step, reading directly from remote storage. This protects the existing cache from being polluted.3. Summary
This PR introduces an Admission Control Manager during the FE query planning phase and transparently passes this control signal through the Split Assignment and Scan Range Generation phases. This ultimately guides the BE side's file readers to selectively use the file cache.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)