[feature](vparquet-reader) Implements parquet file page cache.#59307
Merged
morningman merged 1 commit intoapache:masterfrom Jan 26, 2026
Merged
[feature](vparquet-reader) Implements parquet file page cache.#59307morningman merged 1 commit intoapache:masterfrom
morningman merged 1 commit intoapache:masterfrom
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 35539 ms |
TPC-DS: Total hot run time: 179978 ms |
ClickBench: Total hot run time: 27.34 s |
kaka11chen
force-pushed
the
parquet_page_cache
branch
from
e32d9d4 to
b25ce38
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 35717 ms |
TPC-DS: Total hot run time: 179518 ms |
ClickBench: Total hot run time: 27.98 s |
kaka11chen
force-pushed
the
parquet_page_cache
branch
from
b25ce38 to
7a15d5f
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 36141 ms |
TPC-DS: Total hot run time: 179957 ms |
ClickBench: Total hot run time: 27.16 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
kaka11chen
force-pushed
the
parquet_page_cache
branch
from
7a15d5f to
eeb4023
Compare
Contributor
Author
|
run buildall |
kaka11chen
force-pushed
the
parquet_page_cache
branch
from
eeb4023 to
3d89dda
Compare
Contributor
Author
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
kaka11chen
force-pushed
the
parquet_page_cache
branch
from
3d89dda to
59ebab3
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 35311 ms |
TPC-DS: Total hot run time: 179267 ms |
ClickBench: Total hot run time: 27.13 s |
kaka11chen
force-pushed
the
parquet_page_cache
branch
from
59ebab3 to
a8f6d42
Compare
ClickBench: Total hot run time: 27.91 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
74 tasks
morningman
reviewed
Jan 20, 2026
be/src/io/file_factory.cpp
Outdated
| return unexpected(std::move(reader_res).error()); | ||
| } | ||
| auto file_reader = std::move(reader_res).value(); | ||
| LOG_INFO("create file reader for path={}, size={}, mtime={}", file_description.path, |
Contributor
There was a problem hiding this comment.
Remove this log, or using DEBUG level
| _page_reader->file_end_offset(), _page_reader->header_start_offset()); | ||
| const std::vector<uint8_t>& header_bytes = _page_reader->header_bytes(); | ||
| size_t total = header_bytes.size() + level_bytes.size() + payload.size; | ||
| auto* page = new DataPage(total, true, segment_v2::DATA_PAGE); |
Contributor
There was a problem hiding this comment.
Potential memory leak?
Better use a more safety way?
Contributor
Author
There was a problem hiding this comment.
No leak here: StoragePageCache::insert takes ownership of the DataPage* and the LRU cache frees it via LRUHandle::free() (which deletes LRUCacheValueBase) when the entry is evicted or the
last handle is released, so the allocation in be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp is managed by the cache (be/src/olap/page_cache.cpp, be/src/olap/
lru_cache.h). Even in the LRU‑K “first insert not cached” path, the returned handle release will free the value.
| void ColumnChunkReader<IN_COLLECTION, OFFSET_INDEX>::_insert_page_into_cache( | ||
| const std::vector<uint8_t>& level_bytes, const Slice& payload) { | ||
| StoragePageCache::CacheKey key( | ||
| fmt::format("{}::{}", _stream_reader->path(), _stream_reader->mtime()), |
Contributor
There was a problem hiding this comment.
fmt::format("{}::{}", _stream_reader->path(), _stream_reader->mtime()
this part is same for every page, better cache it to reuse:
class ParquetPageCacheKeyBuilder {
std::string _file_key_prefix; // Cached once per column chunk
public:
void init(const std::string& path, int64_t mtime) {
_file_key_prefix = fmt::format("{}::{}", path, mtime);
}
StoragePageCache::CacheKey make_key(uint64_t end_offset, uint64_t offset) const {
return StoragePageCache::CacheKey(_file_key_prefix, end_offset, offset);
}
};
| _page_statistics.page_cache_hit_counter += 1; | ||
| // Detect whether the cached payload is compressed or decompressed and record | ||
| bool is_cache_payload_decompressed = true; | ||
| if (_cur_page_header.compressed_page_size > 0) { |
Contributor
There was a problem hiding this comment.
Better extract this check logic:
bool should_cache_decompressed(const tparquet::PageHeader* header,
const tparquet::ColumnMetaData& metadata) {
if (header->compressed_page_size <= 0) return true;
if (metadata.codec == tparquet::CompressionCodec::UNCOMPRESSED) return true;
double ratio = static_cast<double>(header->uncompressed_page_size) /
static_cast<double>(header->compressed_page_size);
return ratio <= config::parquet_page_cache_decompress_threshold;
}
And reuse it for both here and in be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp
kaka11chen
force-pushed
the
parquet_page_cache
branch
from
7cb3056 to
c18b9a0
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 32151 ms |
TPC-DS: Total hot run time: 173930 ms |
ClickBench: Total hot run time: 26.92 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
morningman
approved these changes
Jan 26, 2026
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
gavinchou
approved these changes
Jan 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
Release note
[Feature] Implementation of Parquet File Page Cache and Integration with Unified Page Cache Framework
Solution Overview
This PR implements a page-level caching mechanism for Parquet files and integrates it with Apache Doris's existing unified page cache framework, significantly improving query performance by caching decompressed (or compressed) data pages in memory.
Key Features
• Leverages Existing Framework: Directly integrates with Doris's StoragePageCache infrastructure used for internal tables
• Shared Resource Management: Parquet cache shares memory pool and eviction policies with internal table caches
• Consistent Monitoring: Reuses existing cache statistics and RuntimeProfile for unified performance monitoring
• Cache Type Identification: Uses segment_v2::DATA_PAGE as cache page type, consistent with internal table data page caching
• Compression Ratio Awareness: Automatically chooses between caching compressed or decompressed data based on parquet_page_cache_decompress_threshold (default: 1.5)
• Flexible Storage: Caches decompressed data when uncompressed_size/compressed_size ≤ threshold, otherwise caches compressed data if enable_parquet_cache_compressed_pages=true
• Cache Key Design: Uses file_path::mtime::offset as key to ensure cache consistency across file modifications
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)