[feature](variant) add variant doc snapshot mode #59183
Open
+12,420
−1,174
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
csun5285
requested review from
dataroaring,
gavinchou,
yiguolei and
zclllyybb
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
csun5285
force-pushed
the
test_variant
branch
from
b307644 to
7d6f45a
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
csun5285
force-pushed
the
test_variant
branch
from
7d6f45a to
a1b24a3
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run p0 10 |
TPC-H: Total hot run time: 36375 ms |
TPC-DS: Total hot run time: 180018 ms |
ClickBench: Total hot run time: 27.17 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
csun5285
force-pushed
the
test_variant
branch
from
a1b24a3 to
943db93
Compare
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 35042 ms |
TPC-DS: Total hot run time: 178720 ms |
ClickBench: Total hot run time: 28.28 s |
eldenmoon
reviewed
Dec 23, 2025
| Status parse_variant_columns(vectorized::Block& block, const TabletSchema& tablet_schema, | ||
| const std::vector<uint32_t>& column_pos); | ||
|
|
||
| // Moved from `vec/common/schema_util.{h,cpp}`. |
|
|
||
| namespace doris::segment_v2::variant_util { | ||
|
|
||
| // Parse variant columns by picking variant positions from `column_pos` and generating ParseConfig |
| } | ||
|
|
||
| void ColumnVariant::serialize_from_doc_snapshot_to_json_format(int64_t row_num, | ||
| BufferWritable& output, |
| BeConsts::DEFAULT_VARIANT_MAX_SPARSE_COLUMN_STATS_SIZE; | ||
| // default to 0, no shard | ||
| int32_t _variant_sparse_hash_shard_count = 0; | ||
|
|
Member
There was a problem hiding this comment.
better wrap all the variant related properties to a structure
be/src/olap/tablet_schema.h
Outdated
|
|
||
| int64_t _variant_doc_snapshot_min_rows = 0; | ||
|
|
||
| int32_t _variant_doc_snapshot_shard_count = 128; |
Member
There was a problem hiding this comment.
what if specify _variant_sparse_hash_shard_count and _variant_doc_snapshot_shard_count at the same time?
|
|
||
| vectorized::MutableColumnPtr variant_column; | ||
| if (var.is_doc_snapshot_mode()) { | ||
| // doc snapshot mode, we need to parse the doc snapshot column |
| auto converter = std::make_unique<vectorized::OlapBlockDataConvertor>(); | ||
| int column_id = 0; | ||
| int64_t variant_doc_snapshot_min_rows = parent_column.variant_doc_snapshot_min_rows(); | ||
| if (variant_doc_snapshot_min_rows == 0 || |
Member
There was a problem hiding this comment.
variant_doc_snapshot_min_rows == 0 is redundant and could be removed ?
| @SerializedName(value = "variantSparseHashShardCount") | ||
| private final int variantSparseHashShardCount; | ||
|
|
||
| @SerializedName(value = "enableVariantDocSnapshotMode") |
| @SerializedName(value = "enableVariantDocSnapshotMode") | ||
| private final boolean enableVariantDocSnapshotMode; | ||
|
|
||
| @SerializedName(value = "variantDocSnapshotMinRows") |
Member
There was a problem hiding this comment.
doc_materizalization_min_rows?
| @SerializedName(value = "variantDocSnapshotMinRows") | ||
| private final long variantDocSnapshotMinRows; | ||
|
|
||
| @SerializedName(value = "variantDocSnapshotShardCount") |
Contributor
Author
|
run p0 10 |
csun5285
force-pushed
the
test_variant
branch
from
943db93 to
3cb0f62
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
csun5285
force-pushed
the
test_variant
branch
from
1828d8a to
73fd870
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 36638 ms |
TPC-DS: Total hot run time: 179402 ms |
ClickBench: Total hot run time: 28.53 s |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)