[fix](be) Align cast input nullability for Iceberg struct evolution#65370
Open
Gabriel39 wants to merge 1 commit into
Open
[fix](be) Align cast input nullability for Iceberg struct evolution#65370Gabriel39 wants to merge 1 commit into
Gabriel39 wants to merge 1 commit into
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Reading Iceberg struct fields after Spark schema evolution can materialize a column whose root nullability does not match the declared file type used to prepare the cast expression. The cast wrapper unwraps nullable inputs based on the declared input type, so a nullable declared type with a non-nullable actual column can hit an invalid unwrap path and crash BE instead of returning the cast result. This change aligns the declared cast input root nullability with the actual column shape before executing the cast. It also adds a focused BE unit test and an Iceberg regression case that creates a Spark Iceberg struct, evolves col.a from INT to BIGINT, refreshes the Doris catalog, and queries col.a/col.b/col.c with nested pruning enabled.
### Release note
None
### Check List (For Author)
- Test: Manual test
- Ran git diff --check for modified files
- Ran targeted clang-format v16 wrapper for modified BE C++ files
- Attempted BE unit test: ./run-be-ut.sh --run --filter=TableReaderTest.ComplexRematerializeCastsNonNullableScalarChildWithNullableFileType:TableReaderTest.ComplexRematerializeCastsScalarChildToTableType, blocked because local thirdparty/installed is incomplete (missing bin/protoc and Snappy)
- Attempted regression test: ./run-regression-test.sh --run -d external_table_p0/iceberg -s test_iceberg_struct_schema_evolution, blocked because local thirdparty/installed is incomplete (missing bin/thrift)
- Behavior changed: No
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 10th, 2026 6:44 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
Author
|
/review |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29753 ms |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I found one test coverage issue that should be fixed before merging.
Critical checkpoint conclusions:
- Goal and correctness: the BE change is focused on aligning declared cast input nullability with the actual column shape, and the main TableReader path matches the cast wrapper behavior that unwraps nullable inputs according to the declared type.
- Scope: the implementation is small and localized to
_cast_column_to_type; I did not find an unrelated code-path expansion. - Concurrency, lifecycle, configuration, persistence, and protocol compatibility: not materially changed by this PR.
- Parallel paths: the existing nullable-actual/non-nullable-declared case is still handled, and the new opposite case is now handled before the cast expression executes. I did not find optimizer/rewrite or aggregate pushdown semantic regressions.
- Tests: the new BE unit test checks the direct crash path, but the new Iceberg regression query is not an expected-output assertion, so it can miss wrong values after nested pruning/casting.
- Performance and observability: no additional issue found for this narrow cast-path fix.
User focus: no additional user-provided review focus was supplied.
Subagent conclusions:
optimizer-rewrite: no candidates; convergence round 1 returnedNO_NEW_VALUABLE_FINDINGS.tests-session-config: proposedTSC-1, accepted asMAIN-1and submitted inline; convergence round 1 returnedNO_NEW_VALUABLE_FINDINGS.
Validation: git diff --check passed for the changed files. I could not run the targeted BE UT or external regression in this runner because thirdparty/installed is absent, including bin/protoc and bin/thrift.
Contributor
TPC-DS: Total hot run time: 180622 ms |
Contributor
ClickBench: Total hot run time: 25.05 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Reading Iceberg struct fields after Spark schema evolution can materialize a column whose root nullability does not match the declared file type used to prepare the cast expression. The cast wrapper unwraps nullable inputs based on the declared input type, so a nullable declared type with a non-nullable actual column can hit an invalid unwrap path and crash BE instead of returning the cast result. This change aligns the declared cast input root nullability with the actual column shape before executing the cast. It also adds a focused BE unit test and an Iceberg regression case that creates a Spark Iceberg struct, evolves col.a from INT to BIGINT, refreshes the Doris catalog, and queries col.a/col.b/col.c with nested pruning enabled.
Release note
None
Check List (For Author)
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)