fix: generated column expr with SchemaMode::Merge handles missing columns

[veeceey]

Fixes #4169

When appending with SchemaMode::Merge, generated column expressions can reference nullable columns that aren't in the input batch. The problem was that with_generated_columns() resolves expressions against the plan schema before schema evolution runs -- so any column that schema evolution would add as NULL doesn't exist yet, and parse_predicate_expression fails.

The fix catches expression resolution errors in with_generated_columns() and falls back to a typed NULL placeholder. This lets the pipeline continue through schema evolution, which adds the missing columns. The DataValidationExec then sees NULL IS NOT DISTINCT FROM NULL = true, correctly passing validation.

This mirrors the approach already used in add_missing_generated_columns() (the merge path), which also inserts NULL placeholders for missing generated columns.

Test plan

• Added test_generated_column_referencing_missing_column_uses_null_placeholder test that reproduces the exact failure described in the issue
• All 7 generated_columns tests pass
• cargo test -p deltalake-core --lib --features datafusion -- "generated_columns::tests" passes cleanly

[veeceey]

Test Results

$ cargo test -p deltalake-core --lib --features datafusion -- "generated_columns::tests"

running 7 tests
test operations::write::generated_columns::tests::test_empty_generated_columns ... ok
test operations::write::generated_columns::tests::test_existing_columns_pass_through ... ok
test operations::write::generated_columns::tests::test_multiple_generated_columns ... ok
test operations::write::generated_columns::tests::test_generated_column_referencing_missing_column_uses_null_placeholder ... ok
test operations::write::generated_columns::tests::test_missing_non_generated_nullable_column_does_not_error ... ok
test operations::write::generated_columns::tests::test_add_missing_generated_column ... ok
test operations::write::generated_columns::tests::test_mixed_existing_and_generated_columns ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 630 filtered out; finished in 0.09s

[ethan-tyler]

Thanks for the fix here @veeceey. The direction looks right. Requesting changes for the Err(_) catch all in with_generated_columns swallows all parse/resolve failures.

The e2e test is a nice to have.

[ethan-tyler]

This is too broad. We should only catch errors during the schema merge and everything else should surface.

[ethan-tyler]

Bind the error and include it in the log

[ethan-tyler]

Nice test. It would be nice to have an e2e test through WriteBuilder &
SchemaMode::Merge that asserts actual write time values and not just
schema shape.

[ethan-tyler]

Thanks for the changes, lgtm. Just fix the DCO sign off and we should be good to go. Thanks again @veeceey

Looks like you need to fmt as well

[codecov]

Codecov Report

❌ Patch coverage is 88.67925% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.82%. Comparing base (ce6709b) to head (c82fb8a).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...tes/core/src/operations/write/generated_columns.rs	88.67%	4 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4223   +/-   ##
=======================================
  Coverage   76.81%   76.82%           
=======================================
  Files         167      167           
  Lines       48849    48874   +25     
  Branches    48849    48874   +25     
=======================================
+ Hits        37524    37546   +22     
- Misses       9421     9425    +4     
+ Partials     1904     1903    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.