Add `Field` to `Expr::Cast` -- allow logical expressions to express a cast to an extension type #18136

paleolimbot · 2025-10-17T17:33:16Z

Which issue does this PR close?

Closes LogicalPlan Casts can't express a cast to an extension type #18060.

I am sorry that I missed the previous PR implementing this ( #18120 ) and I'm also happy to review that one instead of updating this!

Rationale for this change

Other systems that interact with the logical plan (e.g., SQL, Substrait) can express types that are not strictly within the arrow DataType enum.

What changes are included in this PR?

For the Cast and TryCast structs, the destination data type was changed from a DataType to a FieldRef.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes, any code using Cast { .. } to create an expression would need to use Cast::new() instead (or pass on field metadata if it has it). Existing matches will need to be upated for the data_type -> field member rename.

datafusion/expr/src/expr.rs

paleolimbot · 2025-10-17T17:46:27Z

datafusion/expr/src/expr_schema.rs

+                    f.as_ref()
+                        .clone()
+                        .with_data_type(data_type.data_type().clone())
+                        .with_metadata(f.metadata().clone())
+                    // TODO: should nullability be overridden here or derived from the
+                    // input expression?


I am not sure if this type of cast should be able to express nullability or not.

Currently this PR does not consider the nullability of the cast field, because the destination physical expression won't consider it either (and thus the return fields would be out of sync).

paleolimbot · 2025-10-17T17:49:24Z

datafusion/physical-expr/src/planner.rs

-            data_type.clone(),
+            // TODO: this drops extension metadata associated with the cast
+            data_type.data_type().clone(),


I don't actually need physical expressions to be able to cast things...my vauge plan is to use a logical plan transformation or perhaps optimizer rule to replace casts to extension types with a ScalarUDF call. This should possibly error if there is mismatched metadata between the input and destination (i.e., a physical cast would only ever represent a storage cast, which is usually OK).

I updated this to error at the logical expr -> physical expr stage if there is metadata on the cast field. I think this is better than dropping it (and won't break any existing code because before this PR such a cast could not exist).

paleolimbot · 2025-12-02T00:32:05Z

@alamb If updating DataTypes to FieldRefs is still what we're doing in the logical plan, this PR is ready for review!

adriangb · 2025-12-11T20:03:24Z

This makes sense in general! I only got partially into the review before realizing it conflicts with #19097. Maybe that one is the issue to focus on?

paleolimbot · 2025-12-11T21:46:19Z

Thank you for reviewing!

I didn't see that PR but it looks like it focuses on the physical cast. This PR doesn't really have anything to do with the physical cast (in order to actually execute a cast to an extension type we need a registry of how exactly that should be done, which we don't have yet), although I'm sure there's some merge conflict between them.

adriangb · 2025-12-11T22:18:00Z

Oh right two different layers! But they're both essentially adding Field to the cast operators, which is interesting / why I got confused.

I will try to take another look here tomorrow.

adriangb

Overall this looks good to me.

Is there an example of actually customizing casting, or an issue tracking getting us there? E.g. in postgres you can do select gen_random_uuid() = 'b2bb3fea-2598-449e-ac64-8ad1754df02d';? As far as I can tell that needs more work but this is a step towards that.

adriangb · 2025-12-13T05:40:35Z

datafusion/proto/src/logical_plan/from_proto.rs

            )?);
            let data_type = cast.arrow_type.as_ref().required("arrow_type")?;
-            Ok(Expr::Cast(Cast::new(expr, data_type)))
+            let field = Field::new("", data_type, cast.nullable.unwrap_or(true));


It feels like there should be an UnnamedField or DataTypeWithNullabilityAndMetadata. I saw similar patterns in the literal expressions.

adriangb · 2025-12-13T05:45:33Z

datafusion/physical-expr/src/planner.rs

+            expressions::cast(
+                create_physical_expr(expr, input_dfschema, execution_props)?,
+                input_schema,
+                field.data_type().clone(),


This does end up tying into #19097: I think they'd work well together, we'd just want to pass the field directly here.

adriangb · 2025-12-13T05:48:39Z

datafusion/expr/src/expr.rs

-    pub data_type: DataType,
+    pub field: FieldRef,


Should we keep data_type as a deprecated field that we populate from field.data_type() for a couple of releases?

We could also implement this by adding a metadata: FieldMetadata field rather than switching the data_type to a FieldRef. We mostly just switched DataType to FieldRef in other places so that's what I did here.

adriangb · 2025-12-13T06:25:05Z

datafusion/expr/src/expr_schema.rs

+                    // This currently propagates the nullability of the input
+                    // expression as the resulting physical expression does
+                    // not currently consider the nullability specified here
+                    f.as_ref()
+                        .clone()
+                        .with_data_type(field.data_type().clone())
+                        .with_metadata(f.metadata().clone())


I didn't fully understand this comment / what's going on here. I'm sure it's my fault but could you try rewording the comment, maybe giving f a more meaningful name and replacing _ with _unused_variable_that_has_a_useful_name?

paleolimbot · 2025-12-16T18:15:59Z

Is there an example of actually customizing casting, or an issue tracking getting us there?

There's an issue to create a registry for dyn Extensiony things and a POC PR with not much discussion driving a DataFusion-based solution ( #18223 )...I added a comment there on how we might go about that using the registry. This exact PR allows DataFusion-based projects to implement a workaround (for example, in SedonaDB I'm planning to transform the logical plan to rewrite casts to an extension type to a scalar function call to sd_cast() ). Casts show up in SQL -> LogicalPlan internals and SQL types are now FieldRefs too, so my personal next step was to see if we now have enough to add a UUID type in SQL.

alamb · 2026-01-06T20:36:27Z

Is this waiting for anything else before we merge it? We made a branch-52 now, so merging to main means it will be included in 53.0.0 (or we can backport to branch-52)

paleolimbot · 2026-01-06T21:13:56Z

I'm personally happy with this...it seems like there is no opposition to the main change (DataType -> FielRef in Cast and TryCast). It might be nice to get all the Expr breaking changes out of the way in 52 (I believe Variable was also updated similarly) although I am not in a personal rush. The next steps are to fill in the SQL half (SQL types are already field refs but no built in types have extension types yet) and fill in the extension half (cast registry or something).

github-actions bot added sql SQL Planner logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate substrait Changes to the substrait crate proto Related to proto crate functions Changes to functions implementation labels Oct 17, 2025

paleolimbot mentioned this pull request Oct 17, 2025

feat: Replace DataType with FieldRef in Cast #18120

Closed

paleolimbot commented Oct 17, 2025

View reviewed changes

github-actions bot added the optimizer Optimizer rules label Oct 25, 2025

paleolimbot marked this pull request as ready for review October 27, 2025 14:26

paleolimbot added 12 commits December 1, 2025 14:47

passing tests

f5dd7f5

clippy

89547f4

proto

1e5e1b0

fmt

b23bb42

use the helper

4ac0296

clippy

32c6d26

test not suported

e2e695e

rename data type to field

64980ab

clippy

e6a7e90

maybe better substrait consumer integration

0955d58

no need to update this

2e103a5

comment about nullability

00237be

paleolimbot force-pushed the cast-with-extension branch from 099dc05 to 00237be Compare December 1, 2025 21:06

paleolimbot added 2 commits December 1, 2025 15:11

Merge branch 'main' into cast-with-extension

2d3aed2

nope use the other version

b402c44

paleolimbot mentioned this pull request Dec 3, 2025

Allow Logical expression ScalarVariable to represent an extension type or metadata #18243

Merged

paleolimbot added 2 commits December 8, 2025 14:32

add upgrade guide

5657e89

Merge branch 'main' into cast-with-extension

758070d

fmt

f4d2673

github-actions bot added the documentation Improvements or additions to documentation label Dec 8, 2025

typos

55de0cb

adriangb self-requested a review December 10, 2025 16:49

adriangb approved these changes Dec 13, 2025

View reviewed changes

paleolimbot added 2 commits December 16, 2025 11:24

Merge branch 'main' into cast-with-extension

c621f87

comments

7048c7e

paleolimbot mentioned this pull request Dec 16, 2025

API to register behavior for Extension Types #18223

Open

alamb added the api change Changes the API exposed to users of the crate label Dec 17, 2025

alamb changed the title ~~Allow logical expressions to express a cast to an extension type~~ Add Field to Expr::cast -- allow logical expressions to express a cast to an extension type Dec 17, 2025

alamb changed the title ~~Add Field to Expr::cast -- allow logical expressions to express a cast to an extension type~~ Add Field to Expr::Cast -- allow logical expressions to express a cast to an extension type Dec 17, 2025

paleolimbot added 2 commits January 5, 2026 14:19

Merge branch 'main' into cast-with-extension

8afa764

fix build and format

87ea98a

Add Field to Expr::Cast -- allow logical expressions to express a cast to an extension type #18136

Are you sure you want to change the base?

Add Field to Expr::Cast -- allow logical expressions to express a cast to an extension type #18136

Uh oh!

Conversation

paleolimbot commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paleolimbot commented Dec 2, 2025

Uh oh!

adriangb commented Dec 11, 2025

Uh oh!

paleolimbot commented Dec 11, 2025

Uh oh!

adriangb commented Dec 11, 2025

Uh oh!

adriangb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paleolimbot commented Dec 16, 2025

Uh oh!

alamb commented Jan 6, 2026

Uh oh!

paleolimbot commented Jan 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add `Field` to `Expr::Cast` -- allow logical expressions to express a cast to an extension type #18136

Add `Field` to `Expr::Cast` -- allow logical expressions to express a cast to an extension type #18136

paleolimbot commented Oct 17, 2025 •

edited

Loading