Enable placeholders with extension types #17986

paleolimbot · 2025-10-08T22:13:56Z

Which issue does this PR close?

Closes Logical plan placeholders can't contain metadata #17862

Rationale for this change

Most logical plan expressions now propagate metadata; however, parameters with extension types or other field metadata cannot participate in placeholder/parameter binding.

What changes are included in this PR?

The DataType in the Placeholder struct was replaced with a FieldRef.

Are these changes tested?

They will be! (Work in progress)

Are there any user-facing changes?

Yes, one new function was added to extract the placeholder fields from a plan.

This is a breaking change for code that specifically interacts with the Placeholder struct (but matches on the logical Expr I think are unchanged).

paleolimbot

@timsaucer @alamb I'm happy to add tests for all these components but wanted to make sure this is vaugely headed in the right direction before I do so!

paleolimbot · 2025-10-10T02:38:22Z

datafusion/expr/src/expr.rs

-    pub data_type: Option<DataType>,
+    pub field: Option<FieldRef>,


This is the main change. We can change this to a less severely breaking option (e.g., just add metadata: FieldMetadata to the struct)...I started with the most breaking version to identify its use in as many places as possible.

paleolimbot · 2025-10-10T02:44:26Z

datafusion/core/src/execution/session_state.rs

-    pub(crate) data_types: Vec<DataType>,
+    pub(crate) data_types: Vec<FieldRef>,


This is another change that had some impact

paleolimbot · 2025-10-10T02:46:27Z

datafusion/sql/src/planner.rs

 pub struct PlannerContext {
    /// Data types for numbered parameters ($1, $2, etc), if supplied
    /// in `PREPARE` statement
-    prepare_param_data_types: Arc<Vec<DataType>>,
+    prepare_param_data_types: Arc<Vec<FieldRef>>,


I chose to also do SQL here while I was at it...I could probably isolate these changes into a different PR

paleolimbot · 2025-10-10T02:49:01Z

datafusion/sql/src/planner.rs

-    pub(crate) fn convert_data_type(&self, sql_type: &SQLDataType) -> Result<DataType> {
+    pub(crate) fn convert_data_type(&self, sql_type: &SQLDataType) -> Result<FieldRef> {


I think this change would also enable supporting UUIDs and other SQL types that map to extension types

paleolimbot · 2025-10-10T02:50:33Z

datafusion/sql/src/statement.rs

                    temporary,
                    name,
-                    return_type,
+                    return_type: return_type.map(|f| f.data_type().clone()),


One of the places where metadata is dropped that I didn't update (DdlStatement::CreateFunction args or return type)

paleolimbot · 2025-10-10T02:51:45Z

datafusion/sql/src/expr/mod.rs

+        Ok(Expr::Cast(Cast::new(
+            Box::new(expr),
+            dt.data_type().clone(),
+        )))


Another place where metadata is dropped that I didn't update (casts)

paleolimbot · 2025-10-10T02:53:59Z

datafusion/expr/src/logical_plan/plan.rs

+                                // This check is possibly too strict (requires nullability and field
+                                // metadata align perfectly, rather than compute true type equality
+                                // when field metadata is representing an extension type)
+                                if prev != field {


This highlights something we'll have to fix: how to compute type equality (e.g., is a shredded and unshredded variant the same type?)

paleolimbot · 2025-10-10T02:56:32Z

datafusion/sql/tests/cases/params.rs

        plan,
        @r#"
-    Prepare: "my_plan" [Int32]
+    Prepare: "my_plan" [Field { name: "", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]


This highlights something else we'll have to solve: how to print types. Printing a field is not particularly helpful in this context. (If this change is vaguely in the right direction I'll revert the changes in this file and implement the Debug or DisplayAs trait or wherever these strings are coming from for now).

this is also something @friendlymatthew is likely to run into shortly as he is working on variant support too

paleolimbot · 2025-10-10T02:57:34Z

datafusion/common/src/metadata.rs

+#[derive(Clone, PartialEq, Eq, PartialOrd, Hash, Debug)]
+pub struct FieldMetadata {


I just moved this from the expr crate so I could us it in ParamValues

alamb

Thank you @paleolimbot -- I think this is definitely the right direction. I had some small API suggestions, but overall looks good

The largest open question in my mind is what you have highlighted for customizing behavior for different extension types (e.g. comparing two fields for "equality" and printing them, and casting them, etc.)

@findepi brought up the same thing many months ago when discussing adding new types in

#12644

One idea is to create a TypeRegistry similar to a FunctionRegistry and some sort of ExtensionType trait that encapsulates these behaviors.

The challenge would then be to thread the registry to all places that need it. Though that is likely largely an API design / plumbing exercise

If you think that is an idea worth exploring

alamb · 2025-10-10T10:53:39Z

datafusion/common/src/param_value.rs

 impl ParamValues {
    /// Verify parameter list length and type
-    pub fn verify(&self, expect: &[DataType]) -> Result<()> {
+    pub fn verify(&self, expect: &[FieldRef]) -> Result<()> {


one thing that would be nice to help people upgrade could be to add a new function and deprecate this one -- perhaps something like suggested in https://datafusion.apache.org/contributor-guide/api-health.html#api-health-policy

#[deprecated] pub fn verify(&self, expect: &[DataType]) -> Result<()> { // make dummy Fields let expect = ...; self.verify_fields(&expect) } // new function that has the new signature pub fn verify_fields(&self, expect: &[FieldRef]) -> Result<()> { ... }

alamb · 2025-10-10T10:55:00Z

datafusion/common/src/param_value.rs

                    }
+
+                    if let Some(expected_metadata) = maybe_metadata {
+                        // Probably too strict of a comparison (this is an example of where


Yeah, I agree straight up comparing strings is probably not ideal

If we wanted to introduce type equality, I thing the bigger question is how to thread it through (you would have to have some way to register your types / methods to check equality and ensure that somehow ended up here 🤔 )

alamb · 2025-10-10T10:56:03Z

datafusion/sql/tests/cases/params.rs

        plan,
        @r#"
-    Prepare: "my_plan" [Int32]
+    Prepare: "my_plan" [Field { name: "", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]


this is also something @friendlymatthew is likely to run into shortly as he is working on variant support too

github-actions bot added sql SQL Planner logical-expr Logical plan and expressions proto Related to proto crate physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate common Related to common crate labels Oct 8, 2025

paleolimbot added 10 commits October 9, 2025 20:21

start

6948079

builds!

24c51fd

backward compatible proto

524eab1

move FieldMetadata

a2130c2

maybe working from proto

e764884

oof

02940ff

maybe fix build

83e37d6

fix doc

97261c7

fix the name of the list

4317967

undo trailing whitespace change

04cebe1

paleolimbot force-pushed the params-with-metadata branch from 97a8408 to 04cebe1 Compare October 10, 2025 01:37

maybe fix failing tests

382ba99

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Oct 10, 2025

paleolimbot commented Oct 10, 2025

View reviewed changes

alamb reviewed Oct 10, 2025

View reviewed changes

		pub data_type: Option<DataType>,
		pub field: Option<FieldRef>,

		pub(crate) data_types: Vec<DataType>,
		pub(crate) data_types: Vec<FieldRef>,

		pub(crate) fn convert_data_type(&self, sql_type: &SQLDataType) -> Result<DataType> {
		pub(crate) fn convert_data_type(&self, sql_type: &SQLDataType) -> Result<FieldRef> {

		#[derive(Clone, PartialEq, Eq, PartialOrd, Hash, Debug)]
		pub struct FieldMetadata {

Enable placeholders with extension types #17986

Are you sure you want to change the base?

Enable placeholders with extension types #17986

Uh oh!

Conversation

paleolimbot commented Oct 8, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants