feat: handle struct-based UDT literals in core #613

benbellick · 2025-11-21T19:06:54Z

Support both opaque (google.protobuf.Any) and
structured (Literal.Struct) encodings for user-defined type literals per Substrait spec.

Split UserDefinedLiteral into UserDefinedAny and UserDefinedStruct
Move type parameters to interface level for parameterized types
Add first-class POJO representation for type parameters
Test coverage including roundtrip tests
Throw exception on unhandled struct-based representation in isthmus

Closes #611

core/src/main/java/io/substrait/extension/DefaultExtensionCatalog.java

benbellick · 2025-11-21T19:58:41Z

FYI, this PR makes no attempt to actually validate if the struct representation provided matches the definition in the yaml file. I think that this is the right thing to do, but it turned out to be slightly more complicated than I thought, as it involves threading an ExtensionCollection through the codebase a bit. Thus, I left it for another issue (#614).

Support both opaque (google.protobuf.Any) and structured (Literal.Struct) encodings for user-defined type literals per Substrait spec. - Split UserDefinedLiteral into UserDefinedAny and UserDefinedStruct - Move type parameters to interface level for parameterized types - Comprehensive test coverage including roundtrip tests - Throw exception on unhandled struct-based representation in isthmus

benbellick · 2025-11-24T16:30:36Z

core/src/main/java/io/substrait/expression/proto/ExpressionProtoConverter.java

        extensionCollector.getTypeReference(SimpleExtension.TypeAnchor.of(expr.urn(), expr.name()));
    return lit(
        bldr -> {
-          try {


This exception doesn't happen anymore because we don't parse the Any here. Instead, we have a reference to the pre-parsed proto directly.

benbellick · 2025-11-24T20:03:20Z

core/src/main/java/io/substrait/type/proto/ParameterizedProtoConverter.java

+    public ParameterizedType userDefined(
+        int ref, java.util.List<io.substrait.type.Type.Parameter> typeParameters) {
+      throw new UnsupportedOperationException(
+          "User defined types are not supported in Parameterized Types for now");


This is consistent with the above, where we don't yet support ParamerizedType conversion.

benbellick · 2025-11-24T20:04:20Z

core/src/main/java/io/substrait/type/proto/TypeExpressionProtoVisitor.java

+    public DerivationExpression userDefined(
+        int ref, java.util.List<io.substrait.type.Type.Parameter> typeParameters) {
+      throw new UnsupportedOperationException(
+          "User defined types are not supported in Derivation Expressions for now");


This is also consistent with the above.

core/src/main/java/io/substrait/type/proto/TypeProtoConverter.java

core/src/test/java/io/substrait/type/proto/LiteralRoundtripTest.java

benbellick · 2025-11-24T20:25:49Z

isthmus/src/main/java/io/substrait/isthmus/expression/ExpressionRexConverter.java

+  public RexNode visit(Expression.UserDefinedStructLiteral expr, Context context)
+      throws RuntimeException {
+    throw new UnsupportedOperationException(
+        "UserDefinedStructLiteral representation is not yet supported in Isthmus");


This is #612

…lcite

vbarua

Started taking a look at this and left some comments. Mostly looks reasonable. Want to come back and look at your tests with fresh eyes, and also think about parameterized types with a fresh 🧠.

core/src/main/java/io/substrait/expression/Expression.java

vbarua · 2025-11-26T01:34:06Z

core/src/main/java/io/substrait/expression/Expression.java

-      return Type.withNullability(nullable()).userDefined(urn(), name());
+    public abstract List<io.substrait.type.Type.Parameter> typeParameters();
+
+    public abstract List<Literal> fields();


Is there a reason to use a List<Literal> here instead of just a Expression.StructLiteral, which would map directly to the protobuf?

Sharing this comment because I believe it is relevant.

Basically it is the same issue, which is that the POJO class called StructLiteral is representing a special case of the Literal class.

message Literal { // Both StructLiteral and UserDefinedStructLiteral are representing this _whole_ message oneof literal_type { ... Struct struct = 25; ... UserDefined user_defined = 33; } ... message Struct { // A possibly heterogeneously typed list of literals repeated Literal fields = 1; } message UserDefined { oneof type_anchor_type { // points to a type_anchor defined in this plan uint32 type_reference = 1; // points to a type_alias_anchor defined in this plan. uint32 type_alias_reference = 5; } // The parameters to be bound to the type class, if the type class is // parameterizable. repeated Type.Parameter type_parameters = 3; // a user-defined literal can be encoded in one of two ways oneof val { // the value of the literal, serialized using some type-specific protobuf message google.protobuf.Any value = 2; // the value of the literal, serialized using the structure definition in its declaration Literal.Struct struct = 4; } } }

Back to your comment, switching this member variable to be Expression.StructLiteral would amount to embedding one literal proto inside of another. For example, the proto Struct doesn't actually have a nullable field. But the POJO Expression.StructLiteral does have a nullable field because it inherits from Literal.

This doesn't mean that we couldn't replace the member variable as you suggest, but if we did do that, we would be carrying around extra meaningless data, which I think is more confusing ultimately.

vbarua · 2025-11-26T01:36:12Z

core/src/main/java/io/substrait/expression/Expression.java

+   * User-defined literal with value encoded as {@code Literal.Struct}.
+   *
+   * <p>This encoding uses a structured list of fields to represent the literal value.
+   */


The docstring feels a little inconsistent. You have

literal with value encoded as {@code Literal.Struct}

but in the class the values are encoded as a List<Literal>, and the second part of the docstring is consistent with that.

I guess that this could be made less confusing. The intention was to mean the proto Literal.Struct, no some java class. What do you think about just altering the message to say

User-defined literal with value encoded via the proto message {@code Literal.Struct}.

?

I altered it to instead link to the actual proto. Let me know what you think! I also did the same for the any case.

vbarua · 2025-11-26T01:41:44Z

core/src/main/java/io/substrait/expression/Expression.java

+    @Override
+    public abstract List<io.substrait.type.Type.Parameter> typeParameters();
+
+    public abstract com.google.protobuf.Any value();


Release Notes
Capturing the value as an Any instead of a ByteString does feel nicer ✨

vbarua · 2025-11-26T01:42:24Z

core/src/main/java/io/substrait/expression/Expression.java

+   * @see UserDefinedAnyLiteral
+   * @see UserDefinedStructLiteral
+   */
+  interface UserDefinedLiteral extends Literal {


Release Notes
We should call out that we don't construct UserDefinedLiterals anymore.

vbarua · 2025-11-26T02:13:50Z

core/src/main/java/io/substrait/type/Type.java

+   * parameters (like the {@code 10} in {@code VARCHAR<10>}). This interface provides a type-safe
+   * representation of all possible parameter kinds.
+   */
+  interface Parameter {}


Is the function of https://github.com/substrait-io/substrait-java/blob/main/core/src/main/java/io/substrait/function/ParameterizedType.java applicable here? It feels vaguely related.

🤔 yes that is an interesting point. Looking into it!

So as I understand it, ParameterizedType.java is used for representing abstract types with parameters in yaml files. Where as the Parameter above being introduced is actually a concrete argument passed into the type.

So for example, List<any1> could be a ParameterizedType, whereas List<int32> is a type with parameters [int32].

vbarua · 2025-11-26T02:15:24Z

core/src/main/java/io/substrait/type/Type.java

+  @Value.Immutable
+  abstract class ParameterBooleanValue implements Parameter {
+    public abstract boolean value();
+  }


Is there a usecase for having boolean literal type parameters? I can't imagine a usecase where something like MySpecialType<false> is something you would need.

#613 (comment)

🤷 just wanted to be consistent with the spec

vbarua · 2025-11-26T02:16:32Z

core/src/main/java/io/substrait/type/Type.java

+  @Value.Immutable
+  abstract class ParameterEnumValue implements Parameter {
+    public abstract String value();
+  }


What would this type be used for?

As discussed offline, these were inspired by this portion of the simple extension schema:

type_param_defs: # an array of compound type parameter definitions type: array items: ... properties: ... type: # expected metatype for the parameter type: string enum: - dataType - boolean - integer - enumeration - string

So while I don't understand the usage of it, I thought it was best to include all of them for consistency with the spec.

vbarua · 2025-11-26T02:16:36Z

core/src/main/java/io/substrait/type/Type.java

+  @Value.Immutable
+  abstract class ParameterStringValue implements Parameter {
+    public abstract String value();
+  }


What would this type be used for?

#613 (comment)

core/src/test/java/io/substrait/TestBase.java

benbellick · 2025-12-04T16:37:02Z

FYI, I have a WIP PR for implementing this in Isthmus but I split it into two PRs because the code there is more complicated. This keeps the PRs a bit smaller!

github-actions · 2026-01-05T12:59:03Z

ACTION NEEDED

Substrait follows the Conventional Commits
specification for
release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

core/src/main/java/io/substrait/expression/ExpressionCreator.java

nielspardon · 2026-01-05T14:17:12Z

core/src/main/java/io/substrait/expression/Expression.java

+    @Override
+    public abstract List<io.substrait.type.Type.Parameter> typeParameters();
+
+    public abstract com.google.protobuf.Any value();


One observation/remark I would have here is that in other places we have been trying to not expose protobuf objects in the Substrait core POJO API like e.g. in AdvancedExtension we provide the empty Optimization and Enhancement interfaces and ask users of the Substrait Java SDK to implement protobuf conversion logic for those instead. At first glance it feels like we would be a little inconsistent if for user defined any literals we expose the protobuf any value directly.

Hey Niels, this was a great point. I just pushed a commit addressing this by introducing the interface UserDefinedAnyValue. I'll leave the PR open for a little while to give you the opportunity to provide feedback on that component if you'd like. Otherwise, I'll merge it in later today. Thanks for the review!

Actually, after implementing the abstraction pattern similar to AdvancedExtension, I think we should actually defer this change. The abstraction adds an extra layer (UserDefinedAnyValue) without clear benefits. Users still need to work with com.google.protobuf.Any directly to construct values, and the indirection makes the API harder to understand. Since you mentioned this could be revisited later, let's keep it simple for now and use Any directly as the spec defines.

Yes, on that point I also wasn't sure how much additional value the additional layer of abstraction would bring.

nielspardon · 2026-01-05T14:29:33Z

examples/substrait-spark/src/main/java/io/substrait/examples/util/SubstraitStringify.java

   * Explains the Sustrait relation
   *
-   * @param plan Subsrait relation
+   * @param rel Subsrait relation


side note: this fix is also in #642

nielspardon

LGTM

I'm fine with the changes. Just not sure whether we want to revisit exposing the Any proto value directly in a follow-up.

Co-authored-by: Niels Pardon <[email protected]>

This reverts commit 76cd0f0.

benbellick force-pushed the benbellick/handle-structured-udt2 branch from 133b704 to e4f36ea Compare November 21, 2025 19:08

benbellick changed the title ~~Benbellick/handle structured udt2~~ handle struct-based UDT literals in core Nov 21, 2025

benbellick force-pushed the benbellick/handle-structured-udt2 branch from e4f36ea to cef9798 Compare November 21, 2025 19:16

benbellick commented Nov 21, 2025

View reviewed changes

core/src/main/java/io/substrait/extension/DefaultExtensionCatalog.java Show resolved Hide resolved

benbellick force-pushed the benbellick/handle-structured-udt2 branch from cef9798 to f3379c1 Compare November 21, 2025 19:51

benbellick marked this pull request as ready for review November 21, 2025 20:01

benbellick requested a review from vbarua November 21, 2025 20:01

benbellick added 3 commits November 24, 2025 11:19

test: add test to ensure struct-UDTs work with type parameters

f0e13ef

test: improve UDT literal test by making opaque-ness explicit

f5b6341

benbellick force-pushed the benbellick/handle-structured-udt2 branch from 1eb4fb4 to f5b6341 Compare November 24, 2025 16:19

refactor: rename UserDefined{kind} to UserDefined{kind}Literal

8c54706

benbellick commented Nov 24, 2025

View reviewed changes

benbellick added 3 commits November 24, 2025 14:10

feat: first class support for TypeParameters

36c81f0

docs: update docstring to explain use of Parameter

680ed29

test: add test showing all Type.Parameters preserved

6ac7aab

benbellick commented Nov 24, 2025

View reviewed changes

core/src/main/java/io/substrait/type/proto/TypeProtoConverter.java Outdated Show resolved Hide resolved

benbellick commented Nov 24, 2025

View reviewed changes

core/src/test/java/io/substrait/type/proto/LiteralRoundtripTest.java Show resolved Hide resolved

benbellick commented Nov 24, 2025

View reviewed changes

benbellick added 2 commits November 24, 2025 16:45

fix: correctly handle type conversion with paramterized types

b2063d0

refactor: revert to simpler binary construction for any rep UDT in ca…

ce7985f

…lcite

benbellick mentioned this pull request Nov 24, 2025

Benbellick/handle structured udt #609

Draft

fix: ensure nullability preserved in calcite roundtrip

e8cb862

benbellick force-pushed the benbellick/handle-structured-udt2 branch from 78488b4 to e8cb862 Compare November 25, 2025 19:27

vbarua reviewed Nov 26, 2025

View reviewed changes

chore: addressing some PR comments

77f0bd7

benbellick force-pushed the benbellick/handle-structured-udt2 branch from fea5921 to 77f0bd7 Compare November 26, 2025 18:14

chore: addressing more PR comments

4b4f5ee

benbellick force-pushed the benbellick/handle-structured-udt2 branch from 3814053 to 4b4f5ee Compare November 26, 2025 18:26

chore: simplify documentation

3732cf9

benbellick force-pushed the benbellick/handle-structured-udt2 branch from b2742da to 3732cf9 Compare November 26, 2025 18:55

benbellick mentioned this pull request Nov 26, 2025

feat: add structured UDT literal support with dual encoding #623

Draft

Merge branch 'main' into benbellick/handle-structured-udt2

a216f3e

benbellick requested a review from vbarua December 4, 2025 16:18

nielspardon requested review from andrew-coleman and bestbeforetoday December 4, 2025 16:24

benbellick mentioned this pull request Dec 5, 2025

feat: add initial extension support substrait-io/substrait-rs#395

Merged

benbellick added 2 commits December 16, 2025 13:34

Merge branch 'main' into benbellick/handle-structured-udt2

f61f68c

Merge branch 'main' into benbellick/handle-structured-udt2

02f7920

benbellick changed the title ~~handle struct-based UDT literals in core~~ feat: handle struct-based UDT literals in core Jan 5, 2026

nielspardon reviewed Jan 5, 2026

View reviewed changes

core/src/main/java/io/substrait/expression/ExpressionCreator.java Outdated Show resolved Hide resolved

nielspardon reviewed Jan 5, 2026

View reviewed changes

core/src/main/java/io/substrait/expression/ExpressionCreator.java Outdated Show resolved Hide resolved

nielspardon reviewed Jan 5, 2026

View reviewed changes

nielspardon approved these changes Jan 5, 2026

View reviewed changes

benbellick and others added 3 commits January 5, 2026 12:01

Apply suggestions from code review

d009af4

Co-authored-by: Niels Pardon <[email protected]>

add interface for UserDefinedAnyValue

76cd0f0

Revert "add interface for UserDefinedAnyValue"

a60e0b7

This reverts commit 76cd0f0.

feat: handle struct-based UDT literals in core #613

Are you sure you want to change the base?

feat: handle struct-based UDT literals in core #613

Uh oh!

Conversation

benbellick commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

benbellick commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vbarua left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benbellick Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

benbellick commented Dec 4, 2025

Uh oh!

github-actions bot commented Jan 5, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nielspardon left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

benbellick commented Nov 21, 2025 •

edited

Loading

benbellick Nov 26, 2025 •

edited

Loading