feat(core): handle struct-based UDT literals #613

benbellick · 2025-11-21T19:06:54Z

Support both opaque (google.protobuf.Any) and
structured (Literal.Struct) encodings for user-defined type literals per Substrait spec.

Split UserDefinedLiteral into UserDefinedAny and UserDefinedStruct
Move type parameters to interface level for parameterized types
Add first-class POJO representation for type parameters
Test coverage including roundtrip tests
Throw exception on unhandled struct-based representation in isthmus

Closes #611

core/src/main/java/io/substrait/extension/DefaultExtensionCatalog.java

benbellick · 2025-11-21T19:58:41Z

FYI, this PR makes no attempt to actually validate if the struct representation provided matches the definition in the yaml file. I think that this is the right thing to do, but it turned out to be slightly more complicated than I thought, as it involves threading an ExtensionCollection through the codebase a bit. Thus, I left it for another issue (#614).

Support both opaque (google.protobuf.Any) and structured (Literal.Struct) encodings for user-defined type literals per Substrait spec. - Split UserDefinedLiteral into UserDefinedAny and UserDefinedStruct - Move type parameters to interface level for parameterized types - Comprehensive test coverage including roundtrip tests - Throw exception on unhandled struct-based representation in isthmus

benbellick · 2025-11-24T16:30:36Z

core/src/main/java/io/substrait/expression/proto/ExpressionProtoConverter.java

        extensionCollector.getTypeReference(SimpleExtension.TypeAnchor.of(expr.urn(), expr.name()));
    return lit(
        bldr -> {
-          try {


This exception doesn't happen anymore because we don't parse the Any here. Instead, we have a reference to the pre-parsed proto directly.

benbellick · 2025-11-24T20:03:20Z

core/src/main/java/io/substrait/type/proto/ParameterizedProtoConverter.java

+    public ParameterizedType userDefined(
+        int ref, java.util.List<io.substrait.type.Type.Parameter> typeParameters) {
+      throw new UnsupportedOperationException(
+          "User defined types are not supported in Parameterized Types for now");


This is consistent with the above, where we don't yet support ParamerizedType conversion.

benbellick · 2025-11-24T20:04:20Z

core/src/main/java/io/substrait/type/proto/TypeExpressionProtoVisitor.java

+    public DerivationExpression userDefined(
+        int ref, java.util.List<io.substrait.type.Type.Parameter> typeParameters) {
+      throw new UnsupportedOperationException(
+          "User defined types are not supported in Derivation Expressions for now");


This is also consistent with the above.

core/src/main/java/io/substrait/type/proto/TypeProtoConverter.java

core/src/test/java/io/substrait/type/proto/LiteralRoundtripTest.java

benbellick · 2025-11-24T20:25:49Z

isthmus/src/main/java/io/substrait/isthmus/expression/ExpressionRexConverter.java

+  public RexNode visit(Expression.UserDefinedStructLiteral expr, Context context)
+      throws RuntimeException {
+    throw new UnsupportedOperationException(
+        "UserDefinedStructLiteral representation is not yet supported in Isthmus");


This is #612

…lcite

vbarua

Started taking a look at this and left some comments. Mostly looks reasonable. Want to come back and look at your tests with fresh eyes, and also think about parameterized types with a fresh 🧠.

core/src/main/java/io/substrait/expression/Expression.java

vbarua · 2025-11-26T01:41:44Z

core/src/main/java/io/substrait/expression/Expression.java

+    @Override
+    public abstract List<io.substrait.type.Type.Parameter> typeParameters();
+
+    public abstract com.google.protobuf.Any value();


Release Notes
Capturing the value as an Any instead of a ByteString does feel nicer ✨

vbarua · 2025-11-26T01:42:24Z

core/src/main/java/io/substrait/expression/Expression.java

+   * @see UserDefinedAnyLiteral
+   * @see UserDefinedStructLiteral
+   */
+  interface UserDefinedLiteral extends Literal {


Release Notes
We should call out that we don't construct UserDefinedLiterals anymore.

vbarua · 2025-11-26T02:13:50Z

core/src/main/java/io/substrait/type/Type.java

+   * parameters (like the {@code 10} in {@code VARCHAR<10>}). This interface provides a type-safe
+   * representation of all possible parameter kinds.
+   */
+  interface Parameter {}


Is the function of https://github.com/substrait-io/substrait-java/blob/main/core/src/main/java/io/substrait/function/ParameterizedType.java applicable here? It feels vaguely related.

🤔 yes that is an interesting point. Looking into it!

So as I understand it, ParameterizedType.java is used for representing abstract types with parameters in yaml files. Where as the Parameter above being introduced is actually a concrete argument passed into the type.

So for example, List<any1> could be a ParameterizedType, whereas List<int32> is a type with parameters [int32].

core/src/main/java/io/substrait/type/Type.java

core/src/test/java/io/substrait/TestBase.java

benbellick · 2025-12-04T16:37:02Z

FYI, I have a WIP PR for implementing this in Isthmus but I split it into two PRs because the code there is more complicated. This keeps the PRs a bit smaller!

github-actions · 2026-01-05T12:59:03Z

ACTION NEEDED

Substrait follows the Conventional Commits
specification for
release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

core/src/main/java/io/substrait/expression/ExpressionCreator.java

nielspardon · 2026-01-05T14:17:12Z

core/src/main/java/io/substrait/expression/Expression.java

+    @Override
+    public abstract List<io.substrait.type.Type.Parameter> typeParameters();
+
+    public abstract com.google.protobuf.Any value();


One observation/remark I would have here is that in other places we have been trying to not expose protobuf objects in the Substrait core POJO API like e.g. in AdvancedExtension we provide the empty Optimization and Enhancement interfaces and ask users of the Substrait Java SDK to implement protobuf conversion logic for those instead. At first glance it feels like we would be a little inconsistent if for user defined any literals we expose the protobuf any value directly.

Hey Niels, this was a great point. I just pushed a commit addressing this by introducing the interface UserDefinedAnyValue. I'll leave the PR open for a little while to give you the opportunity to provide feedback on that component if you'd like. Otherwise, I'll merge it in later today. Thanks for the review!

Actually, after implementing the abstraction pattern similar to AdvancedExtension, I think we should actually defer the change you are suggesting. The abstraction adds an extra layer (UserDefinedAnyValue) without clear benefits. Users still need to work with com.google.protobuf.Any directly to construct values, and the indirection makes the API harder to understand. Since you mentioned this could be revisited later, let's keep it simple for now and use Any directly as the spec defines.

Yes, on that point I also wasn't sure how much additional value the additional layer of abstraction would bring.

nielspardon · 2026-01-05T14:29:33Z

examples/substrait-spark/src/main/java/io/substrait/examples/util/SubstraitStringify.java

   * Explains the Sustrait relation
   *
-   * @param plan Subsrait relation
+   * @param rel Subsrait relation


side note: this fix is also in #642

nielspardon

LGTM

I'm fine with the changes. Just not sure whether we want to revisit exposing the Any proto value directly in a follow-up.

Co-authored-by: Niels Pardon <[email protected]>

This reverts commit 76cd0f0.

benbellick force-pushed the benbellick/handle-structured-udt2 branch from 133b704 to e4f36ea Compare November 21, 2025 19:08

benbellick changed the title ~~Benbellick/handle structured udt2~~ handle struct-based UDT literals in core Nov 21, 2025

benbellick force-pushed the benbellick/handle-structured-udt2 branch from e4f36ea to cef9798 Compare November 21, 2025 19:16

benbellick commented Nov 21, 2025

View reviewed changes

core/src/main/java/io/substrait/extension/DefaultExtensionCatalog.java Show resolved Hide resolved

benbellick force-pushed the benbellick/handle-structured-udt2 branch from cef9798 to f3379c1 Compare November 21, 2025 19:51

benbellick marked this pull request as ready for review November 21, 2025 20:01

benbellick requested a review from vbarua November 21, 2025 20:01

benbellick added 3 commits November 24, 2025 11:19

test: add test to ensure struct-UDTs work with type parameters

f0e13ef

test: improve UDT literal test by making opaque-ness explicit

f5b6341

benbellick force-pushed the benbellick/handle-structured-udt2 branch from 1eb4fb4 to f5b6341 Compare November 24, 2025 16:19

refactor: rename UserDefined{kind} to UserDefined{kind}Literal

8c54706

benbellick commented Nov 24, 2025

View reviewed changes

benbellick added 3 commits November 24, 2025 14:10

feat: first class support for TypeParameters

36c81f0

docs: update docstring to explain use of Parameter

680ed29

test: add test showing all Type.Parameters preserved

6ac7aab

benbellick commented Nov 24, 2025

View reviewed changes

core/src/main/java/io/substrait/type/proto/TypeProtoConverter.java Outdated Show resolved Hide resolved

benbellick commented Nov 24, 2025

View reviewed changes

core/src/test/java/io/substrait/type/proto/LiteralRoundtripTest.java Show resolved Hide resolved

benbellick commented Nov 24, 2025

View reviewed changes

benbellick added 2 commits November 24, 2025 16:45

fix: correctly handle type conversion with paramterized types

b2063d0

refactor: revert to simpler binary construction for any rep UDT in ca…

ce7985f

…lcite

benbellick mentioned this pull request Nov 24, 2025

Benbellick/handle structured udt #609

Closed

fix: ensure nullability preserved in calcite roundtrip

e8cb862

benbellick force-pushed the benbellick/handle-structured-udt2 branch from 78488b4 to e8cb862 Compare November 25, 2025 19:27

vbarua reviewed Nov 26, 2025

View reviewed changes

chore: addressing some PR comments

77f0bd7

benbellick force-pushed the benbellick/handle-structured-udt2 branch from fea5921 to 77f0bd7 Compare November 26, 2025 18:14

chore: addressing more PR comments

4b4f5ee

benbellick force-pushed the benbellick/handle-structured-udt2 branch from 3814053 to 4b4f5ee Compare November 26, 2025 18:26

chore: simplify documentation

3732cf9

benbellick force-pushed the benbellick/handle-structured-udt2 branch from b2742da to 3732cf9 Compare November 26, 2025 18:55

benbellick mentioned this pull request Nov 26, 2025

feat: add structured UDT literal support with dual encoding #623

Draft

Merge branch 'main' into benbellick/handle-structured-udt2

a216f3e

benbellick requested a review from vbarua December 4, 2025 16:18

nielspardon requested review from andrew-coleman and bestbeforetoday December 4, 2025 16:24

benbellick mentioned this pull request Dec 5, 2025

feat: add initial extension support substrait-io/substrait-rs#395

Merged

benbellick added 2 commits December 16, 2025 13:34

Merge branch 'main' into benbellick/handle-structured-udt2

f61f68c

Merge branch 'main' into benbellick/handle-structured-udt2

02f7920

benbellick changed the title ~~handle struct-based UDT literals in core~~ feat: handle struct-based UDT literals in core Jan 5, 2026

nielspardon reviewed Jan 5, 2026

View reviewed changes

core/src/main/java/io/substrait/expression/ExpressionCreator.java Outdated Show resolved Hide resolved

nielspardon reviewed Jan 5, 2026

View reviewed changes

core/src/main/java/io/substrait/expression/ExpressionCreator.java Outdated Show resolved Hide resolved

nielspardon reviewed Jan 5, 2026

View reviewed changes

nielspardon approved these changes Jan 5, 2026

View reviewed changes

benbellick and others added 3 commits January 5, 2026 12:01

Apply suggestions from code review

d009af4

Co-authored-by: Niels Pardon <[email protected]>

add interface for UserDefinedAnyValue

76cd0f0

Revert "add interface for UserDefinedAnyValue"

a60e0b7

This reverts commit 76cd0f0.

vbarua changed the title ~~feat: handle struct-based UDT literals in core~~ feat(core): handle struct-based UDT literals Jan 13, 2026

vbarua approved these changes Jan 13, 2026

View reviewed changes

benbellick merged commit 13309df into main Jan 13, 2026
13 checks passed

benbellick deleted the benbellick/handle-structured-udt2 branch January 13, 2026 19:37

feat(core): handle struct-based UDT literals #613

feat(core): handle struct-based UDT literals #613

Uh oh!

Conversation

benbellick commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

benbellick commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vbarua left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benbellick commented Dec 4, 2025

Uh oh!

github-actions bot commented Jan 5, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benbellick Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nielspardon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

benbellick commented Nov 21, 2025 •

edited

Loading

benbellick Jan 5, 2026 •

edited

Loading