feat: add initial extension support #395

wackywendell · 2025-09-15T19:42:24Z

Summary

Begins addressing #367 and #342 by adding support for parsing the types in YAML Simple Extension Files into Rustic types - with validity enforced. This includes a string text parser handling built-in types, compound types, named structs, custom types, and validated parameter constraints in the Simple Extension YAML files.

Scope

Types-only: no functions or call validation yet.
Public API exposes parsed types (ExtensionFile, Registry, CustomType, ConcreteType) and enforces validation of those on creation / read.

Key Changes

Type system
- New BuiltinType, CompoundType, ConcreteType, CustomType with Display/round‑trip support for alias and named‑struct structures.
- Parameter constraints: data type, integer (with min/max), enum (validated/deduped), boolean, string.
- Parsing to and from the YAML structures (TryFrom<TypeParamDefsItem>, Parse<RawType>)
File/Registry type: abstraction for handling YAML files
Context and proto glue
- Separates out ProtoContext from Context, to distinguish between things needed for Protobuf parsing (ProtoContext)
Type expression parser
- Parses simple, user‑defined (u!Name) and type variables; visits extension references for linkage bookkeeping.
Build/CI
- parse feature includes serde_yaml; include!(extensions.in) is gated behind extensions feature.
- Aligns actions/checkout to v4, updates Cargo dependency set, and bumps the substrait submodule.

Compatibility Notes

New trait bound ProtoContext on proto parsing that previously required only Context.
extensions.in now compiled only with features=["extensions"].
Minimal, types-only round‑trip implemented; other sections remain empty when converting back to text.

Testing

New unit tests cover:
- Type parsing and round‑trip for alias and named‑struct.
- Parameter constraint handling including enum validation and integer bounds (with current truncation behavior).
- Registry creation and type lookup; core registry smoke test behind features=["extensions"].

benbellick

Left a few comments but still haven't gone through most of the PR. Just wanted to flush what I have so far, but I will review more later. Thanks!

Cargo.toml

src/parse/context.rs

benbellick

Left a few more comments, but excited to get this in! Sorry that this PR has been sitting so long 😅

src/parse/text/simple_extensions/registry.rs

src/parse/text/simple_extensions/file.rs

src/parse/text/simple_extensions/type_ast.rs

src/parse/text/simple_extensions/parsed_type.rs

Refactored ProtoContext trait (removed), by replacing with a concrete type. Some other small refactoring simplifications and code cleanup.

benbellick

Left a few more comments. Had to run so I didn't get to entirely finish the review but will revisit later. Thanks

src/parse/text/simple_extensions/types.rs

benbellick · 2025-12-05T19:14:20Z

src/parse/text/simple_extensions/types.rs

+        /// Sub-second precision digits
+        precision: i32,
+    },
+}


Thank you for these wonderful comments 🙏

src/parse/text/simple_extensions/types.rs

src/parse/context.rs

src/parse/proto/mod.rs

src/parse/proto/version.rs

src/parse/text/simple_extensions/file.rs

Add TODO referencing Substrait type proto for unsupported type parameters not yet implemented.

Use a single type for non-recursive builtin types.

Also change get_simple_extension_urn to just return an option, and update some comments.

benbellick

I once again have to run before getting to fully finish everything. I actually feel quite confident in everything except for the types.rs file, which is the one file I just didn't get to finish going through. But everything is looking great!

Thank you for bearing with my (sometimes excessive) comments 😄

src/parse/text/simple_extensions/parsed_type.rs

src/parse/text/simple_extensions/types.rs

src/parse/text/simple_extensions/extensions.rs

src/parse/text/simple_extensions/type_ast.rs

benbellick · 2025-12-09T21:32:10Z

src/parse/text/simple_extensions/type_ast.rs

+
+    #[test]
+    fn test_user_defined_and_parameters() {
+        let cases = [


awesome tests, these inspire a lot of confidence in the parser.

src/parse/text/simple_extensions/types.rs

benbellick

Few more comments, thanks!

benbellick · 2025-12-10T19:25:33Z

src/parse/text/simple_extensions/types.rs

+                parsed_type.visit_references(&mut link);
+                let concrete = ConcreteType::try_from(parsed_type)?;
+
+                // Structure representation cannot be nullable


Is it necessarily the case that calling parse on RawType is always for parsing structure? If it is, It isn't entirely obvious to me from looking at this code block alone. For example, consider the below excerpt from an extension file:

urn: extension:io.substrait:functions_boolean scalar_functions: - name: or description: ... impls: - args: - value: boolean? name: a variadic: min: 0 return: boolean?

would we then later need to call this parse implementation on "boolean?"?

Ah I misunderstood. The structured representation of a type always means that it is non nullable. Marking this one as resolved. Thanks!

Sorry, actually unresolving this one. I believe this is incorrect. My read of the schema is that: a string representation can be nullable, but not the object based representation (which is the case below).

Ah, ok, I looked at this more carefully, and I think the check was in the wrong place, and I moved it.

My interpretation was that if a structure is specified, it cannot ever be nullable, whether specified by string or YAML - because the type itself is the non-nullable version, and the nullability should always go outside the type. As in, if I define a type count with structure i64, then count? is the same as i64?; but if I define count: i64?, then... what is count?? Is that a nullable nullable i64?

I may be over-thinking / over-interpreting this, but see the test / its comment in 02227c4.

Repeating the schema definition here:

$defs: type: oneOf: - type: string # any data type - type: object # syntactic sugar for a non-nullable named struct

Ah, my interpretation is different. Any function can return any type, so for example we can have a function return STRUCT?<f1:i32?, f2:i64> (a nullable struct). But someone introduced this "structured" representation in yaml files merely as syntactic surgar, so that you can do:

structure: # equivalent to struct<i32?: f2:i64> f1: i32? f2: i64

But there just isn't syntax available ATM to say whether or not the above struct is nullable. So the convention is that if you optionally choose to use the structured representation, then you just can't have a nullable struct simply because it cannot be expressed. But if you wanted your function to return a nullable struct, that is perfectly valid. You should have to use the string-based representation.

That is my understanding at least 😅

https://substrait.io/types/type_classes/#communicating-user-defined-types

substrait-io/substrait#920

After some discussion, I think we have a good plan now:

There is some ambiguity in the spec to clarify; this is now split out in docs: clarify nullable-ness of string representation of UDT structure substrait#920.

In the meantime - I'll remove this check from the code, and put in a comment pointing to this issue; we should clarify the spec before adding enforcement.

Done - see latest commit. I removed the relevant test, as it asserted that a nullable type should not be allowed, which it now is. I could have changed it to #[ignore] instead, but it didn't seem worth keeping around.

src/parse/text/simple_extensions/types.rs

Replace Parse impls for proto versions with TryFrom, as they don't need a Context.

Applied some good suggested changes from code review.

Addressing more PR comments.

Use IndexMap for YAML object parsing to keep simple extension struct fields in declared order and flag duplicates, aligning with Substrait expectations.

benbellick

Now just a few more small comments, but I think after this it is good to go! Thanks for your patience 🙇

src/parse/proto/version.rs

src/parse/text/simple_extensions/file.rs

src/parse/text/simple_extensions/types.rs

benbellick · 2025-12-11T21:38:36Z

src/parse/text/simple_extensions/types.rs

+                parsed_type.visit_references(&mut link);
+                let concrete = ConcreteType::try_from(parsed_type)?;
+
+                // Structure representation cannot be nullable


Sorry, actually unresolving this one. I believe this is incorrect. My read of the schema is that: a string representation can be nullable, but not the object based representation (which is the case below).

src/parse/text/simple_extensions/types.rs

Addressing PR feedback by tightening the API and docs.

…arse Disallow nullable structure representations when parsing SimpleExtensionsTypesItem so errors surface before conversion, and adjust tests to cover full item parsing.

# Files: # src/parse/text/simple_extensions/types.rs - reject nullable structure types; add test # Edit header/body then save/quit to commit

benbellick

LGTM!! 🎉 thanks for bearing with all of my reviews :)

benbellick · 2025-12-17T22:13:44Z

src/parse/text/simple_extensions/registry.rs

+//!
+//! This module provides registries for Substrait extensions:
+//! - **Global Registry**: Immutable, reusable across plans, URI+name based lookup
+//! - **Local Registry**: Per-plan, anchor-based, references Global Registry (TODO)


nit (and probably opinionated): I wonder if it is worth dropping the line about Local Registry? I could imagine this being a different struct/file in the future. But I'm not bothered by just leaving it in.

wackywendell and others added 15 commits August 12, 2025 16:38

feat: ExtensionRegistry, wip

076998e

wip compiles

57e5dee

working through some implications

68f0e7d

Switched types over, compiles

dfef7ee

Stripped down to just types

facb09b

Enable test only with correct feature flag

5a5ac23

Code cleanup

7869676

Merged with existing parse

bc2a86b

Mostly working but builtin types are a bit messed up

1b49a6c

now looks better

6dc4302

On the way, trying to get validation working

9c593a1

Extensions should be included when the extensions feature is enabled

7a8584e

Delete some dead code

ac447d8

Start merging argument and types

9b47219

Some updates to type handling; compiles and passes tests

8a44db4

wackywendell marked this pull request as draft September 15, 2025 20:20

wackywendell mentioned this pull request Sep 15, 2025

Feature: Add a 'Rustic' Ergonomic and Validating Layer #367

Open

wackywendell added 13 commits September 15, 2025 17:20

Merge remote-tracking branch 'upstream/main' into parse-extensions

47fbf18

Update to match URN change

5b1e5df

Some renames

6444f5c

Update tests

162e8ab

Removed string / enum type parameter, that's not a thing

421d957

Time should be primitive, precisiontime parameterized

46b6468

Enforce ranges on fixed-range types

83a4249

Enforce float / int bounds

f415dc8

Validate missing types

0cbfc34

Add error for type variations

93b41ef

Fix casing on type display, add tests

470aa00

Stable field ordering for structures

202747a

fix(parse): apply Substrait identifier rules for CustomType names

ac479d9

wackywendell marked this pull request as ready for review October 20, 2025 15:32

benbellick reviewed Dec 2, 2025

View reviewed changes

Cargo.toml Show resolved Hide resolved

src/parse/context.rs Outdated Show resolved Hide resolved

src/parse/context.rs Outdated Show resolved Hide resolved

benbellick reviewed Dec 2, 2025

View reviewed changes

wackywendell added 3 commits December 4, 2025 18:55

fix: refactor to address code review

145500c

Refactored ProtoContext trait (removed), by replacing with a concrete type. Some other small refactoring simplifications and code cleanup.

fix(parse): error on duplicate simple extension URNs

5eed40d

fix: minimum supported rust version

d3e3627

wackywendell force-pushed the parse-extensions branch from 5fae289 to d3e3627 Compare December 5, 2025 01:43

test(parse): table-driven tests

2a43134

benbellick reviewed Dec 5, 2025

View reviewed changes

wackywendell added 3 commits December 8, 2025 14:06

chore: note missing type parameter variants

1a59ee4

Add TODO referencing Substrait type proto for unsupported type parameters not yet implemented.

refactor(parse): unify builtin type handling

5d7abfa

Use a single type for non-recursive builtin types.

refactor(parse): limit parsing traits to crate scope

bb9be17

Also change get_simple_extension_urn to just return an option, and update some comments.

benbellick reviewed Dec 9, 2025

View reviewed changes

benbellick reviewed Dec 10, 2025

View reviewed changes

wackywendell added 3 commits December 11, 2025 11:02

refactor(parse): use TryFrom for version parsing

a2a48ca

Replace Parse impls for proto versions with TryFrom, as they don't need a Context.

refactor(simple_extensions): rename builtin type enum and tighten docs

ffba4de

Applied some good suggested changes from code review.

refactor(parse): streamline type parameter conversion

09b7939

Addressing more PR comments.

wackywendell force-pushed the parse-extensions branch from 9853681 to 09b7939 Compare December 11, 2025 17:18

wackywendell added 2 commits December 11, 2025 13:58

fix(parse): preserve named struct field order

9efe984

Use IndexMap for YAML object parsing to keep simple extension struct fields in declared order and flag duplicates, aligning with Substrait expectations.

refactor(parse): rename parsed_type module to type_ast

eb35933

benbellick reviewed Dec 11, 2025

View reviewed changes

wackywendell added 3 commits December 15, 2025 17:22

refactor(parse): tighten extension visibility and tests

7ba843b

Addressing PR feedback by tightening the API and docs.

fix(parse): reject nullable simple extension structures during item p…

02227c4

…arse Disallow nullable structure representations when parsing SimpleExtensionsTypesItem so errors surface before conversion, and adjust tests to cover full item parsing.

fix: remove assertion not backed by spec, and reference issue.

31d6295

# Files: # src/parse/text/simple_extensions/types.rs - reject nullable structure types; add test # Edit header/body then save/quit to commit

benbellick approved these changes Dec 17, 2025

View reviewed changes

benbellick merged commit f097b31 into substrait-io:main Dec 18, 2025
12 checks passed

github-actions bot mentioned this pull request Dec 18, 2025

chore: release v0.62.3 #438

Open

This was referenced Dec 19, 2025

Add Extension Registry #342

Closed

function registration in extension registry #439

Open

benbellick mentioned this pull request Jan 15, 2026

scalar functions in extension registry #445

Closed

feat: add initial extension support #395

feat: add initial extension support #395

Uh oh!

Conversation

wackywendell commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

Key Changes

Compatibility Notes

Testing

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

benbellick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wackywendell Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wackywendell commented Sep 15, 2025 •

edited

Loading

wackywendell Dec 15, 2025 •

edited

Loading