Spec Formalization: Field References

[jacques-n]

As we have started to make reasonable progress on the type system, I'd love to start a discussion on the concepts outlined in the field references section of the specification:

https://substrait.io/expressions/field_references/

The sketched binary representation is here to better help people think through the concepts.

https://github.com/substrait-io/substrait/blob/main/binary/selection.proto

[cpcloud]

Can you clarify the need/desire for Map KeyExpression?

It seems like it's pushing the boundaries of what a field reference is, since it includes a bunch of additional concepts like "simplified regex" and greedy versus non-greedy.

Why can't we represent this operation as a function?

[jacques-n]

Fair feedback. In some ways, making it a function doesn't substantially influence what can be done. The big difference I see is that if you formalize it at the spec level (as opposed to an internal function implementation) it increases the likelihood that generalized optimizations could be done. For example, if a new file format has metadata that reports the list of available keys, one may be able to decide at planning time that a expression resolves to none of them. This could also be done if it was encapsulated in the function but it depends on interpretation of the internals of the function and the arguments of the function (which would be required to be constants).

I'm happy to remove this as first class for now to better simplify. (It could always be added later.) Will hold off on PR until we get more feedback.

[westonpace]

Array slice also seems like it might be a bit of a stretch. Do you have examples of where an array slice could be optimized at planning time? I suppose if you somehow knew the maximum length of a value. But then that would seem to justify something like string-slice as well.