Import deeply nested Rust struct/enum with custom types

[v1gnesh]

In DuckDB, when I have a ndjson file, I can use CREATE TABLE t AS SELECT * FROM read_ndjson('file.ndjson');.
I would avoid the interim ndjson step if I can, as the objective is to go from deeply nested Rust struct/enums to the Arrow world, as transparently as possible.
Currently, I'm serializing BigStruct to ndjson String, one at a time, and then writing out the ndjson file (8x the size of the source file). Then, using DuckDB's SQL above, I'm able to automagically get the data types back from plaintext (JSON).

It is very desirable to go directly from Rust data types to inserting the native struct data type of DataFusion/DuckDB, etc.
Note that I don't have a Vec<BigStruct>; BigStructs are being produced in an async Stream.
I understand this could be an ArrayOfStruct to StructOfArray 'problem', but I don't have an ArrayOfStruct to begin with, as they are produced in a streaming fashion (too many to keep it all in memory).
In addition to this example of writing JSON into DuckDB not working (it just writes the hex bytes in decimal), I lose all type information (read_ndjson via the CLI recreates all of it though), native support for Rust data types is a work in progress.

Do you think this is something that is good to have for DataFusion, and if so, is it something in the works already?
Are there any examples I can look at?

Oh, and inferred schema would be best. The BigStructs are quite big, and conceal a whole lot of variations. It would be a nightmare to write the schema for all of them.

Thanks in advance.

[tustvold]

but I don't have an ArrayOfStruct to begin with, as they are produced in a streaming fashion (too many to keep it all in memory

You could perhaps buffer up the DF target batch size number of structs, e.g. ~8000, and this ArrayOfStruct to StructOfArray

You could possibly make use of https://docs.rs/arrow/46.0.0/arrow/#serde-compatibility to do this, or hand-roll your builders as shown https://docs.rs/arrow-array/46.0.0/arrow_array/builder/index.html#nested-usage.

If you're only interested in converting to arrow, you shouldn't need anything as heavy as a query engine like DuckDB or DataFusion

Oh, and inferred schema would be best. The BigStructs are quite big, and conceal a whole lot of variations. It would be a nightmare to write the schema for all of them.

I'm not sure there is a way around this, there is some schema inference logic but it is very limited in scope.

enum

Enums are quite tricky to map to arrow, there is support for UnionArrays but the support for these is extremely limited across the ecosystem

[tustvold]

theoretically 'impossible' thing

Self-recursive data structures, i.e. trees, are the only ones that I'm aware of being impossible in arrow. Otherwise the support just lessens as you get further away from a tabular model, to the point where things like UnionArray simply aren't supported by parquet

as long as I don't have to construct the schema upfront

I'm afraid I'm not aware of a way to do this with the currently available APIs

[v1gnesh]

Okay, thanks for clarifying.
I see that Avro also needs schema upfront.
It's a shame (for me) that I see the Arrow ecosystem evolving excellently, and can't experience its benefits, trying to move data from row-land.

[tustvold]

I'm not sure it is so much a question of row-oriented vs columnar, avro is a row-oriented format after all, rather it is about having well-defined schema up-front vs semi-structured, record-oriented data models like JSON. Not sure that really helps you though... I personally find it hard to envisage a data pipeline without strongly defined schemas, but that is probably just my background 😅

[v1gnesh]

There is a schema, but it's so... detailed... that I don't want to repeat myself (once for building up the structs/enums, then again for the schema).

[tustvold]

I've wanted a proc-macro for this for a while but never got around to it, it might be an interesting project if I could tempt you? 😅 If not I may get around to it at some point, but not sure when