Parallelize Serialization of Trino Data to RecordBatches #7
Description
Is your feature request related to a problem or challenge?
Serializing trino json-like data record to RecordBatches is likely to be a performance bottleneck.
Describe the solution you'd like
Parallelizing serialization on the Relay so the trino client continues to fetch data while also serializing on multiple cores.
This should be straightforward to tokio spawn serialization tasks, but the complexity may be in creating a RecordBatchStream from an mpsc async receiver of the parallel tasks.
Describe alternatives you've considered
Don't parallelize serialization as network IO may bottleneck anyway, though I find this unlikely if trino connector were to go up against FlightSQL in the same query.
Additional context
No response