-
Notifications
You must be signed in to change notification settings - Fork 14
Closed
Description
I noticed in #205 that we configure FlightDataEncoderBuilder without setting with_max_flight_data_size. I believe this means that any RecordBatch larger than 2MB gets split up to be under 2MB. I imagine this adds a lot of overhead for larger datasets. I think it would be best to set that to be unlimited and instruct users to configure their gRPC server/clients to allow larger messages. I'm not sure why gRPC has a default 4MB message size, I'm guessing it's for security in public facing APIs, but datafusion-distributed is intended to be all internal communication so I think it's fine to have uncapped message sizes.
Metadata
Metadata
Assignees
Labels
No labels