Investigate use of comet shuffle

[milenkovicm]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Can it be investigated if use of comet shuffle will benefit ballista? If it would can we share the implementation. More details in #1245

https://github.com/apache/datafusion-comet/blob/main/native/core/src/execution/shuffle/shuffle_writer.rs

Describe the solution you'd like

Describe alternatives you've considered

Additional context

• Proposal for more efficient disk-based shuffle mechanism #660