File name and folder restructure

[gabotechs]

This PR provides a better organization for the project colocating into modules the different pieces based on the features they are providing.

The new folder structure, as it colocates things related to each other more efficiently, allows to tweak the privacy of some structs/methods so that they are not expose to outside unrelated modules, and therefore reducing the chances of future coupling.

All the code is mostly untouched, only a couple of code changes are shipped for making this possible:

• The ExecutionTask now has two forms:
  • ExecutionTask: struct with the Url already parsed and the partition groups already as usizes, so that we don't need to do juggling in the code for parsing Urls or casing u64s to usizes
  • ExecutionTaskProto: the protobuf representation, decoupled from the actual useful struct
• The test in the do_get method was using directly the proto representation of the ExecutionStage and the ExecutionTask, but they are no longer exposed to the outside, so instead the normal structs are used, which then are converted to protos with the proto_to_stage function. This actually results in a nice cleanup that saves some lines of code.

common/
    mod.rs
    composed_extension_codec.rs
    ttl_map.rs
errors/
    mod.rs
    arrow_error.rs
    datafusion_error.rs
    io_error.rs
    objectstore_error.rs
    parquet_error.rs
    parser_error.rs
    schema_error.rs
execution_plans/
    mod.rs
    arrow_flight_read.rs
    partition_isolator.rs
    stage.rs
flight_service/
    mod.rs
    do_get.rs
    service.rs
    session_builder.rs
protobuf/
    mod.rs
    distributed_codec.rs
    stage_proto.rs
    user_codec.rs
test_utils/
    mod.rs
    insta.rs
    localhost.rs
    mock_exec.rs
    parquet.rs
    tpch.rs
lib.rs
channel_resolver_ext.rs
config_extension_ext.rs
distributed_ext.rs
distributed_physical_optimizer_rule.rs

[NGA-TRAN]

Nice refactor and rename.

[NGA-TRAN]

I like you rename it to distributed_physical_optimizer_rule. Make it clearer

[gabotechs]

yeah, I've tried to rename files to be a bit more consistent with the content.

[NGA-TRAN]

StageExec makes it more like DF style 👍

[gabotechs]

Yeah, it seems like an unwritten rule to suffix all ExecutionPlan implementations with *Exec

[NGA-TRAN]

This isn’t directly related to this PR, but might be something we consider updating or adding in a follow-up.

This RoundRobinBatch got me thinking 🤔. On one hand, it's a valid scenario we want our tests to cover. On the other, round-robin repartitioning isn’t ideal from a performance standpoint. How about we introduce a variety of partitioning schemes in our mock tests? That way, we can surface more realistic and appropriate cases while still including edge scenarios like this one.

[gabotechs]

Yeah, we should do something with RoundRobin repartitions...

We probably should extend our parquet files test datasets so that we can cover more cases and more esoteric distribution patterns.

For example, it would be nice to have two test parquet files with related data that we can use to stress joins in weird ways.

[gabotechs]

Probably something to follow up soon