Distributed CLI / Benchmarking Tool

[jayshrivastava]

benchmarking infrastructure with a proper setup based on AWS EC2 for example could prove value

It would be valuable to have a CLI which can instantiate distributed datafusion servers on a collection of remote workers.

It should be able to

• install fixtures on each server (ex. tell each server to download tpch parquet tables)
• issue SQL to a server for distributed execution

Each server should be able to

• recognize fixtures from a directory as tables
• execute distributed queries, measure metrics, and report these metrics back
• optionally stream the results back to the user

Aside: It would be awesome to extend the datafusion-cli to be distributed, but this will require a lot more effort and may not be worth it

We can borrow many of ideas from

• https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachprod
• https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest
  Which do exactly what we need in this issue

[gabotechs]

It would be valuable to have a CLI which can instantiate distributed datafusion servers on a collection of remote workers.

Sounds good!

I think it'd be nice to identify what makes sense to bake into the CLI itself and what makes sense to ship as Infrastructure As Code. For example:

• "install fixtures on each server (ex. tell each server to download tpch parquet tables)": IMO this should be done with IAC and not with a CLI, either terraform or AWS CDK (my preferred option). I'd say that the files should probably live in S3 rather than a disk, as that's how a real setup would look like.
• "issue SQL to a server for distributed execution": This makes sense to ship as a CLI

[gabotechs]

Also, the value of creating such environment for benchmarking is to also be able to run the same queries against other systems like Spark, Ballista or Trino for comparison. That can greatly complicate things, but I'd anyways try to keep it in mind if possible.

[gabotechs]

Closed by #227