add controller package and agent CLI #215
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jayshrivastava
force-pushed
the
js/benchmarking
branch
3 times, most recently
from
80dae61 to
d63b805
Compare
jayshrivastava
force-pushed
the
js/benchmarking
branch
3 times, most recently
from
6858376 to
542eaf2
Compare
This change relates to #214. It adds a new package to this crate called `datafusion-distributed-controller` which will contain control-plane-related tooling for running a distributed datafusion clusters. This utility will be used for large-scale integration tests and benchmarking. This package does not necessarily need to contain rust code. We can The proposed architecture is: - have a local CLI "controller" which can interface with agents running on workers - the agents can run code on the machine (ex. download data from blob store, start arrow flight server, execute sql on the cluster etc.) This change adds an agent. The controller will be added in a future commit. Usually an agent runs on a port and is interfaced with via RPC, but since this utility is just for testing, this change proposes that the agent can be interfaced with via a CLI.
jayshrivastava
force-pushed
the
js/benchmarking
branch
from
542eaf2 to
695c126
Compare
|
Since the new plan is implement the controller in the aws cdk (typescript), we should refactor this so it's just |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change relates to #214. It adds a new package to this crate called
datafusion-distributed-controllerwhich will contain control-plane-related tooling for running a distributed datafusion clusters. This utility will be used for large-scale integration tests and benchmarking.The proposed architecture is:
This change adds an agent. The controller will be added in a future commit. Usually an agent runs on a port and is interfaced with via RPC, but since this utility is just for testing, this change proposes that the agent can be interfaced with via a CLI.