Built a real-time observability user interface

[gabotechs]

Some distributed query engines have an interface for visualizing in real time what's happening in the cluster:

• in-flight queries
• available workers
• load on each worker
• resource utilization
• etc...

It would have been very useful to have something like this for #240, as I pretty needed to fill the whole code with debug statements and deploy it to the EC2-based benchmarking cluster to verify what was happening.

I've seen this be implemented as graphical UIs in other systems, that can be accessed by a browser, but I had a very good experience with https://github.com/tokio-rs/console for debugging issues in this project, so a terminal-based UI could be really useful and could be very remote-environment friendly.

[JSOD11]

The above architecture could work for this project.

A worker node receives a query. It creates the distributed query plan and fans out tasks to other workers.

To start the distributed datafusion observability UI, a user opens a separate terminal and enters the start command. The terminal UI (TUI) attaches to the worker node that received the query and begins polling for data on some regular interval (say, every 500 ms). We use ratatui to display live observability data to the user.

Having the TUI program speak only with a single worker node leads to a clean interface between the tool and the workers but means that the coordinator-worker will need to aggregate the relevant observability data from the other workers. To achieve this, the workers could publish this data to the coordinator-worker in a push-based manner as they take actions. There is already some telemetry tooling available in the project that could be helpful here. We'll need to figure out the communication details and take care to ensure there is no harm to performance.

I'll be managing an intern arriving after the new year and think this project with this sort of architecture could be a great fit. A project in this realm is technically deep, requiring contact with Rust, query engines/DataFusion, our new distributed version of DataFusion, and so on, all in addition to an intern's general onboarding and learning. This specific project has the benefit of being high impact but low urgency, green-field, and expanded or shrunk depending on the direction the intern would like to take it, allowing them to also express their own personality and creativity through it. It'll also be great for an intern to have a tangible open source product at the end that we can demo and present clearly to others, and progressing through this work can clearly lead to a coherent understanding of the project and a strong new contributor.

Claude-generated mock-ups, shown below, give us a sense of what a final product could look like with the aid of ratatui.

[gabotechs]

Sounds good! although I'm going to try to push back on the decision of having a single worker be central proxy over which updates are streamed.

Without a strict definition of what "clean" means, I'd say that having the TUI communicate to each worker treating them all as equals fits better the no coordinator/worker split, and will prevent us from writing extra proxy code within each worker.

For example, a cluster with workers A, B and C, if we had a worker be an intermediate proxy, we would need to:

• Connect to worker A from the TUI
• Let worker A connect to B and C
• Stream updates from B and C to A
• Stream updates from A to the TUI

But if we let each worker be a normal worker and not be special, we can just:

• Connect to worker A, B and C to the TUI concurrently
• Stream updates from workers A, B and C seamlessly to the TUI

Some other disadvantages of having a single worker be the proxy:

• If the worker that act as proxy is deadlocked, you lose visibility to the whole cluster
• The proxy workers acts as a network bandwidth bottleneck. If there's a cluster with 10K nodes, there are 10K data streams that need to pass through a single worker

[JSOD11]

All great points — so to confirm my understanding, we modify the design to look like this?

I like this better as well for all the reasons you mentioned.

Any other thinking to do here regarding the high-level architecture? I'm trying to think of any other open questions or things to consider but feel like the design in broad strokes pretty much makes sense.

Once we reach agreement here, the next step will be for me to sit down and begin to map out a more detailed development path with tasks and sub-tasks, and then work with the intern once they arrive to define that further and begin prototyping.

[gabotechs]

I think that was my only comment. Everything else looks pretty well thought and the tools suggested look like the correct ones 👍.

I imagine several challenges will appear along the way, but the proposal is as good as it can get at this stage.