[EPIC]: Ballista Reloaded, Roadmap Proposal

[milenkovicm]

Ballista Reloaded - Roadmap Proposal

As it looks like we reached some kind of consensus about moving Ballista from application to a library, I'd like to propose few targets that I see as short to medium term goals for ballista. This would address comments from @alamb & @Dandandan.

Personally, I see two main short term goals, improving ballista usability, and decreasing maintainable code. Robustness may come up as one important goal, for which I don't see bandwidth or infrastructure at this point.

0. Keep up with DataFusion releases

Nothing else to add :)

• Update to DataFusion 43 #1124
• chore: update to DF.44 #1153
• chore: update datafusion to v45 #1176
• chore: update datafusion to 46 #1201

1. Usability

It would be great if we could make writing ballista application as easy as DataFusion, ideally it should be very hard to spot the difference between them.

1.1 BallistaContext removal or evolution

Can we replace BallistaContext with SessionContext? It would definitely improve usability as we would get most of the methods available in SessionContext also, some DataFusion applications would be deployable to Ballista with single line change.

let ctx = SessionContext::ballista_standalone().await?;

This approach may bring DataFusion Python on board as well, not sure how easy would it be.

There are clear benefits of deprecation of BallistaContext, decision may hurt us in a long rung.

SessionContext may bring usability issues with UDF support, configuration and basically all functionalities which need to be propagated across the cluster to work, and which may not be trivial to address. We may try to be address the by "turning off" those methods in ballista or just by documenting it, still some effort is needed. Or maybe its not issue at all?

• Replace BallistaContext with SessionContext #1081
• Ballista Python Update #1091
• Executor configuration extended .. #1099
• Ergonomic way to setup/configure SessionContextExt #1096
• Update Documentation with SessionContextExt #1105

1.2 Scheduler/executor binaries

Ballista to a library should keep scheduler and executors binaries, as they would improve overall ballista usability and provide a quick way to bootstrap ballista cluster, for easy on boarding and testing purposes.

We should focus our effort would be to provide a methods which would would help making custom scheduler/executors binaries easy. We could provide a way to create new scheduler/executor with default configurations, or add a way to plug in object store registries, configurations, protocols, session context factories ...

• Easy setup/customization of Scheduler/Executor #1115
• Expose more configuration options for executor #1098
• Make it easier to build custom scheduler and executor processes that extend Ballista #613
• Remove flight-sql from ballista in 46.0.0 #1227

1.3 Ballista Contrib

Move some of the components which are now optional to a separate sub-projects.

2. Protocol (client - scheduler - executor)

Two protocols we may need to have a look at, client-scheduler and scheduler-executor.
Two major use cases may be support for user defined functions, configuration propagation and replacement of protocol itself.

• feat: executor supports pluggable arrow flight server #1170

2.1 Propagate SessionContext configuration from client to executor

At the moment SessionContext or some other state is not propagated from client to scheduler and executors.
Enabling this would simplify overall configuration, it would enable use-cases where configuration can hold
secret keys, object store configuration or similar.

• configuration cleanup Trim down BallistaConfig  #1108
• update client side (propagate SessionConfig when scheduler_client.create_session called) Trim down BallistaConfig  #1108
• update scheduler side, make session config first class citizen Trim down BallistaConfig  #1108

2.2 Support for user defined functions

I'm not aware of any examples where rust based UDFs are made serializable and shipped from client to server,
many examples where python functions are shipped, so this effort may focus on python UDF. This effort
would probably impact DataFusion plans, more details to follow.

2.3 Make client-scheduler protocol plugable

Current client-scheduler protocol will be improved, also as there are new protocols coming out we may provide
a way to replace default protocol.

One (new) protocol example is Spark Connect, it is well thought approach covering most if not all cases for layered data processing. Users could be able to provide support for it and deploy frameworks like Sail on top of Ballista or even spark applications. Personally I find this interesting and with growing operators from DataFusion Comet supports it might bring interesting possibilities.

Also, this is needed if flight-sql is made optional and moved to 'contrib' project.

2.4 Bridge the gap between datafusion and ballista

Some operations are supported by datafusion but not ballista, we should try to bridge the gap

• LogicalPlan serde is not yet implemented for Dml datafusion#13616
• DDL Statement Propagation (INSERT INTO support) #1164

3. Shuffle improvements

@andygrove mentioned, re-implement the shuffle writer/reader to re-use the logic in Comet which has a more efficient shuffle implementation based on Spark. It would be great if we could see this implemented in short term.

4. Scheduler

Improvements to internal scheduler could be a mid to long term goal, where users can bring their own strategies. Not many use-cases come to my mind apart from HDFS collocation or caching.

Two possible items here:

• pluggable task distribution policies #1238
• Adding/improving Failure detector(s)

5. Observability

As UI has been removed, and rest-api may be moved to contrib API we need to come up with notification mechanism external systems can subscribe to get scheduling events, execution metrics ... We would need to put some more effort to break down this functionality. I guess we could learn from Apache Spark

6. Testing

Effort into getting more tests and covering edge cases. It may not be easy as it needs additional infrastructure and lot of effort for testing. It would be great if we can re-use DataFusion set of tests somehow

7. Python bindings

Make ballista available for datafusion-python users

• added a BallistaContext to ballista to allow for Remote or standalone #1100
• Ballista Python Issue(s) #1142
• Update ballista python dependencies #1132
• Create PyPi Release Process #1120
• Expose Ballista Scheduler and Executor in Python #1107

[drauschenbach]

Re: 2.2 User Defined Functions:

Rust compiles to WASM, and WASM explicitly supports over-the-air deployment. This is one possible way to express UDF's in Rust.

Some other projects allow UDFs in "pickle-able Python", although use of transitive dependencies rapidly descends into Python packaging hell.

[wpf375516041]

Does starrocks java udf have any reference value?

[tbar4]

I think pictures and graphs help me better understand what is being updated, so I made this to reflect what I think you are saying should be updated. I am happy to update and create a proposed Architecture that we can maybe put on the site.

[milenkovicm]

Just one note, core ballista should not care about UDFs, if they can serialise to logical/physical plan ballista should provide extension points to run them. My opinion is that rather than providing specific UDF implementation in core ballista, we should provide a way to plug your own UDF implementation

[milenkovicm]

Does starrocks java udf have any reference value?

@wpf375516041 its not too hard to implement java udf on top of datafusion. I did example implementation some time ago https://github.com/milenkovicm/adhesive

[milenkovicm]

After epic 6 months, I'm closing this issue. I would propose to have further discussion about ballista in #1245

Thank you everyone for your help, support and contribution over the last while.