Clarify usage of advertise_flight_sql_endpoint

[sebbegg]

Hi,

I hope creating an issue for this is okay...
Is there some more documentation / an example available for the use of the advertise_flight_sql_endpoint scheduler config?

I was trying run ballista in docker-compose and connecting from the outside, with code based on https://github.com/milenkovicm/ballista_delta.
When fetching results, ballista seems to connect to the executor that produced the result.

My understanding of the docs on advertise_flight_sql_endpoint is that this would allow fetching the result from the scheduler?
Just setting advertise_flight_sql_endpoint to "localhost:12345" didn't show any effect.
Is there any example available that demonstrates how this is intended to be used/configured?

I also tried searching the codebase of usage of this config - but actually only found it used in the config.

[milenkovicm]

It looks like it was added in 926605e as a configuration option only, I'm not sure if there was follow up implementation afterward. Ballista had some source code / functionality trimming in last year or so, and it could be that functionality was removed, but I really don't think so as it would be useful option to have.

So we can either remove this option from source as it is not used, or if someone would like to contribute the functionality that would be great as well

[sebbegg]

Thanks for the clarification (and the "fix").
Generally speaking I'd be happy to maybe contribute, but with the exception of experimenting with datafusion/ballista I'm all new to rust.
This doesn't feel like a good first issue though.

If you had some rough guidelines on where this proxy should go, I'd be curious to do some research.

[milenkovicm]

It might not be too complicated after all.

I believe to facilitate this functionality a new GRPC methods should be created, we could have default implementation located at scheduler GRPC endpoint (which might now work well in all cases).

We would need to adopt the scheduler protocol to advertise SQL endpoint as well. Based on this information ballista client on the client side could request data from the "scheduler" which would use ballista client to connect to executors.

let me know if you need any help