Trino connector to downstream trino

[naude-r]

hi,

we need to connect to downstream trino instances provided by different teams. the trino-gateway it not a 100% fit as we need to join between different tables from the various instances.

is there a trino-to-trino plugin available that will allow a trino instance to connect to other trino instances? should we consider a different solution?

[hashhar]

Trino itself doesn't have something like this but some vendors have a connector like this (disclosure that I work for Starburst). See https://docs.starburst.io/latest/connector/starburst-stargate.html

[hashhar]

@mosabua might know if there are alternatives.

[mosabua]

I am not aware of alternatives but know that there are other internal versions available at other companies as well. To create such a tool yourself you could combine the example jdbc connector with a trino jdbc driver. This would get you to some initial working prototype .. and then the details are where it gets tricky in terms of security, performance and so on.

See https://github.com/trinodb/trino/tree/master/plugin/trino-example-jdbc

The other approach of course is that you create a cluster that contains all relevant data sources as catalogs .. and yes.. that can have performance, network traffic and cost issues. Some of this could be mitigated by using this cluster only for these workloads and maybe even reducing these by changing them from regular run analytics to ETL workloads that move the relevant data into one shared location and then do analytics there.

[bitsondatadev]

IIRC, @dprophet and team have a Trino to Athena connector that could probably be easily made Trino to Trino similar to what Comcast has. Comcast has had theirs for a while, but it's got a lot to tease out before it is ready for open source so may be something to ask Erik and team if that can go on the roadmap. I think they were interested in open sourcing as much as they can from Bloomberg so they don't have to manage it any more. They are in the works of open sourcing Snowflake and recently finished the first round of Trino gateway.

[naude-r]

Thank you everyone for the answers and direction! It was really helpful.

[dprophet]

Yes. I suspect we will be working on, what I have been calling, a Trino Mesh. The most important piece of this is a trino-trino connector.

we need to connect to downstream trino instances provided by different teams.

This is also a nice way to isolate compute between teams.

I can confirm we dont have it but we do share similar use case/needs as yourself.