support UDTF in pathway.

[tempbottle]

Is your feature request related to a problem? Please describe.
Query external tables in pathway task exec.

Describe the solution you'd like
Solution like datafusion, it support UDTF to query another database. For example, https://github.com/datafusion-contrib/datafusion-table-providers can support many database.

Describe alternatives you've considered

Additional context

[KamilPiechowiak]

Hey @tempbottle,
thank you for a suggestion. However, I'd like to clarify what exactly you're proposing. If you mean operating on multiple rows at once in a single UDF, it is on our roadmap. If you mean the possibility to query other sources than those currently available, you can do that using the python connector. If you mean adding the possibility to write new connectors in rust by external users, it is not currently planned, but we may consider it.

[tempbottle]

@KamilPiechowiak Thanks for your answer. I'll investigate the python connector. Is there any document or example for how to write rust connectors?

[KamilPiechowiak]

Hey @tempbottle,
Sorry for the late answer. There are no docs regarding writing rust connectors. External users are not supposed to do that. If you have a specific connector in mind, you can name it and we'll consider adding it. Or you may create a PR with a new connector (if you think it would be useful for the pathway community) but we don't have any guides. You can only extract information from other connector implementations.