[FR] support integration with other data systems

[zhang2014]

Summary

We need to introduce the Source into catalog for better integrate with other systems.

Source may has the following advantages(different multiple table engine):

• We don't have to mix with the table engine (explicit)
• Allow new query syntax(e.g: CREATE KAFKA SOURCE(columns) WITH brokers='ip1:port;ip2:port', topic='topic_name', format='TSV')

[BohuTANG]

Cool.

1. parser need extend, this is easy
2. catalog provide external data source interface (@dantengsky may has plans to support, but probably later)

[dantengsky]

Looking forward to this feature!
Catalog and datasource are being separated gradually, I think we still need some small iterations for it to be ready.

Some rough ideas :

• Lift local_database to TableProvider (and DatasourceRegistry)
  which can produce Tables by the type of engine, and the Table accesses storage(local fs, ODBC....) according to the table options.
• Remove local database
  Saving all the database meta in a metastore, which can be backended by a remote-metastore or an embedded kv store.

[dantengsky]

@ariesdevil any suggestions?

[ariesdevil]

I think we also need some mechanism to unify the data layer abstraction, such as dataframe API or table API.

[leiysky]

This is a very big scope.

Except supporting DataSource API and unifying the data format, we can also support pluggable optimization rules for different data sources in the future. Thus we can implement computation push down and access method choosing.