[FLINK-37863] add DynamicKafkaSource interface #177
Conversation
|
Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) |
There was a problem hiding this comment.
@lgo Thanks for the PR. Seems good to me although it's still marked as a draft:)
Just left one minor comment. Besides, could you also update the doc corresponding to https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/dynamic-kafka/ ?
| # Metadata service | ||
| # --------------------------------------------------------------------- | ||
|
|
||
| def set_kafka_metadata_service(self, kafka_metadata_service: JavaObject) -> \ |
There was a problem hiding this comment.
What about also introducing some basic built-in KafkaMetadataService implementations, e.g. SingleClusterTopicMetadataService, etc. Otherwise, it would be difficult for Python users to use.
There was a problem hiding this comment.
Sounds good. I'll see what it'll take to make that happen, although I have not yet looked at how much of the Java<>Python bindings work and whether anything special is needed for this.
Any chance you have a pointer to any easy solution for packaging up an arbitrary JavaObject?
There was a problem hiding this comment.
The implementation of Python DynamicKafkaSource in this pull request could be seen as just a wrapper of an arbitrary JavaObject?
|
Regarding being in draft - I'll change that once I get the time to actually try it out versus just some partially baked testing. |
Addresses FLINK-37863, adding the Python interfaces for using the DynamicKafkaSource in Python.
TODO: