Best architecture implementation for Stream and IncrementalStream for same Soure #38114
icychocolate98
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Good afternoon people !
I'm creating a new connector, is a basic Source that will support full-refresh and incremental one. My doubt is mainly about the architecture or how to invoke both Streams.
I'm creating the module
streams.py
where I define 2 Streams. My first Stream was created to process full-refresh mainly, asFirstStream(Stream)
as a common Stream and another one that inherits from FirstStream so it's something like this:IncrementalStream(FirstStream, IncrementalMixin)
.At the final my first Stream does nothing, so all logic to distinguish between full-refresh or incremental is implemented on IncrementalStream.
Then, in my source module, I only instantiate Incremental one, and inside this, I create some logic to perform incremental read or not in read_records
Slices have a field 'event_date' that should be my incremental field.
I do not know if I should return both streams in source.py module, only incremental ( as it inherits from common one), or what's best for a source of this type.
All help is welcome, thank you !
Beta Was this translation helpful? Give feedback.
All reactions