Filter rows at source for historic table data #31782
Unanswered
pidanou
asked this question in
Connector Questions
Replies: 1 comment
-
Maybe the question wasn't clear and too complex at first. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi,
I have a table that has a lot of data, I need to sync this table to a new database but I don't require the historic data only the new data from the point I create the connection.
The source is OracleDB and the destination is Postgres
Now to make things worse, since my app is a multi tenant SaaS app, there is an undefined number of OracleDB that I need to connect to.
I saw this question here: https://discuss.airbyte.io/t/filter-data-at-source/4239
It is basically the same use case (+ the fact that I need to make things dynamic to address the multi tenant related issues).
I also saw there was the filter feature here: https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/record-selector
So I should create my custom connector? And have the filter be a dynamic value that will be the connection creation date (if that is even possible)
Is there any simpler way? Using only the API for example?
Thanks a lot!
Beta Was this translation helpful? Give feedback.
All reactions