Feasibility for a customized incremental sync for Database to FileSystem

[ShahBinoy]

Hi @flarco, I am looking for ways on how I can perform incremental fetch of data from DB to Parquet, till the feature #134 is resolved ?

So below is the use case:

• First day run, select * from table where date < today().. It basically gets all data to parquet
• Second and further day run. select * from table where date between (today() and (today() - 1))
• Use SQL date math of some kind embedded in the sql
  OR
• Each Day run assisted by ENV Variable select * from my custom_query_table where date between ${START_DATE} and ${END_DATE}
  • Here I can inject tokenized START_DATE and END_DATE during sling run time as environment variables

Would either of approach be feasible ?

I understand I can inject the dates into a dynamic SQL string by my own, but I was inquiring if existing tokenization for sql statements will allow for keywords other than {incremental_where_cond} or {incremental_value} like {START_DATE} etc to be interpolated from environment variables

[flarco]

Hi yes, you can inject any vars via the env key.

Also, the run-time vars that you can use are documented here.

The second method could work better, with between ${START_DATE} and ${END_DATE}. just use a replication default custom sql with run-time vars as well env vars, so it renders properly for all stream children. Feel free to share the config if you're having issues.

#134 is still on roadmap, just not a priority atm. Focusing on overall stability and steady connector addition.

[ShahBinoy]

Oh wow, not sure how I missed that. So it is feasible and recommended as documented. Thank you for your recommendation