Download dynamic URL

[jamesmstone]

Hey!

I am brand new to pg_timetable and still trying to wrap my head around it all. So far it seems like a really nifty tool.

One thing I haven't been able to determine is how to have dynamic chain_execution_parameters.

One of the reasons I think this would be needed / useful is downloading a daily file from a URL that contains the date.

e.g. https://example.com/api/YYYY-MM-DD where YYYY-MM-DD is today's date.

I don't think it is possible to dynamically set chain_execution_parameters, so I can think of two possible ways to get around this.

1. have a task chain that:

   1. Updates the chain_execution_parameters of the download task
   2. Downloads the file
   3. Inserts the file

2. have a task chain that runs daily which sole job is to create another (self destructing) chain that downloads and inserts that day's files.

I would be curious how you would tackle this?

Cheers

[pashagolub]

hey!

Thanks a lot for your kind words. Yeap, the second solution seems to be straightforward. :) I'll try to think about this a little bit more. Having some kind of reference to a value seems to be a cool idea, e.g. "take this value from that table.column/GUC/function"

Regards

[jamesmstone]

Hey thats an awesome idea of being able to pass a value along a task chain.

I guess every task would need the concept of an in and an out.

For SQL tasks I guess that could be a parameter and the return value (maybe always cast to TEXT ?)

For PROGRAM tasks I guess that could be STDIN and STDOUT

For builtins, that gets a bit harder...

---

I guess another way of doing this is via a reference table, that all tasks get past to it.

[pashagolub]

SQL task is more flexible. It can read/write if it wants to. But built-in and program are tied to the configuration and need to accept different input parameter sources.