Tips and tricks for backfilling significant data history with the daily 250 BC download limit

[davewsail]

I'm relatively new to PST and very new to bc-utils so forgive me if this is an ignorant question...

I signed up for BC premier and was happy to start backfilling individual futures contracts. That happiness faded quite rapidly when I used up my daily limit with 4 futures contracts. I was downloading individual contracts back to 2000. It will take me months to download the data I want (think I need) to get my data for PST ready for production. Am I missing something basic?

[bug-or-feature]

Either:

• 1 you are not logged in to the correct Barchart account (read the docs)
• 2 there is something wrong with the Barchart account (contact Barchart)

[davewsail]

I think the problem is that each future that I download has multiple individual contracts per year multiplied by 25 years so yes, my 250 daily limit goes fast. I think I need to rethink how I'm going to do this. My issue is that the preloaded data that comes with PST stops at the end of March 2024 currently. Since I can only download a year of historical data from IB, that leaves me with a gap in price data. I intended to just download all the individual contracts from BC but it seems I'll need to instead splice in data from BC to update the preloaded data and then use IB to continue updating. I'm guessing my issue is an edge case and will need to employ a somewhat manual process to do this. If there is an easier way, I'm all ears.

[davewsail]

For anyone in the future with the same question-- I found what I was looking for here: https://github.com/robcarver17/pysystemtrade/blob/develop/docs/data.md#multiple-prices

There is discussion and code for splicing in data.