Questions about using 'sysinit/futures/rollcalendars_from_db_prices_to_csv.py'

[SamJohn27149]

Hi everyone,
I've just downloaded contract price date for CHFJPY from 2020 to 2025 by using barchart, and i used the script sysinit/futures/rollcalendars_from_db_prices_to_csv.py to generate roll calendar and used sysinit/futures/multipleprices_from_db_prices_and_csv_calendars_to_db.py to generate multiple prices. However i found that the generated roll calendar only goes to 2025-06-13

~/pysystemtrade$ python /home/zhangxin/pysystemtrade/sysinit/futures/rollcalendars_from_db_prices_to_csv.py
Configuring sim logging
2025-10-24 20:49:10 DEBUG asyncio Using selector: EpollSelector
Will overwrite existing roll calendar are you sure?! CTL-C to aborty
Instrument code?CHFJPY
                     current_contract  next_contract  carry_contract
2020-03-13 05:00:00          20200300       20200600        20200600
2020-06-12 05:00:00          20200600       20200900        20200900
2020-09-14 05:00:00          20200900       20201200        20201200
2020-12-14 06:00:00          20201200       20210300        20210300
2021-03-12 06:00:00          20210300       20210600        20210600
2021-06-14 05:00:00          20210600       20210900        20210900
2021-09-13 05:00:00          20210900       20211200        20211200
2021-12-13 06:00:00          20211200       20220300        20220300
2022-03-14 05:00:00          20220300       20220600        20220600
2022-06-13 05:00:00          20220600       20220900        20220900
2022-09-13 05:00:00          20220900       20221200        20221200
2022-12-13 06:00:00          20221200       20230300        20230300
2023-03-13 05:00:00          20230300       20230600        20230600
2023-06-13 05:00:00          20230600       20230900        20230900
2023-09-13 05:00:00          20230900       20231200        20231200
2023-12-13 06:00:00          20231200       20240300        20240300
2024-03-13 05:00:00          20240300       20240600        20240600
2024-06-13 05:00:00          20240600       20240900        20240900
2024-09-13 05:00:00          20240900       20241200        20241200
2024-12-13 06:00:00          20241200       20250300        20250300
2025-03-13 05:00:00          20250300       20250600        20250600
2025-06-13 05:00:00          20250600       20250900        20250900

and my multiple prices shows that the PRICE_CONTRACT doesn't change to 20251200 from 2025-09-15 to 2025-10-17

It's confused since i got the contract price data till 2025-10-17, i thought the last roll date would be around 2025-09-13. After i check the code i found that in file sysinit/futures/build_roll_calendars.py , the function adjust_to_price_series would ignore the last line of approx calendar which i think is just 2025-09-13

    adjusted_roll_calendar_as_list = _listOfRollCalendarRows()
    idx_of_last_row_in_data = len(approx_calendar.index) - 1

    for row_number in range(len(approx_calendar.index)):
        local_row_data = _get_local_data_for_row_number(
            approx_calendar, row_number, idx_of_last_row_in_data
        )
        if local_row_data is _last_row:
            break

My question is :

1. Since generate correct multiple prices needs correct roll calendar, should i manually add the last line of roll calendar everytime after i generated roll calendar?
2. Is it correct that everytime i update the contract prices data, i should follow the data workflow like below?
   download contract prices data(no matter from barchart or from IB) -->generate roll calendar based on rollconfig and contract prices data -->generate multiple price -->generate back adjusted price

Hopefully someone can answer my question or point out my mistake, thanks in advance.:)

[tgibson11]

I think you misunderstand the purpose of roll calendars.

The purpose of roll calendars is only to generate historical multiple price data. That should be a one-time thing (for each instrument).

1. Since generate correct multiple prices needs correct roll calendar, should i manually add the last line of roll calendar everytime after i generated roll calendar?

Technically, yes. But "everytime" is really just once per instrument.

I don't know why that script would ignore the last entry, but it does seem like you need it to be there. The script was probably written with a different use-case in mind.

2. Is it correct that everytime i update the contract prices data, i should follow the data workflow like below?
download contract prices data(no matter from barchart or from IB) -->generate roll calendar based on rollconfig and contract prices data -->generate multiple price -->generate back adjusted price

No. The normal process would be:

Update sampled contracts --> download contract prices --> generate multiple prices --> generate adjusted prices

Update sampled contracts (sysproduction/update_sampled_contracts.py) determines when new contracts are available or old contracts are expired. It keeps track of what contracts need price data downloaded. If you have your own way of doing this, then maybe you don't need it.

When it is time to roll a contract, you would use sysproduction/interactive_update_roll_status.py to do the rolls.

[SamJohn27149]

To Todd:
Thank you Todd, you have already answered many of my questions, really appreciate it.
You are right. After checking sysproduction/interactive_update_roll_status.py (which i hadn't carefully read before), i found that it helps us roll contracts and directly updates the multiple prices after a roll occurs, rather than using the roll calendar.
But i have a further question, since you said

When it is time to roll a contract, you would use sysproduction/interactive_update_roll_status.py to do the rolls.

If we are trading many contracts in production, how do we know when we should run sysproduction/interactive_update_roll_status.py to roll certain contracts? Does this mean I should run this script everyday to check whether any contracts need to be rolled that day?

To davewsail:
Thank you for sharing your valuable experience, it helps a lot.
Yes, i'm not at the production stage yet. I’m not very experienced with programming, so before moving into production stage, I’d like to walk through and run each of the production scripts step by step, checking for any potential issues along the way. I think this process will also help me develop a deeper understanding of how pst works.
I noticed that you mentioned

2. update all your futures contract data to bring them up to the current date. If you want to use IB data, DON'T use "seed_price_data_from_IB" unless you have no other data for the futures because they will be overwritten (ask me how I know). You want to add to what is already there, not overwrite.

Should we use sysproduction/update_historical_prices.py instead?

[tgibson11]

If we are trading many contracts in production, how do we know when we should run sysproduction/interactive_update_roll_status.py to roll certain contracts? Does this mean I should run this script everyday to check whether any contracts need to be rolled that day?

I run it every day. I think Rob has said he runs it at least once a week.

If you aren't ready for live trading, and aren't updating your price data that frequently, then maybe you'd prefer to do something more like @davewsail is suggesting.

OTOH, you will need to do these things daily/weekly eventually, so maybe you should start doing them now anyway.

Also, as a rule of thumb, sysinit scripts are generally for things that you would do one-time or infrequently - to initialize your system or add instruments, for example. sysproduction scripts are for running your system on an on-going basis - daily or as-needed, but part of regular operations.

[tgibson11]

One more thing...don't fall into the trap of thinking your multiple prices need to follow the roll schedule like clockwork, only deviating due to weekends or holidays.

In live trading, when you have positions that will cost actual money to roll, there are many factors that will cause you to want to roll earlier or later than the roll config would suggest. Rolling precisely on a preset schedule is not necessary or desirable.

[davewsail]

Sam, to your question "Should we use sysproduction/update_historical_prices.py instead?"

It depends. That code is intended to update active contracts. If you have expired contracts that need up to date data, you need to do something else. I ended up making my own version of seed_price_data_from_IB that updates all contracts vs. overwrites. It is a one line of code change. I put the details in the discussion on restarting posting of up to date csv multiple_prices. #1553 (reply in thread) Remember though that IB data only goes back a year or so. Depending on when your existing data stops, you still might have gaps.

[SamJohn27149]

Ｍany thanks, Todd and davewsail, your explanations helped me noticed a lot of things that i didn't even know i didn't know, and it will certainly save me from futures mistakes. :)