Walkforward windows: should each window use an independent Portfolio?

[Pirat83]

Context

While scoping V3 parallel-training work (#231 thread), one design question kept surfacing that deserves its own place to breathe: should walkforward windows share a single Portfolio instance (current behavior) or get independent Portfolio instances per window?

This is a design / intent question, not a bug report. I'd like to understand the reasoning behind the current choice before proposing any change.

Today's behavior

_run_walkforward (strategy.py:1360-1445) threads a single Portfolio instance through backtest_executions for every window. The following state accumulates window N -> N+1:

• cash, equity, market_value, margin, pnl, fees
• orders (deque), trades (deque), bars (equity curve), position_bars
• long_positions / short_positions (open positions cross window boundaries unclosed)
• win_rate / loss_rate / _wins
• Monotonic IDs: _order_id, _entry_id, _trade_id
• _stop_data, _stop_records
• Plus sessions: defaultdict(dict) (lives outside Portfolio but is also window-shared)

Combined with the walkforward_split (strategy.py:672-815) behavior — test periods are contiguous in time, only the train range rolls back — this produces a single continuous trading simulation with periodic model refresh on a rolling training window. Textbook Pardo-style walkforward optimization.

Why I'm raising it — parallelism / distribution

I'd prefer independent Portfolio instances per window because it makes parallelism and distribution straightforward:

• Windows become genuinely independent units of work. You can joblib / concurrent.futures.Executor / Ray / Dask them across cores or nodes without reasoning about shared mutable state.
• No need to order-enforce window execution — windows can finish out of order and still produce correct per-window results.
• Distributed model training fits naturally: pre-compute all train/test index pairs up front, ship each window to a worker, collect results. This is what most ML frameworks assume.

The shared-Portfolio design blocks all of that without significant refactoring.

Open question 1 — is there an intentional reason for the current design?

Before proposing a change I want to understand whether the shared-Portfolio behavior is a deliberate semantics choice or a consequence of the original implementation. Candidates I can see:

1. Realism. A single continuous portfolio reflects how a live strategy would actually run — you can't go back in time and reset your account between retrains. Independent portfolios would produce a set of "what if you'd started fresh each N months" simulations, which is arguably not what a trader wants to see.
2. Compounding as the point of walkforward. If window 1 doubles the account, window 2 deploys 2x the capital with the refreshed model — that compounding IS the realism. Independent portfolios lose that signal.
3. sessions continuity. The sessions dict is designed for users to stash per-symbol state across the execution. If each window got a fresh Portfolio, what happens to sessions? Reset? Preserved separately?
4. Open-position carry-over. A position opened near the end of window N is held into window N+1 and closed there. Independent portfolios would force a decision: force-close at window boundary? Transfer positions with cost basis as of the boundary? Report the P&L of window N with the position still "open" at window end?
5. API continuity. TestResult exposes a single portfolio, positions, orders, trades, metrics etc. Shape change implications if we go independent-per-window.

If any of (1)-(4) is load-bearing, the "reuse one Portfolio" choice is correct and independent-per-window should be opt-in, not the default.

Open question 2 — if independent, how do windows merge back?

Even if we accept independent Portfolios per window, the result eventually needs to collapse into something users can work with. This is an open question for me too:

• Concatenate. Stitch the per-window equity curves / trade lists / order lists end-to-end. But the equity curves won't line up — window 2 starts from initial cash, not from wherever window 1 ended. You'd either accept discontinuities or rescale after the fact.
• Per-window report. Return N separate TestResult-shaped objects plus an aggregate. Users who want the single continuous view run without parallel_windows=True; users who want per-window statistics opt in. This is the ML cross-validation shape.
• Aggregate statistics only. Mean / std of per-window metrics (Sharpe, max drawdown, win rate), no raw per-window curves unless requested. Simpler API; loses per-window detail.

I don't have a strong opinion between these three. The "right" choice probably depends on what question users are asking — walkforward as a realism check vs walkforward as an ML evaluation protocol.

Scope of this discussion

Not asking for code changes today. Asking:

1. Was the shared-Portfolio design an intentional semantics commitment? Which of the reasons above (or another) drove it?
2. If independent-per-window is acceptable as an opt-in mode, what's the expected merge shape on the way out?
3. Anything about the current design I'm missing that would bite an independent-window implementation?

If the answers converge on "yes, go ahead, opt-in, concatenate" (or similar), I'm happy to draft the API shape as a follow-up PR and we can iterate there. If the shared-Portfolio design is load-bearing and should stay, I'll scope V3 to training-only parallelism (#231 option A) and close this out.

Cross-reference: #231 (performance campaign; V3 discussion).

[Pirat83]

Answered authoritatively on edtechre/pybroker#231 by the maintainer:

I would say A. Because conceptually independent windows don't make sense since the windows represent a contiguous time series. The walkforward test simulates running a strategy that must periodically re-train on newer data. As the walkforward proceeds, the previous window's test data is added to the training of the subsequent window. Because the windows are sequential, the portfolio state should be shared throughout.

Closing the loop: V3 proceeds as option A — parallelize only train_models within each window (embarrassingly parallel across model-symbols), keep backtest_executions sequential with the shared Portfolio. No semantics change to walkforward.

Leaving this discussion open as a discoverable record for anyone who finds it searching for the same design question.