Merge pull request #978 from Lumiwealth/version/4.4.55

v4.4.55 - IBKR day lookup fixes and tearsheet metrics hooks

Author: grzesir

CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 4.4.55 - 2026-03-15
+
+### Added
+- `BACKTESTING_PARAMETERS` environment variable support for parameter injection in backtest runs.
+- Machine-readable `*_tearsheet_metrics.json` artifacts (summary-first) with placeholder output on insufficient/degenerate returns.
+- New strategy lifecycle hook `tearsheet_custom_metrics(...)` for appending custom metrics to tearsheet HTML and JSON artifacts.
+- Regression coverage for multi-timeframe day-timestep stock lookup and tearsheet metrics/custom-hook passthrough.
+
+### Changed
+- Backtest analysis and trader APIs now accept `tearsheet_metrics_file`; default output filename is `*_tearsheet_metrics.json`.
+- QuantStats `metrics_json` generation now runs in `summary_only` mode and forwards custom metrics to both HTML and JSON outputs.
+- Documentation updates for tearsheet metrics/lifecycle hooks and TradingFee guidance (`per_contract_fee` usage).
+
+### Fixed
+- Day-timestep asset lookup regression for multi-timeframe stock/index backtests (including minute->day fallback paths where appropriate).
+- IBKR stale no-data cache reuse now forces refresh when requested windows extend beyond cached coverage.
+- ProjectX order processing race-condition and tracking hardening merged from `dev`.
+
+Deploy marker: `15e8e268` ("deploy 4.4.55")
+
 ## 4.4.54 - 2026-03-08
 
 ### Added

docsrc/backtesting.tearsheet_html.rst

@@ -19,6 +19,16 @@ These metrics are accompanied by various graphs such as:
 - **Cumulative Returns vs Benchmark:** Shows the strategy's cumulative returns compared to a benchmark.
 - **Cumulative Returns (Log Scaled):** A log-scaled version of cumulative returns for better visualization of exponential growth.
 
+Machine-readable tearsheet metrics
+----------------------------------
+
+Alongside ``*_tearsheet.html``, LumiBot also writes ``*_tearsheet_metrics.json``.
+
+- This JSON contains summary tearsheet metrics in a machine-readable structure.
+- It is intended for downstream automation (agents, dashboards, APIs).
+- You can append strategy-specific metrics by implementing
+  ``Strategy.tearsheet_custom_metrics(...)``.
+
 .. figure:: _html/images/tearsheet_condor_martingale.png
    :alt: Tearsheet example 1
    :width: 600px

docsrc/environment_variables.rst

@@ -43,6 +43,17 @@ BACKTESTING_BUDGET
   - When set, this value is preferred over any ``budget=`` passed in strategy code, so it can be controlled per-run via injected environment variables.
   - Default (when unset and no code budget is provided): ``100000``.
 
+BACKTESTING_PARAMETERS
+^^^^^^^^^^^^^^^^^^^^^^
+
+- Purpose: Override or inject strategy parameters via environment variable, without modifying strategy code.
+- Format: JSON string representing a dictionary. Example: ``{"symbol": "AAPL", "quantity": 10}``
+- Notes:
+  - When set, the parsed dict is merged on top of the strategy's existing ``parameters`` dict with highest priority (wins over both class-level defaults and code-level overrides).
+  - Useful for parameter sweeps: run the same strategy code with different parameter sets per backtest.
+  - Nested dicts are supported (e.g. ``{"ALLOCATION": {"SPY": 0.50, "IWM": 0.50}}``).
+  - Invalid JSON or non-dict values are ignored with a warning.
+
 BACKTESTING_DATA_SOURCE
 ^^^^^^^^^^^^^^^^^^^^^^^
 

docsrc/examples.rst

@@ -723,8 +723,8 @@ Futures Backtesting
 
     if __name__ == "__main__":
         if IS_BACKTESTING:
-            # Use flat fees for futures (typical: $0.50 per contract)
-            trading_fee = TradingFee(flat_fee=0.50)
+            # Use per-contract fees for futures (typical: $0.85 per standard contract, $0.50 for micros)
+            trading_fee = TradingFee(per_contract_fee=0.85)
 
             results = FuturesStrategy.backtest(
                 DataBentoDataBacktesting,
@@ -733,6 +733,32 @@ Futures Backtesting
                 sell_trading_fees=[trading_fee]
             )
 
+Options Backtesting Fees
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+For options strategies, use ``per_contract_fee`` instead of ``flat_fee``. ``per_contract_fee`` is multiplied
+by the number of contracts in each order, which correctly models broker commissions like IBKR's $0.65/contract.
+
+.. code-block:: python
+
+    from lumibot.entities import TradingFee
+
+    # IBKR charges $0.65 per contract per leg
+    # For a 40-contract spread, that's $26.00 per leg
+    trading_fee = TradingFee(per_contract_fee=0.65)
+
+    result = OptionsStrategy.backtest(
+        ThetaDataBacktesting,
+        benchmark_asset=Asset("SPY", Asset.AssetType.STOCK),
+        buy_trading_fees=[trading_fee],
+        sell_trading_fees=[trading_fee],
+    )
+
+.. note::
+    Do NOT use ``flat_fee`` for options or futures commissions. ``flat_fee`` is a fixed amount per order
+    regardless of contract count. For example, ``TradingFee(flat_fee=0.65)`` charges only $0.65 total on a
+    40-contract order, while ``TradingFee(per_contract_fee=0.65)`` correctly charges $26.00 (40 x $0.65).
+
 Multiple Futures Contracts
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docsrc/faq.rst

@@ -301,9 +301,9 @@ Only DataBento supports futures:
 
     from lumibot.backtesting import DataBentoDataBacktesting
 
-    # Use flat fees for futures (typical: $0.50 per contract)
+    # Use per-contract fees for futures (typical: $0.85 per standard contract, $0.50 for micros)
     from lumibot.entities import TradingFee
-    trading_fee = TradingFee(flat_fee=0.50)
+    trading_fee = TradingFee(per_contract_fee=0.85)
 
 Why does my minute-level backtest fail with "no data"?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docsrc/getting_started.rst

@@ -221,9 +221,11 @@ If you want to add trading fees to your backtesting, you can do so by setting up
     from lumibot.backtesting import YahooDataBacktesting
     from lumibot.entities import TradingFee
 
-    # Create two trading fees, one that is a percentage and one that is a flat fee
-    trading_fee_1 = TradingFee(flat_fee=5)  # $5 flat fee
+    # Create trading fees: flat (per order), percent (of order value), or per-contract
+    trading_fee_1 = TradingFee(flat_fee=5)  # $5 flat fee per order
     trading_fee_2 = TradingFee(percent_fee=0.01)  # 1% trading fee
+    # For options/futures, use per_contract_fee instead:
+    # trading_fee = TradingFee(per_contract_fee=0.65)  # $0.65 per contract
 
     backtesting_start = datetime(2020, 1, 1)
     backtesting_end = datetime(2020, 12, 31)

docsrc/lifecycle_methods.rst

@@ -23,6 +23,7 @@ When building strategies, lifecycle methods needs to be overloaded. Trading logi
    lifecycle_methods.on_abrupt_closing
    lifecycle_methods.on_bot_crash
    lifecycle_methods.trace_stats
+   lifecycle_methods.tearsheet_custom_metrics
    lifecycle_methods.on_new_order
    lifecycle_methods.on_partially_filled_order
    lifecycle_methods.on_filled_order

docsrc/lifecycle_methods.tearsheet_custom_metrics.rst

@@ -0,0 +1,112 @@
+def tearsheet_custom_metrics
+============================
+
+``tearsheet_custom_metrics`` is a strategy lifecycle hook that runs during backtest
+analysis, immediately before LumiBot writes:
+
+- ``*_tearsheet.html``
+- ``*_tearsheet_metrics.json``
+
+Use this hook when you want strategy-defined metrics in both artifacts.
+
+When it runs
+------------
+
+1. Backtest trading completes.
+2. LumiBot computes strategy/benchmark return series and drawdown context.
+3. LumiBot calls ``tearsheet_custom_metrics(...)``.
+4. Returned metrics are appended to the tearsheet metrics table and JSON scalar metrics.
+
+Method signature
+----------------
+
+.. code-block:: python
+
+    def tearsheet_custom_metrics(
+        self,
+        stats_df: pd.DataFrame | None,
+        strategy_returns: pd.Series,
+        benchmark_returns: pd.Series | None,
+        drawdown: pd.Series,
+        drawdown_details: pd.DataFrame,
+        risk_free_rate: float,
+    ) -> dict:
+        ...
+
+Parameter structure
+-------------------
+
+``stats_df`` (``pd.DataFrame | None``)
+  Backtest stats dataframe (same data used for ``*_stats.csv/parquet``).
+
+``strategy_returns`` (``pd.Series``)
+  Strategy return series used for tearsheet metric calculations.
+
+``benchmark_returns`` (``pd.Series | None``)
+  Benchmark return series when a benchmark is available; otherwise ``None``.
+
+``drawdown`` (``pd.Series``)
+  Strategy drawdown series derived from cumulative returns.
+
+``drawdown_details`` (``pd.DataFrame``)
+  Drawdown periods table (columns such as start/end/valley/days/max drawdown when available).
+
+``risk_free_rate`` (``float``)
+  Effective risk-free rate used by tearsheet metrics.
+
+Return format
+-------------
+
+Return a ``dict`` mapping metric names to values.
+
+Supported formats:
+
+.. code-block:: python
+
+    {"Custom Metric A": 1.23}
+    {"Custom Metric B": {"strategy": 1.23, "benchmark": 0.91}}
+
+Behavior rules:
+
+1. Return ``{}`` if no custom metrics apply.
+2. Returning ``None`` is treated as no custom metrics.
+3. Returning a non-dict is ignored (with a warning).
+4. Exceptions inside the hook are caught; tearsheet generation continues.
+
+Example
+-------
+
+.. code-block:: python
+
+    class MyStrategy(Strategy):
+        def tearsheet_custom_metrics(
+            self,
+            stats_df,
+            strategy_returns,
+            benchmark_returns,
+            drawdown,
+            drawdown_details,
+            risk_free_rate,
+        ):
+            if strategy_returns.empty:
+                return {}
+
+            p95 = float(strategy_returns.quantile(0.95))
+            avg_dd_days = (
+                float(drawdown_details["days"].mean())
+                if not drawdown_details.empty and "days" in drawdown_details.columns
+                else 0.0
+            )
+
+            return {
+                "95th Percentile Daily Return": p95,
+                "Average Drawdown Days": avg_dd_days,
+            }
+
+API reference (source of truth)
+-------------------------------
+
+The method docstring below is auto-loaded from the Strategy class and should be
+treated as the canonical API reference.
+
+.. automethod:: lumibot.strategies.strategy.Strategy.tearsheet_custom_metrics

llms.txt

@@ -122,6 +122,24 @@ class MyStrategy(Strategy):
             self.submit_order(order)
 ```
 
+### Trading Fees
+```python
+# Stocks: use percent_fee (percentage of order value)
+TradingFee(percent_fee=0.001)  # 0.1% of order value
+
+# Options: use per_contract_fee (multiplied by number of contracts)
+TradingFee(per_contract_fee=0.65)  # $0.65 per contract (IBKR standard)
+
+# Futures: use per_contract_fee
+TradingFee(per_contract_fee=0.85)  # $0.85 per standard contract
+
+# Flat fee: fixed amount per ORDER regardless of quantity
+TradingFee(flat_fee=5.00)  # $5 per order
+
+# Pass to backtest:
+result = MyStrategy.backtest(datasource, buy_trading_fees=[fee], sell_trading_fees=[fee])
+```
+
 ## Backtesting Data Sources
 - `YahooDataBacktesting` - Free, good for stocks
 - `PolygonDataBacktesting` - Crypto and stocks (requires API key)

lumibot/backtesting/interactive_brokers_rest_backtesting.py

@@ -28,6 +28,7 @@ class InteractiveBrokersRESTBacktesting(PandasData):
     MIN_TIMESTEP = "minute"
     ALLOW_DAILY_TIMESTEP = True
     SOURCE = "InteractiveBrokersREST"
+    PREFER_NATIVE_DAY_BARS_FOR_STOCK_INDEX = True
 
     def __init__(
         self,