v4.4.48

Release v4.4.48

Author: grzesir

AGENTS.md

@@ -34,6 +34,7 @@ This repo is frequently edited by **multiple AI sessions**. To avoid lost work:
 - **Test gating (STRICT):** do not introduce new environment variables just to skip/disable tests or to paper over CI failures.
   - Prefer existing pytest markers (`apitest`, `acceptance_backtest`, etc.) and normal test skips with clear reasons.
   - If a new env var is truly required for a user-facing feature, document it in `docsrc/environment_variables.rst` in the same PR.
+- **Full suite verification:** prefer pushing commits to GitHub on the shared `version/X.Y.Z` branch so sharded CI validates the full suite. Local runs should focus on targeted tests or marker-filtered subsets.
 
 1. **Never launch ThetaTerminal locally WITH PRODUCTION CREDENTIALS.** Production has the only licensed session for that account. Starting the jar with prod credentials (even briefly or via Docker) instantly terminates the prod connection and halts all customers.
 2. **Use the downloader for backtests.** All tests/backtests must set `DATADOWNLOADER_BASE_URL` and `DATADOWNLOADER_API_KEY` via the runtime environment. Do not short-cut by hitting Theta directly.

CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 4.4.48 - 2026-02-10
+
+### Added
+- Backtesting artifacts: emit Parquet siblings for `*_indicators.csv`, `*_trades.csv`, `*_stats.csv`, and `*_trade_events.csv` (zstd + PyArrow). CSV remains the compatibility layer.
+
+### Changed
+- Tradier: support OAuth payload + access token refresh; add runtime notes for the refresh flow.
+- Tests: mark DataBento backtest coverage as `apitest` so the default CI suite stays deterministic without vendor credentials.
+- Docs: clarify auto-expiry futures behavior and IBKR crypto roots.
+
+### Fixed
+- Data: handle `timeshift=None` in Data bars.
+- Futures (auto-expiry): make selection roll-aware and harden IBKR conid negative cache behavior.
+
 ## 4.4.47 - 2026-02-07
 ### Added
 - Backtesting: support `BACKTESTING_BUDGET` environment override for strategy backtest cash/budget.

docs/BACKTESTING_TESTS.md

@@ -7,6 +7,17 @@ This project relies on a layered test strategy:
 3. **Acceptance backtests** (manual, end-to-end): run from `Strategy Library/` and inspect artifacts
    (`*_trades.html`, `*_tearsheet.html`, `*_stats.csv`) for realism and regressions.
 
+## Recommended workflow (local + GitHub CI)
+
+Local runs are great for fast feedback, but the full suite is often faster and more reliable on GitHub because CI runs tests in parallel (sharded).
+
+Recommended approach on `version/X.Y.Z` branches:
+
+- Run targeted tests locally for quick iteration.
+- Push early/often so GitHub CI can run the full sharded suite for release confidence.
+
+Treat **green GitHub CI** as the “release-ready” signal since it matches the release workflow environment more closely than a single local machine.
+
 ## Test authority (“Legacy tests win”)
 
 When tests fail, **how you fix them depends on how old the test is**:

docs/DEPLOYMENT.md

@@ -86,6 +86,12 @@ Publishing is **tag-driven** via `.github/workflows/release.yml`.
      - Confirm there are no: `*.env`, `*.log`, `dist/`, `tmp/`, large stray binaries, or accidental artifacts.
    - Manual code review (security, best-effort):
      - Scan the diff for unexpected behavior: new process execution, credential handling, network calls, filesystem writes, or workflow changes.
+     - Explicitly look for “malicious” indicators:
+       - obfuscated code (large base64 blobs, weird string concatenation around URLs/commands)
+       - `eval`/`exec`, unsafe deserialization (`pickle.loads`) in new code
+       - new network destinations / hard-coded private endpoints
+       - silent secret capture/exfil paths (reading `.env`, keychains, `~/.ssh`, AWS creds)
+       - changes under `.github/workflows/` (must be intentional and reviewed)
      - If new/renamed modules were added, ensure they’re “boring” (no hidden side effects at import time).
      - If any new binary is added, confirm it’s expected and justified (size + provenance).
      - If anything feels off, stop and escalate before merging/releasing.

docs/IBKR_FUTURES_BACKTESTING.md

@@ -77,6 +77,22 @@ IBKR futures bar history requires a specific contract identifier (conid). LumiBo
 For deterministic acceptance and correct lookup behavior, **explicit futures** should be used:
 - `Asset("MES", asset_type="future", expiration=date(2025, 12, 19))`
 
+### Auto-expiry futures (rolling wrapper; recommended when you don't want to hardcode an expiration)
+
+If you want “front month” behavior without hardcoding an expiration date, use an auto-expiry futures asset:
+- `Asset("MES", asset_type="future", auto_expiry=Asset.AutoExpiry.FRONT_MONTH)`
+
+Semantics:
+- This is a **rolling wrapper** resolved by the data source/broker as time advances (similar to `cont_future` stitching).
+- `Asset.expiration` remains `None` unless you explicitly provide `expiration=...`.
+- This keeps backtests and live behavior aligned and avoids `date.today()` skew when backtesting historical periods.
+
+### Crypto futures roots (IBKR / CME)
+
+IBKR’s futures root symbols can differ from spot tickers, especially for CME crypto products. Examples:
+- Bitcoin: `MBT` (Micro Bitcoin futures), `BRR` (Bitcoin Reference Rate futures root)
+- Ether: `MET` (Micro Ether futures), `ETHUSDRR` (Ether Reference Rate futures root)
+
 ### Expired contracts (critical)
 
 IBKR Client Portal cannot reliably discover conids for **expired** futures. For backtests that reference expired

docs/handoffs/2026-02-10_PARQUET_BACKTEST_ARTIFACTS.md

@@ -0,0 +1,66 @@
+# PARQUET BACKTEST ARTIFACTS (BOTSPOT)
+
+> Emit Parquet siblings for backtest result artifacts (indicators/trades/stats/trade_events) to speed BotSpot analysis and UI queries.
+
+**Last Updated:** 2026-02-10  
+**Status:** Draft  
+**Audience:** Developers + AI Agents  
+
+---
+
+## Overview
+
+BotSpot frequently queries `*_indicators.csv` (and related artifacts) via DuckDB inside `botspot_node`. CSV parsing + repeated scans can be slow (10s–40s per query in worst cases). This change makes LumiBot emit Parquet versions of the same artifacts so downstream services can prefer Parquet (faster scans, typed columns, compression) while keeping CSV as the compatibility layer.
+
+This is intentionally additive:
+
+- **CSV stays the source-of-compatibility** (existing consumers keep working).
+- **Parquet becomes the source-of-speed** for analytics (DuckDB, UI endpoints, agent tools).
+
+## What Changed
+
+LumiBot now attempts to write these Parquet artifacts alongside existing outputs:
+
+- `*_indicators.parquet` (sibling of `*_indicators.csv`)
+- `*_trades.parquet` (sibling of `*_trades.csv`)
+- `*_stats.parquet` (sibling of `*_stats.csv`)
+- `*_trade_events.parquet` (sibling of `*_trade_events.csv`)
+
+Implementation notes:
+
+- Uses `pandas.DataFrame.to_parquet(engine="pyarrow", compression="zstd")`.
+- **Best-effort:** Parquet export failures are logged as warnings and do **not** fail a backtest (CSV remains the fallback).
+
+## Why This Helps Performance
+
+Downstream (BotSpot) improvements unlocked by emitting Parquet:
+
+- DuckDB scans Parquet without CSV parsing/inference overhead.
+- Parquet is columnar, so projecting a few columns is much cheaper than reading entire CSV rows.
+- Enables Parquet-first behavior in `botspot_node` (fallback to CSV when Parquet is missing).
+
+## Verification (Local)
+
+Parquet generation is covered by unit tests:
+
+- Indicators Parquet: `python3 -m pytest tests/test_indicators_detail_text_edge_cases.py -q`
+- Trades Parquet: `python3 -m pytest tests/test_indicator_subplots.py::test_plot_returns_preserves_cash_settled_status -q`
+- Stats Parquet: `python3 -m pytest tests/test_strategy_dump_stats_regression.py -q`
+- Trade events Parquet: `python3 -m pytest tests/test_backtesting_broker.py -q`
+
+Manual sanity check (after running any backtest with `show_indicators=True`):
+
+- Confirm `logs/*_indicators.csv` and `logs/*_indicators.parquet` both exist and have the same row count.
+
+## Rollout Order (BotSpot)
+
+1. Deploy LumiBot (so new backtests produce Parquet artifacts).
+2. Deploy Bot Manager upload pipeline (so Parquet artifacts get uploaded with results).
+3. Deploy `botspot_node` (Parquet-first querying + timing logs).
+4. Deploy `botspot_react` (ensure indicators Parquet remains non-downloadable in UI).
+
+## Risks / Notes
+
+- Parquet typing can differ from DuckDB CSV inference (usually a net improvement). We rely on CSV fallback if anything is missing.
+- This feature assumes `pyarrow` is present in the runtime image. (It is already pinned in LumiBot deps; if it is absent, Parquet writes will warn and CSV continues to work.)
+

docs/investigations/2026-02-07_tradier-oauth-refresh.md

@@ -0,0 +1,66 @@
+# Tradier OAuth Payload + Refresh Support (LumiBot)
+
+Date: 2026-02-07  
+Owner: Codex (implementation + research)  
+Status: Implemented in `lumibot` (commit `52eef70b`)
+
+## Why this exists
+
+BotSpot deployments can link Tradier via OAuth. Tradier OAuth access tokens expire (per Tradier docs), so a long-running bot needs refresh-token support (when available) to keep trading without requiring users to re-link every day.
+
+## Runtime Inputs (env vars)
+
+This implementation supports both existing “manual token” Tradier usage and OAuth usage.
+
+### Existing (manual token)
+
+- `TRADIER_ACCESS_TOKEN`
+- `TRADIER_ACCOUNT_NUMBER`
+- `TRADIER_IS_PAPER` (`true`/`false`)
+
+### OAuth mode (BotSpot)
+
+BotSpot injects:
+
+- `TRADIER_TOKEN`
+  - base64url JSON payload from the OAuth token exchange
+  - expected fields: `access_token`, `expires_in` (optional), `issued_at` (optional), `refresh_token` (optional)
+- `TRADIER_REFRESH_TOKEN` (optional)
+- `TRADIER_OAUTH_CLIENT_ID` (required to refresh)
+- `TRADIER_OAUTH_CLIENT_SECRET` (required to refresh)
+
+Notes:
+
+- Tradier refresh tokens are **partner-only**; if `TRADIER_REFRESH_TOKEN` is not present, the bot can still start, but it may stop working once the access token expires.
+
+## What changed
+
+File: `lumibot/brokers/tradier.py`
+
+- Added support for decoding `TRADIER_TOKEN` (base64url JSON).
+  - If `access_token` is missing/blank in config/args, it is sourced from the decoded payload.
+- Added best-effort refresh support via:
+  - proactive refresh near expiry (when expiry metadata exists)
+  - forced refresh and single retry when an API call fails with `401`
+- Refresh uses Tradier endpoint:
+  - `POST https://api.tradier.com/v1/oauth/refreshtoken`
+  - Basic Auth: `TRADIER_OAUTH_CLIENT_ID:TRADIER_OAUTH_CLIENT_SECRET`
+  - Body: `grant_type=refresh_token&refresh_token=<TRADIER_REFRESH_TOKEN>`
+- When a refresh succeeds, the broker updates the access token across:
+  - broker’s `lumiwealth_tradier` client
+  - data source’s `lumiwealth_tradier` client
+
+## Limitations / Operational Notes
+
+- If Tradier rotates the refresh token on refresh, the new refresh token cannot be persisted back into task env vars.
+  - The code logs a warning if refresh-token rotation is detected.
+  - If rotation happens, users may need to re-link after the old refresh token becomes invalid.
+
+## Tests
+
+File: `tests/test_tradier.py`
+
+- Added unit tests for:
+  - decoding OAuth payload into `access_token`
+  - refreshing token when payload is expired (requests mocked; no network)
+

lumibot/brokers/broker.py

@@ -2367,6 +2367,20 @@ def export_trade_events_to_csv(self, filename):
             output_df = self._trade_event_log_df.set_index("time")
             output_df.to_csv(filename)
 
+            parquet_filename = (
+                filename[:-4] + ".parquet" if str(filename).lower().endswith(".csv") else str(filename) + ".parquet"
+            )
+            try:
+                self._trade_event_log_df.to_parquet(
+                    parquet_filename,
+                    index=False,
+                    engine="pyarrow",
+                    compression="zstd",
+                )
+            except Exception as exc:
+                # Never fail end-of-run export due to parquet; CSV is the compatibility layer.
+                self.logger.warning("Failed to write trade events parquet file %s: %s", parquet_filename, exc)
+
     def set_strategy_name(self, strategy_name):
         """
         Let's the broker know the name of the strategy that is using it for logging purposes.