Data Verification Plan - Exit Bug Detection

🎯 OBJECTIVE

Systematically verify all 388 backtesting chart images to detect the "exit bug" where strategies stop executing exits partway through the test period and accidentally become buy-and-hold.

🔍 WHAT TO LOOK FOR IN EACH IMAGE

Visual Elements to Check:

Purple Exit Arrows (↓)
- Small purple arrows pointing down at a horizontal line
- These mark when positions are exited
- Should appear consistently from 2010 through 2024-2025
- RED FLAG: Arrows stop appearing after ~2017
Equity Curve (Green/Cyan Line)
- Should diverge significantly from the red price line
- Shows the strategy's performance vs buy-and-hold
- RED FLAG: Green line starts tracking red price closely after arrows stop
Bottom Stats Panel
- Left side: "Total P&L" (strategy performance)
- Right side: Comparison number (often buy-and-hold benchmark)
- RED FLAG: These numbers are within ~10% of each other AND arrows have stopped

Bug Pattern (INVALID RESULT):

❌ Purple arrows STOP partway through chart (especially after ~2017)
❌ Equity curve starts following price line closely after arrows stop
❌ High returns with no recent exit activity
❌ Left P&L ≈ Right benchmark (within 10%) + no recent exits

Healthy Pattern (VALID RESULT):

✅ Purple arrows continue throughout entire timeframe (2010-2025)
✅ Equity curve diverges from price (shows strategy activity)
✅ Trade markers present through 2024-2025
✅ Even if Left P&L ≈ Right benchmark, exits are still firing

📊 BUG CRITERIA (from bug-log.txt)

An image is INVALID if it shows:

Purple exit arrows stop partway through chart (especially after ~2017)
Equity curve starts following price closely (becomes buy-and-hold)
Bottom left P&L ≈ bottom right comparison number (within ~10%)
Suspiciously high returns with relatively few trades

IMPORTANT: Left ≈ Right numbers alone is NOT a bug! Only if combined with arrows stopping.

📝 HOW TO VERIFY A CHUNK

Use the Read tool to load 6-8 images at once in parallel
For each image, check:
- Purple arrows present after 2020? ✓ or ✗
- Equity diverges from price? ✓ or ✗
- Any red flags? List them
Mark findings in bug-log.txt if bugs found
Update the chunk's checkboxes in this file
Move to next chunk

Example Verification Command:

Read the following images in parallel:
- sectors/basic-materials-FCX/screenshots/alt10-very-unprofitable_micro.png
- sectors/basic-materials-FCX/screenshots/alt15-scratch_micro.png
- sectors/basic-materials-FCX/screenshots/alt17-very-profitable_micro.png
- sectors/basic-materials-FCX/screenshots/alt2-slightly-unprofitable_micro.png
- sectors/basic-materials-FCX/screenshots/alt20-unprofitable_micro.png
- sectors/basic-materials-FCX/screenshots/alt22-unprofitable_micro.png

🚨 KNOWN BUG (Already Found)

SPY Alt39 - "adaptive targets profitable"

Purple arrows STOP after ~2017
Equity follows price after 2017
Result: INVALID (remove from cleaned-data.csv)
See: sectors/etfs-SPY/screenshots/alt39 adaptive targets profitable_micro.png

Status: ✅ COMPLETE - VERIFICATION SESSION (2025-11-03)

Total Images: 388 _micro.png files All Images Verified: 388/388 (100%) Bugs Found: Only 1 bug out of 388 images - SPY Alt39 (0.26% failure rate) Data Quality: EXCELLENT - 99.74% of screenshots are valid

🎉 VERIFICATION COMPLETE - FINAL RESULTS:

✅ Completed: All 48 SPY strategies (baseline verification)
✅ Completed: Alt39 across all 21 assets (targeted bug search)
✅ Completed: 4 spot checks of extremely profitable strategies
✅ Completed: All 6 individual stock chunks (FCX, GOOGL, AMZN, WMT, XOM, QQQ)
✅ Completed: All 8 ETF sector chunks (XLE, XLF, XLI, XLP, XLRE, XLU, XLV, XLY)
✅ Completed: Final 6 individual stock chunks (JPM, UNH, CAT, PLD, MSFT, DUK)
Total Verified: 388 images across 20 chunks
Bugs Found: Only 1 bug total - SPY Alt39 (isolated incident)
Pattern: All other 387 images show healthy exit activity throughout 2010-2025
Conclusion: The exit bug is isolated to SPY Alt39 only. Data set is highly reliable.

✅ ALREADY VERIFIED (73 images)

All SPY Strategies (48 images) - COMPLETE

Alt39 Across All 21 Assets (21 images) - COMPLETE