Document testing requirements for CI failure analysis and fixes

Add comprehensive guidelines to distinguish between tested fixes and
untested hypotheses when working on CI failures.

Key Changes to CLAUDE.md:

1. New "CRITICAL - LOCAL TESTING REQUIREMENTS" section:
   - Never claim a test is "fixed" without local verification
   - Distinguish: "This fixes..." (tested) vs "This SHOULD fix..." (untested)
   - Require clear explanations when local testing isn't possible
   - Document what was tested and results obtained

2. Enhanced "Replicating CI Failures Locally" section:
   - First reproduce failures locally before proposing fixes
   - Clearly state why local testing isn't possible when applicable
   - Mark all proposed fixes as UNTESTED until verified
   - Provide reproduction steps even if unable to execute

3. Testing documentation requirements:
   - Include test commands in commit messages
   - Note which fixes were tested vs hypothetical in PRs
   - Explain testing limitations encountered

Updates to CI_FAILURES_2024-11-21.md:

1. Added CRITICAL DISCLAIMER at top:
   - Clearly states analysis contains untested hypotheses only
   - Documents environment limitations (Conductor workspace)
   - Lists what can/cannot be tested in current environment
   - Warns against assuming fixes will work without verification

2. Marked all fix recommendations with "⚠️ UNTESTED HYPOTHESIS"
   - Priority 1: Integration test asset loading
   - Priority 2: Pro console replay test
   - Priority 3: Hung Pro JS tests

3. Added testing limitation notice before recommendations:
   - Cannot run Rails app, webpack, databases in isolated workspace
   - All fixes must be tested in actual repository
   - Hypotheses based on log analysis only

Rationale:
This prevents confusion between analysis (what we think is wrong) and
verification (what we know is wrong through testing). Makes it clear
when fixes are theoretical vs proven, especially important when working
in isolated environments like Conductor workspaces.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>

Author: justin808

.claude/docs/analysis/CI_FAILURES_2024-11-21.md

@@ -5,6 +5,33 @@
 **Analyst**: Claude Code Review
 **Status**: 🔴 NOT READY TO MERGE - Multiple Test Failures
 
+---
+
+## ⚠️ CRITICAL DISCLAIMER
+
+**This analysis contains UNTESTED hypotheses only.**
+
+**Environment**: Analysis performed in Conductor isolated workspace with limited capabilities:
+
+- ❌ Cannot run full Rails application
+- ❌ Cannot execute integration tests
+- ❌ Cannot test webpack build pipeline
+- ❌ Cannot verify fixes against actual failing tests
+- ✅ Can analyze CI logs and source code
+- ✅ Can identify suspicious commits
+- ✅ Can provide reproduction steps
+
+**What this means**:
+
+- All "fixes" below are **hypotheses that MUST be tested locally**
+- Do NOT assume any proposed fix will work without verification
+- All investigation is based on log analysis and code inspection only
+- Actual root causes may differ from analysis after local testing
+
+**Before implementing any fix**: Test it locally in the actual React on Rails repository first.
+
+---
+
 ## Executive Summary
 
 **All tests PASSED on merge base** (`5e033c716` - v16.2.0.beta.12) ✅
@@ -324,8 +351,23 @@ git bisect good/bad
 
 ## Fix Recommendations
 
+**⚠️ IMPORTANT: All recommendations below are UNTESTED hypotheses based on CI log analysis.**
+
+**Testing Limitation**: This analysis was performed in a Conductor isolated workspace without:
+
+- Full Rails application environment
+- Webpack build pipeline
+- Database/Redis services
+- Integration test infrastructure
+
+**All proposed fixes MUST be tested locally in the actual React on Rails repository before claiming they work.**
+
+---
+
 ### Priority 1: Integration Test Asset Loading (CRITICAL) 🔥
 
+**Status**: ⚠️ UNTESTED HYPOTHESIS
+
 **Hypothesis**: Prepare scripts don't run correctly in workspace context, OR webpack output paths changed.
 
 **Investigation**:
@@ -354,6 +396,8 @@ bundle exec rspec spec/system/integration_spec.rb:23
 
 ### Priority 2: Pro Console Replay Test 🟡
 
+**Status**: ⚠️ UNTESTED HYPOTHESIS
+
 **Hypothesis**: buildConsoleReplay parameter fix changed output format OR test HTML parsing broken.
 
 **Investigation**:
@@ -380,6 +424,8 @@ bundle exec rspec spec/requests/renderer_console_logging_spec.rb:13 --format doc
 
 ### Priority 3: Hung Pro JS Tests 🟡
 
+**Status**: ⚠️ UNTESTED HYPOTHESIS
+
 **Immediate**: Cancel the 80+ minute CI run
 
 **Investigation**:

CLAUDE.md

@@ -29,6 +29,32 @@ Top-level documentation (like README.md, CONTRIBUTING.md) should remain at the r
 
 These requirements are non-negotiable. CI will fail if not followed.
 
+**CRITICAL - LOCAL TESTING REQUIREMENTS:**
+
+1. **NEVER claim a test is "fixed" without running it locally first**
+   - If working in Conductor workspace or similar isolated environment, clearly state: "Cannot test locally in isolated workspace"
+   - If test requires specific environment (database, Redis, etc.), state: "Requires [X] setup not available in current environment"
+
+2. **Distinguish hypothetical fixes from confirmed fixes:**
+   - ✅ Use "This fixes..." or "Fixed" ONLY after local verification
+   - ⚠️ Use "This SHOULD fix..." or "Proposed fix (UNTESTED)" when you haven't verified
+   - 📋 Use "Analysis suggests..." or "Root cause appears to be..." for investigation without fixes
+
+3. **When analyzing CI failures:**
+   - Clearly mark all analysis as "UNTESTED - requires local reproduction" unless verified
+   - Provide exact commands to reproduce and test the fix locally
+   - State why you cannot test if applicable (workspace restrictions, missing services, etc.)
+
+4. **Prefer local testing over CI iteration:**
+   - Don't push "hopeful" fixes and wait for CI feedback
+   - Test locally first whenever technically possible
+   - Document what you tested and what results you got
+
+5. **Document your testing:**
+   - Include test commands in commit messages for complex fixes
+   - Note in PR descriptions which fixes were tested locally vs. which are hypothetical
+   - Explain any testing limitations encountered
+
 **🚀 AUTOMATIC: Git hooks are installed automatically during setup**
 
 Git hooks will automatically run linting on **all changed files (staged + unstaged + untracked)** before each commit - making it fast while preventing CI failures!
@@ -73,6 +99,21 @@ Pre-commit hooks automatically run:
 
 **CRITICAL: NEVER wait for CI to verify fixes. Always replicate failures locally first.**
 
+**When Analyzing CI Failures:**
+
+1. **First, reproduce the failure locally** using the tools below
+2. **If you cannot reproduce locally**, clearly state why:
+   - "Working in Conductor isolated workspace - cannot run full Rails app"
+   - "Requires Docker/Redis/PostgreSQL not available in current environment"
+   - "Integration tests need full webpack build pipeline not set up locally"
+3. **Mark all proposed fixes as UNTESTED** until they can be verified:
+   - ❌ DON'T: "This fixes the integration test failures"
+   - ✅ DO: "This SHOULD fix the integration test failures (UNTESTED - requires local Rails app with webpack)"
+4. **Provide reproduction steps** even if you can't execute them:
+   - Include exact commands to run
+   - Document environment requirements
+   - Explain what success looks like
+
 #### Switch Between CI Configurations
 
 The project tests against two configurations: