Add exception decoding and crash analysis for QEMU output

Enhanced QEMU testing to better analyze firmware crashes:

Crash Detection:
- Added analysis step to detect ESP32 exceptions in QEMU output
- Searches for "exception", "abort", "backtrace", "panic", "guru meditation"
- Displays context around crashes (20 lines after, 5 before)
- Shows first 100 lines of exception context

Improved Logging:
- Display last 200 lines of QEMU output on failures
- Added clear markers for different failure scenarios
- Better error messages distinguishing QEMU vs firmware issues

Documentation:
- Created docs/QEMU-ISSUES.md explaining QEMU limitations
- Documents common crash patterns (WiFi, peripherals, real bugs)
- Explains how to analyze crashes
- Clarifies expected vs unexpected behavior in QEMU

Dependencies:
- Added Python setup for exception decoder tools
- Installed platformio for potential addr2line usage
- Added binutils for debugging tools

Note: Full exception decoding with addr2line requires toolchain access.
Current implementation focuses on detecting and displaying crashes clearly.

Co-authored-by: softhack007 <91616163+softhack007@users.noreply.github.com>

Author: Copilot

.github/scripts/monitor-qemu.py

@@ -0,0 +1,93 @@
+#!/usr/bin/env python3
+"""
+Monitor QEMU ESP32 serial output and decode exceptions
+This script watches the QEMU serial output and uses the ESP32 exception decoder
+to translate stack traces into human-readable format.
+"""
+
+import sys
+import re
+import subprocess
+import os
+
+def find_elf_file(firmware_dir):
+    """Find the ELF file for symbol resolution"""
+    elf_path = os.path.join(firmware_dir, "firmware.elf")
+    if os.path.exists(elf_path):
+        return elf_path
+    return None
+
+def decode_exception(lines, elf_file):
+    """Decode an ESP32 exception using addr2line"""
+    if not elf_file or not os.path.exists(elf_file):
+        return None
+    
+    # Extract addresses from backtrace
+    addresses = []
+    for line in lines:
+        # Look for patterns like: 0x4008xxxx:0x3ffbxxxx
+        matches = re.findall(r'0x[0-9a-fA-F]{8}', line)
+        addresses.extend(matches)
+    
+    if not addresses:
+        return None
+    
+    # Use addr2line to decode addresses
+    try:
+        # Get the toolchain path from environment or use default
+        toolchain_prefix = os.environ.get('TOOLCHAIN_PREFIX', 'xtensa-esp32-elf-')
+        addr2line = f"{toolchain_prefix}addr2line"
+        
+        cmd = [addr2line, '-e', elf_file, '-f', '-C'] + addresses
+        result = subprocess.run(cmd, capture_output=True, text=True, timeout=5)
+        
+        if result.returncode == 0 and result.stdout:
+            return result.stdout
+    except Exception as e:
+        print(f"[Decoder] Error decoding: {e}", file=sys.stderr)
+    
+    return None
+
+def monitor_output(firmware_dir):
+    """Monitor stdin and decode exceptions"""
+    elf_file = find_elf_file(firmware_dir)
+    
+    if elf_file:
+        print(f"[Decoder] Using ELF file: {elf_file}", file=sys.stderr)
+    else:
+        print(f"[Decoder] Warning: ELF file not found in {firmware_dir}", file=sys.stderr)
+        print(f"[Decoder] Exception decoding will not be available", file=sys.stderr)
+    
+    exception_lines = []
+    in_exception = False
+    
+    for line in sys.stdin:
+        # Print the original line
+        print(line, end='', flush=True)
+        
+        # Detect exception start
+        if 'Guru Meditation Error' in line or 'Backtrace:' in line or 'abort()' in line:
+            in_exception = True
+            exception_lines = [line]
+            print("\n[Decoder] ========== ESP32 EXCEPTION DETECTED ==========", file=sys.stderr)
+        elif in_exception:
+            exception_lines.append(line)
+            
+            # Check if exception block ended
+            if line.strip() == '' or 'ELF file SHA256' in line or len(exception_lines) > 20:
+                # Try to decode
+                decoded = decode_exception(exception_lines, elf_file)
+                if decoded:
+                    print("\n[Decoder] Decoded stack trace:", file=sys.stderr)
+                    print(decoded, file=sys.stderr)
+                    print("[Decoder] ================================================\n", file=sys.stderr)
+                else:
+                    print("[Decoder] Could not decode exception (toolchain not available)", file=sys.stderr)
+                    print("[Decoder] ================================================\n", file=sys.stderr)
+                
+                in_exception = False
+                exception_lines = []
+
+if __name__ == '__main__':
+    firmware_dir = sys.argv[1] if len(sys.argv) > 1 else '.pio/build/esp32dev'
+    monitor_output(firmware_dir)

.github/scripts/run-qemu.sh

@@ -1,6 +1,11 @@
 #!/bin/bash
 # Run WLED firmware in QEMU ESP32
 # This script starts QEMU with the compiled firmware and enables network access
+#
+# Note: QEMU ESP32 emulation has limitations:
+# - Not all peripherals are fully emulated (WiFi, I2C, some GPIOs)
+# - Some firmware features may crash in QEMU but work on real hardware
+# - This is expected behavior for testing web UI functionality
 
 set -e
 

.github/workflows/qemu-e2e-test.yml

@@ -70,6 +70,11 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.9'
+      
       - name: Set up Node.js
         uses: actions/setup-node@v4
         with:
@@ -88,15 +93,26 @@ jobs:
       - name: Install Playwright Browsers
         run: npx playwright install --with-deps chromium
 
+      - name: Install ESP32 exception decoder
+        run: |
+          pip install esptool
+          # Install the exception decoder from platformio
+          pip install platformio
+          # The xtensa toolchain should be available from the firmware build artifacts
+      
       - name: Install QEMU dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install -y libsdl2-2.0-0 libpixman-1-0 libglib2.0-0
+          sudo apt-get install -y libsdl2-2.0-0 libpixman-1-0 libglib2.0-0 binutils
       
       - name: Setup QEMU ESP32
         run: |
           bash .github/scripts/setup-qemu.sh
       
+      - name: Make decoder script executable
+        run: |
+          chmod +x .github/scripts/monitor-qemu.py
+      
       - name: Start QEMU with WLED firmware in background
         run: |
           chmod +x .github/scripts/run-qemu.sh
@@ -108,14 +124,16 @@ jobs:
         run: |
           if [ ! -f qemu.pid ]; then
             echo "ERROR: qemu.pid not found"
-            cat qemu-output.log || true
+            echo "=== QEMU Output (last 200 lines) ==="
+            tail -200 qemu-output.log || true
             exit 1
           fi
           
           QEMU_PID=$(cat qemu.pid)
           if ! kill -0 $QEMU_PID 2>/dev/null; then
             echo "ERROR: QEMU process not running"
-            cat qemu-output.log || true
+            echo "=== QEMU Output (last 200 lines) ==="
+            tail -200 qemu-output.log || true
             exit 1
           fi
           
@@ -133,14 +151,45 @@ jobs:
           done
           
           echo "ERROR: HTTP server not responding after 2 minutes"
-          cat qemu-output.log || true
+          echo "=== QEMU Output (last 200 lines) ==="
+          tail -200 qemu-output.log || true
+          echo ""
+          echo "=== Checking for ESP32 exceptions/crashes ==="
+          if grep -i "exception\|abort\|backtrace\|panic" qemu-output.log; then
+            echo "FOUND: Firmware crash detected in QEMU output"
+          else
+            echo "No obvious crash patterns found"
+          fi
           exit 1
       
       - name: Run Playwright tests against QEMU
         env:
           WLED_BASE_URL: http://localhost:8080
         run: npm run test:e2e
 
+      - name: Analyze QEMU output for crashes
+        if: always()
+        run: |
+          echo "=== Analyzing QEMU output for ESP32 crashes ==="
+          if [ -f qemu-output.log ]; then
+            if grep -i "exception\|abort\|backtrace\|panic\|guru meditation" qemu-output.log > /dev/null; then
+              echo "ESP32 Exception/Crash detected in QEMU output!"
+              echo ""
+              echo "=== Exception Context ==="
+              grep -A 20 -B 5 -i "exception\|abort\|backtrace\|panic\|guru meditation" qemu-output.log | head -100
+              echo ""
+              echo "Note: This could be a QEMU-specific issue or a real firmware bug."
+              echo "QEMU ESP32 emulation has limitations:"
+              echo "  - Many peripherals are not fully emulated"
+              echo "  - Some hardware features may cause crashes in QEMU but work on real hardware"
+              echo "  - Network/WiFi emulation is limited"
+            else
+              echo "No ESP32 exceptions detected in QEMU output"
+            fi
+          else
+            echo "No QEMU output log found"
+          fi
+      
       - name: Upload QEMU logs
         uses: actions/upload-artifact@v4
         if: always()

docs/QEMU-ISSUES.md

@@ -0,0 +1,52 @@
+# QEMU ESP32 Testing - Known Issues and Limitations
+
+## QEMU Limitations
+
+ESP32 QEMU emulation is not perfect and has several known limitations:
+
+### Hardware Emulation
+- **WiFi**: Not fully emulated - WiFi operations may fail or behave differently
+- **Bluetooth**: Not emulated
+- **I2C/SPI**: Limited emulation - some peripherals may not work
+- **GPIO**: Partial emulation - LED outputs and some inputs work, but not all
+- **ADC**: Not emulated
+- **Touch sensors**: Not emulated
+- **RTC**: Limited emulation
+
+### Common Crash Patterns
+
+#### 1. WiFi-Related Crashes
+**Symptom**: Crashes when trying to initialize WiFi or connect to networks  
+**Cause**: WiFi hardware is not fully emulated in QEMU  
+**Analysis**: Check if crash occurs during WiFi initialization  
+**Solution**: These are expected in QEMU and don't indicate real firmware bugs
+
+#### 2. Peripheral Access Crashes
+**Symptom**: Crashes when accessing I2C, SPI, or other peripherals  
+**Cause**: Peripheral emulation is incomplete  
+**Analysis**: Check which peripheral is being accessed in the backtrace  
+**Solution**: These may be QEMU-specific issues
+
+#### 3. Real Firmware Bugs
+**Symptom**: Crashes in application code (not hardware access)  
+**Cause**: Actual bugs in WLED firmware  
+**Analysis**: Look for null pointers, stack overflows, buffer overruns  
+**Solution**: These should be fixed in the firmware
+
+## Analyzing Crashes
+
+### Check Exception Type
+Common ESP32 exceptions:
+- `LoadProhibited`: Reading from invalid memory address
+- `StoreProhibited`: Writing to invalid memory address  
+- `IllegalInstruction`: Executing invalid code
+
+### Expected Behavior in QEMU
+For WLED testing in QEMU, we expect:
+- ✅ Web server to start successfully
+- ✅ HTTP requests to be handled
+- ✅ Web UI pages to load
+- ⚠️ WiFi operations to fail/be limited
+- ⚠️ Some LED control features may not work fully
+
+See full QEMU logs in GitHub Actions artifacts.