Add comprehensive troubleshooting guide for Docker and GitHub Actions

- Create troubleshooting-docker-workflows.mdc rule
- Document reusable workflow issues (secrets, permissions, boolean conversion)
- Document Docker cache and QEMU emulation debugging
- Add platform-specific testing commands
- Include force rebuild pattern and verification checklist
- Capture all debugging patterns from pytest-embedded fix session

Author: hacker-cb

.cursor/rules/troubleshooting-docker-workflows.mdc

@@ -0,0 +1,232 @@
+---
+alwaysApply: false
+description: Troubleshooting guide for Docker builds and GitHub Actions workflows
+---
+
+# Troubleshooting Docker Builds and GitHub Actions Workflows
+
+## GitHub Actions Workflow Issues
+
+### Symptom: Reusable Workflow Build Job Doesn't Run
+
+**Problem:** Only the `prepare` job completes, but the `build` job never starts.
+
+**Common Causes:**
+1. Missing `secrets: inherit` in workflow call
+2. Incorrect boolean conversion for inputs
+3. Workflow syntax errors
+
+**Solution:**
+```yaml
+build:
+  needs: prepare
+  uses: ./.github/workflows/docker-build.yml
+  permissions:
+    contents: read
+    packages: write
+  secrets: inherit  # ← Must have this!
+  with:
+    force_rebuild: ${{ github.event.inputs.force_rebuild == 'true' }}  # ← Not || false
+```
+
+**Verification:**
+```bash
+# Check if build job ran
+gh api /repos/:owner/:repo/actions/runs/<run-id>/jobs | \
+  jq -r '.jobs[] | "\(.name): \(.conclusion)"'
+
+# Should show both prepare AND build jobs
+```
+
+### Symptom: Boolean Input Not Working
+
+**Problem:** `force_rebuild=true` passed but cache still used.
+
+**Cause:** workflow_dispatch inputs are strings, not booleans.
+
+**Wrong:**
+```yaml
+force_rebuild: ${{ github.event.inputs.force_rebuild || false }}
+```
+
+**Correct:**
+```yaml
+force_rebuild: ${{ github.event.inputs.force_rebuild == 'true' }}
+```
+
+## Docker Build Issues
+
+### Symptom: Packages Missing Despite Dockerfile Having Install Command
+
+**Problem:** Image builds successfully but installed packages are missing.
+
+**Common Causes:**
+1. Docker layer caching - using old cached layer
+2. Multi-platform build with QEMU emulation failure (ARM64)
+3. Installation command exits silently without error
+
+**Diagnosis:**
+```bash
+# Check build logs for layer status
+gh run view <run-id> --log | grep "CACHED\|DONE"
+
+# Look for suspiciously fast completion (0.1s = cached or failed)
+gh run view <run-id> --log | grep -A 2 "pip install"
+
+# Verify packages in published image
+docker run --rm <image> pip list | grep <package>
+```
+
+**Solution 1: Force Rebuild**
+```bash
+# Bypass all caching
+gh workflow run "🐳 Workflow Name" --ref master \
+  -f version=v1.0.0 \
+  -f force_rebuild=true
+```
+
+**Solution 2: Check Platform-Specific Builds**
+```bash
+# Test amd64
+docker run --platform=linux/amd64 --rm <image> <command>
+
+# Test arm64
+docker run --platform=linux/arm64 --rm <image> <command>
+```
+
+### Symptom: ARM64 Build Completes in 0.1s (QEMU Emulation Failure)
+
+**Problem:** ARM64 build shows `DONE 0.1s` for steps that should take minutes.
+
+**Example Log:**
+```
+#17 [linux/arm64 5/7] RUN bash -c '. ${IDF_PATH}/export.sh && pip install ...'
+#17 0.057 qemu-aarch64 version 10.0.4
+#17 DONE 0.1s  ← Should take 1-2 minutes!
+```
+
+**Cause:** QEMU emulation fails when executing shell initialization (source, export.sh).
+
+**Solutions:**
+1. **Use direct venv paths for installation:**
+   ```dockerfile
+   # Instead of:
+   RUN bash -c '. ${IDF_PATH}/export.sh && pip install packages'
+   
+   # Use:
+   RUN /opt/esp/python_env/idf5.4_py3.12_env/bin/pip install packages
+   ```
+
+2. **For verification, use sourced environment (amd64 only):**
+   ```dockerfile
+   RUN bash -c '. ${IDF_PATH}/export.sh && pytest --version'
+   ```
+
+3. **Build amd64 only if ARM64 fails:**
+   ```yaml
+   # In docker-build.yml
+   platforms: linux/amd64  # Instead of linux/amd64,linux/arm64
+   ```
+
+### Symptom: Verification Step Fails - "No module named pytest"
+
+**Problem:** `python3 -m pytest` fails with module not found.
+
+**Common Causes:**
+1. System python3 vs venv python3
+2. Packages installed in venv but verification uses system python
+3. Environment not activated
+
+**Wrong Approaches:**
+```dockerfile
+# Uses system Python (doesn't have packages)
+RUN python3 -m pytest --version
+
+# Hardcoded paths (not maintainable)
+RUN /opt/esp/python_env/idf5.4_py3.12_env/bin/python3 -m pytest --version
+```
+
+**Correct Approach:**
+```dockerfile
+# Activate environment first, then use simple commands
+RUN bash -c '. ${IDF_PATH}/export.sh && pytest --version'
+```
+
+## Testing Published Images
+
+### Quick Verification Commands
+
+```bash
+# Check if package is installed
+docker run --rm <image> pip list | grep <package>
+
+# Check if command works
+docker run --rm <image> <command> --version
+
+# Test Python import
+docker run --rm <image> python3 -c 'import <module>; print(<module>.__version__)'
+
+# Interactive testing
+docker run -it --rm <image>
+```
+
+### Platform-Specific Testing
+
+```bash
+# Force specific platform
+docker pull --platform=linux/amd64 <image>
+docker run --platform=linux/amd64 --rm <image> <command>
+
+# Check current platform
+docker run --rm <image> uname -m
+docker inspect <image> --format='{{.Architecture}}'
+```
+
+## Force Rebuild Pattern
+
+When Docker cache causes issues:
+
+```bash
+# 1. Trigger force rebuild
+gh workflow run "🐳 Workflow Name" --ref master \
+  -f version=v1.0.0 \
+  -f force_rebuild=true
+
+# 2. Wait for workflow to start
+sleep 15
+
+# 3. Get run ID
+RUN_ID=$(gh run list --workflow="🐳 Workflow Name" --limit 1 \
+  --json databaseId -q '.[0].databaseId')
+
+# 4. Monitor progress
+gh run watch $RUN_ID
+
+# 5. Check if succeeded
+gh run view $RUN_ID --json conclusion -q '.conclusion'
+
+# 6. Pull and test new image
+docker pull <image>:latest
+docker run --rm <image>:latest <test-command>
+```
+
+## Common Pitfalls
+
+1. **Auto-build uses cache** - Automatic builds from commits use cache by default
+2. **Multiple layers can be cached** - Force rebuild is needed to ensure fresh build
+3. **GHCR tags aren't updated** - Check digest to verify new image is different
+4. **Platform mismatch** - ARM64 Mac pulls arm64 by default, may need `--platform=linux/amd64`
+5. **Dev branch doesn't push** - Only master branch publishes to GHCR
+
+## Debugging Checklist
+
+When packages are missing from published image:
+
+- [ ] Check if Dockerfile has the package in pip install
+- [ ] Verify workflow ran successfully (both prepare AND build jobs)
+- [ ] Check build logs for "CACHED" vs actual execution
+- [ ] Look for 0.1s completion times (indicates cache or QEMU failure)
+- [ ] Test both platforms if multi-platform build
+- [ ] Trigger force rebuild to bypass cache
+- [ ] Pull fresh image and test with exact commands
+- [ ] Verify image digest changed after rebuild