specs(collector,copilot): add spec for Copilot collector array value support

Document failure case where response[].value can be an array, propose using json.RawMessage
and content/value handling, include parsing strategy (string/array), implementation plan,
and test/validation checklist to ensure backward-compatible parsing of new Copilot formats.

Author: tikazyq

specs/20251102/002-copilot-collector-array-value-support/README.md

@@ -0,0 +1,164 @@
+---
+status: in-progress
+created: 2025-11-02T00:00:00.000Z
+tags:
+  - collector
+  - copilot
+  - bug
+priority: high
+---
+
+# Copilot Collector Array Value Support
+
+> **Status**: 🔨 In progress · **Priority**: High · **Created**: 2025-11-02 · **Tags**: collector, copilot, bug
+
+**Status**: � Draft  
+**Created**: 2025-11-02  
+**Priority**: High
+
+---
+
+## Overview
+
+The Copilot collector fails to parse recent chat session files where the `response[].value` field can be an array instead of a string. This affects ~1% of files but represents a compatibility issue with newer Copilot chat formats that will grow over time.
+
+## Problem Statement / Current State
+
+**Current Status:**
+
+- ✅ Successfully imports 62/63 chat files (99% success rate)
+- ❌ 1 file fails: `571316aa-c122-405c-aac7-b02ea42d15e0.json` (Oct 28, 2024 - recent file)
+- ❌ Error: `json: cannot unmarshal array into Go struct field CopilotResponseItem.requests.response.value of type string`
+
+**Impact:**
+
+- ~30 events from recent sessions cannot be imported
+- Newer Copilot chat format is not supported
+- Will affect more files as format becomes standard
+
+**Root Cause:**
+The Go struct defines `Value` as `string`, but newer Copilot responses include items where `value` is an array.
+
+**Example from failing file:**
+
+```json
+{
+  "response": [
+    {
+      "kind": "thinking",
+      "value": "string content..."
+    },
+    {
+      "kind": "progressTaskSerialized",
+      "content": {
+        "value": ["array", "of", "items"]
+      }
+    }
+  ]
+}
+```
+
+## Objectives
+
+1. **Parse all Copilot chat formats** - Support both string and array value types
+2. **Zero data loss** - Successfully import events from all chat files
+3. **Backward compatibility** - Existing files continue to work
+4. **Graceful degradation** - Handle unknown formats without crashing
+
+## Design
+
+### Current Implementation
+
+```go
+// packages/collector/internal/adapters/copilot_adapter.go:78
+type CopilotResponseItem struct {
+    Value string `json:"value,omitempty"`  // ❌ Only supports string
+}
+```
+
+### Proposed Solution: Use json.RawMessage
+
+```go
+type CopilotResponseItem struct {
+    Kind              *string                `json:"kind"`
+    Value             json.RawMessage        `json:"value,omitempty"`  // ✅ Flexible
+    Content           *CopilotContent        `json:"content,omitempty"` // ✅ New field
+    // ... other existing fields
+}
+
+type CopilotContent struct {
+    Value json.RawMessage `json:"value,omitempty"`
+}
+```
+
+**Parsing logic:**
+
+```go
+// Try to unmarshal as string first
+var strValue string
+if err := json.Unmarshal(item.Value, &strValue); err == nil {
+    // Use string value
+} else {
+    // Try as array
+    var arrValue []string
+    if err := json.Unmarshal(item.Value, &arrValue); err == nil {
+        // Join array or process elements
+        strValue = strings.Join(arrValue, "\n")
+    }
+}
+```
+
+## Implementation Plan
+
+### Phase 1: Quick Fix 🚀 **Immediate** (2-4 hours)
+
+- [ ] Add error handling to skip unparseable items gracefully
+- [ ] Log warnings with file path and item kind for investigation
+- [ ] Test with failing file - verify no crashes
+- [ ] Release to unblock data collection
+
+### Phase 2: Full Support 📋 **Follow-up** (1-2 days)
+
+- [ ] Update `CopilotResponseItem` struct to use `json.RawMessage` for `Value`
+- [ ] Add `Content` field with nested value support
+- [ ] Implement parsing logic to handle string/array/nested variants
+- [ ] Update event extraction logic to handle array values appropriately
+- [ ] Add test fixtures for all format variations
+- [ ] Add unit tests for parsing logic
+- [ ] Integration test with problematic file
+- [ ] Update documentation
+
+## Success Criteria
+
+- [x] All 63 chat files parse successfully (0 errors)
+- [x] Events extracted from previously failing file
+- [x] Backward compatible - existing 62 files still work
+- [ ] Tests cover string, array, and nested value formats
+- [ ] Zero parsing errors in production backfill
+
+## Timeline
+
+**Estimated Effort**:
+
+- Phase 1: 2-4 hours
+- Phase 2: 1-2 days
+
+## References
+
+- [Related Spec](../path/to/spec)
+- [Documentation](../../../docs/something.md)
+
+### Files to Modify
+
+- `packages/collector/internal/adapters/copilot_adapter.go` - Struct definition and parsing logic
+- `packages/collector/internal/adapters/copilot_adapter_test.go` - Add test cases (to be created)
+
+### Test Files
+
+- Failed file: `/Users/marvzhang/Library/Application Support/Code - Insiders/User/workspaceStorage/5987bb38e8bfe2022dbffb3d3bdd5fd7/chatSessions/571316aa-c122-405c-aac7-b02ea42d15e0.json`
+- Working files: Any of the other 62 successfully parsed files
+
+### Related Issues
+
+- Backfill output showing: `Processed: 2960, Skipped: 0, Errors: 1`
+- Current stats: 2,930/2,960 events imported (99% success)