feat: add detailed documentation for fixing workspace upsert bug blocking backfill

Author: tikazyq

specs/014-workspace-upsert-backfill-bug/README.md

@@ -0,0 +1,143 @@
+---
+status: planned
+created: '2025-11-11'
+tags:
+  - bug
+  - backfill
+  - api
+  - database
+priority: high
+created_at: '2025-11-11T15:06:38.494Z'
+---
+
+# Fix Workspace Upsert Bug Blocking Backfill
+
+> **Status**: 📅 Planned · **Priority**: High · **Created**: 2025-11-11
+
+## Overview
+
+The backfill feature successfully parses Copilot chat sessions and sends events to the API, but no data reaches PostgreSQL due to a workspace upsert bug. When the API tries to upsert a workspace that already exists, it fails with "Unique constraint failed on the fields: (`workspace_id`)", causing all batch inserts to fail.
+
+## Problem
+
+The collector's backfill process:
+
+1. ✅ Parses chat session files correctly (798 events parsed)
+2. ✅ Sends batches to the API endpoint
+3. ❌ API fails to upsert workspace with unique constraint error
+4. ❌ All batches retry 4 times and fail
+5. ❌ Zero data inserted into `agent_sessions` or `agent_events` tables
+
+## Root Cause
+
+The API's workspace upsert logic doesn't properly handle the case when a workspace already exists in the database. The Prisma upsert is failing on the unique constraint for `workspace_id`.
+
+## Impact
+
+- Backfill feature is completely blocked
+- Historical data cannot be imported
+- Manual workspace creation doesn't help (creates the conflict)
+
+## Related Issues
+
+- Spec #006: Go Collector Next Phase (backfill feature)
+- Database has proper constraints and foreign keys
+- Collector batch retry logic works correctly (retries 4 times as expected)
+
+## Design
+
+### Error Analysis
+
+The API returns:
+
+```
+Status 500: "Unique constraint failed on the fields: (`workspace_id`)"
+```
+
+This occurs when the API tries to upsert a workspace that already exists. The Prisma upsert operation should use:
+
+- `where: { workspace_id }` to find existing workspace
+- `update: { ...fields }` to update if found
+- `create: { ...fields }` to insert if not found
+
+### Likely Causes
+
+1. **Incorrect upsert key**: Using wrong unique field in `where` clause
+2. **Missing update data**: Empty `update` object causes constraint violation
+3. **Conflicting unique fields**: Trying to update other unique fields that conflict
+4. **Transaction issues**: Workspace upsert happening inside a transaction that conflicts
+
+### Solution Approach
+
+1. **Investigate API endpoint**: Find the workspace upsert code in the API
+2. **Fix Prisma upsert**: Ensure proper `where`, `update`, `create` clauses
+3. **Add logging**: Log workspace upsert operations for debugging
+4. **Test with existing workspaces**: Verify upsert works when workspace already exists
+
+## Plan
+
+- [ ] Locate the API endpoint handling batch inserts (`/api/observability/batch` or similar)
+- [ ] Find the workspace upsert code in the API handler
+- [ ] Review Prisma upsert query structure
+- [ ] Fix upsert to properly handle existing workspaces
+- [ ] Add error handling and logging for workspace operations
+- [ ] Test with manually created workspace
+- [ ] Test with backfill on fresh database
+- [ ] Verify data reaches `agent_sessions` and `agent_events` tables
+
+## Test
+
+### Prerequisites
+
+- Project id=1 exists in database
+- Machine record exists
+- Workspace `aebecdd872cc19008a36d00765d84755` exists
+
+### Test Cases
+
+- [ ] **Fresh backfill**: Run backfill with no existing workspace → should succeed
+- [ ] **Existing workspace**: Run backfill with pre-existing workspace → should upsert and succeed
+- [ ] **Concurrent requests**: Multiple batches trying to upsert same workspace → should handle gracefully
+- [ ] **Data verification**: Verify `agent_sessions` count > 0 and `agent_events` count > 0 after backfill
+- [ ] **Event integrity**: Verify events have correct workspace_id, project_id, session_id references
+
+### Success Criteria
+
+```bash
+# After backfill:
+SELECT COUNT(*) FROM agent_sessions; -- Should be > 0
+SELECT COUNT(*) FROM agent_events;   -- Should be > 0
+SELECT COUNT(*) FROM workspaces WHERE workspace_id = 'aebecdd872cc19008a36d00765d84755'; -- Should be 1
+```
+
+## Notes
+
+### Investigation Log (2025-11-11)
+
+**Setup:**
+
+- Created project id=1 manually
+- Created machine and workspace manually
+- Prisma client was missing (regenerated)
+- Web API had TypeScript compilation errors (fixed by regenerating Prisma)
+
+**Backfill Results:**
+
+- 798 events parsed from chat sessions
+- All batch inserts failed with workspace upsert error
+- Collector retry logic worked (4 attempts per batch)
+- Final result: 0 rows in agent_sessions, 0 rows in agent_events
+
+**Error Pattern:**
+
+```
+Failed to send batch (attempt 1/4): unexpected status 500:
+"Unique constraint failed on the fields: (`workspace_id`)"
+```
+
+### Next Steps
+
+1. Find the batch insert API endpoint code
+2. Examine the Prisma upsert query
+3. Fix the upsert logic
+4. Re-test backfill