feat(chat): Refactor ChatHub service to support new client-server architecture and enhance chat data processing

Author: tikazyq

.devlog/entries/279-remove-legacy-codehist-references-from-codebase.json

@@ -7,7 +7,7 @@
   "status": "in-progress",
   "priority": "medium",
   "createdAt": "2025-07-24T13:59:47.120Z",
-  "updatedAt": "2025-07-24T14:11:12.587Z",
+  "updatedAt": "2025-07-24T14:22:20.666Z",
   "notes": [
     {
       "id": "0f7318d0-40dc-4111-bca5-5a5949cc6e46",
@@ -38,6 +38,34 @@
         "packages/ai/src/services/index.ts",
         "packages/web/app/api/workspaces/[id]/chat/import/route.ts"
       ]
+    },
+    {
+      "id": "c2f7723d-c165-4143-b0d0-11a1672b81a0",
+      "timestamp": "2025-07-24T14:15:31.060Z",
+      "category": "idea",
+      "content": "🔄 **MAJOR ARCHITECTURE INSIGHT**: We had the data flow backwards!\n\n**❌ Current (wrong) approach**: \n- Server tries to import from local VS Code/AI agent storage\n- Requires server to access local file systems\n- Complex cross-platform discovery logic on server\n\n**✅ Correct approach**:\n- **Local clients** (CLI tools, IDE extensions, desktop apps) read their own chat history  \n- **Clients stream/upload** chat data to API endpoints\n- **Server (ChatHub)** becomes a data ingestion and processing service\n- Clean separation: clients extract, server processes\n\nThis changes ChatHub from an \"importer\" to a \"receiver/processor\" of chat data streams."
+    },
+    {
+      "id": "cdd06fb7-f382-474b-bb09-e808ed4a190e",
+      "timestamp": "2025-07-24T14:17:27.278Z",
+      "category": "idea",
+      "content": "🔗 **Related Architecture Work**: Created devlog #280 to extract CLI from @devlog/ai into separate @devlog/cli package.\n\nThis supports the new client-server architecture where:\n- **@devlog/cli** = local client for chat extraction and API communication\n- **ChatHub service** = server-side data ingestion and processing\n- **Clean separation** of concerns and better package management\n\nThe ChatHub refactoring in this entry aligns with the new CLI package structure."
+    },
+    {
+      "id": "024685a1-37de-4a59-b0bc-3089a66c613c",
+      "timestamp": "2025-07-24T14:18:25.216Z",
+      "category": "progress",
+      "content": "🔄 **Pivoting to Receiver Architecture**: Repurposing ChatHub as a data receiver/processor instead of local importer.\n\n**New ChatHub Methods**:\n- `ingestChatSessions()` - Process incoming chat sessions from clients\n- `ingestChatMessages()` - Process incoming chat messages from clients  \n- `processIncomingData()` - Handle bulk chat data uploads\n- Remove local file extraction logic (moves to @devlog/cli)"
+    },
+    {
+      "id": "d9a3ce7f-8cbd-4070-bb3f-779e575cec20",
+      "timestamp": "2025-07-24T14:22:20.666Z",
+      "category": "progress",
+      "content": "🎉 **ChatHub Receiver Architecture Complete**!\n\n✅ **New ChatHub Service Methods**:\n- `ingestChatSessions()` - Process incoming chat sessions from clients\n- `ingestChatMessages()` - Process incoming chat messages from clients  \n- `processBulkChatData()` - Handle bulk chat data uploads with progress tracking\n- Removed local file extraction logic (CopilotParser, etc.)\n\n✅ **Updated API Endpoint**:\n- POST now expects `{sessions: [], messages: [], source: 'github-copilot', workspaceInfo: {}}` \n- Validates incoming data format\n- Uses ChatHub to process received data instead of triggering local imports\n- Clean client-server separation\n\n✅ **Architecture Benefits**:\n- Server no longer needs local file system access\n- Clients handle extraction, server handles processing\n- Much cleaner separation of concerns\n- Ready for @devlog/cli integration\n\n**Status**: Core receiver architecture complete! Ready for CLI client development.",
+      "files": [
+        "packages/ai/src/services/chat-hub-service.ts",
+        "packages/web/app/api/workspaces/[id]/chat/import/route.ts"
+      ]
     }
   ],
   "files": [],

.devlog/entries/280-extract-cli-functionality-from-devlog-ai-into-sepa.json

@@ -0,0 +1,47 @@
+{
+  "id": 280,
+  "key": "extract-cli-functionality-from-devlog-ai-into-sepa",
+  "title": "Extract CLI functionality from @devlog/ai into separate @devlog/cli package",
+  "type": "refactor",
+  "description": "Extract the CLI functionality currently embedded in @devlog/ai package into a dedicated @devlog/cli package for better organization and management. The new CLI package should handle chat history extraction from various local sources (VS Code, Cursor, Claude Code, etc.) and stream the data to the ChatHub API endpoints instead of processing locally.",
+  "status": "new",
+  "priority": "medium",
+  "createdAt": "2025-07-24T14:17:07.399Z",
+  "updatedAt": "2025-07-24T14:17:19.583Z",
+  "notes": [
+    {
+      "id": "70109f21-ca4c-4c52-b85a-0cc0516eec83",
+      "timestamp": "2025-07-24T14:17:19.583Z",
+      "category": "idea",
+      "content": "**Package Architecture Vision**:\n\n**Current State**: \n- CLI buried in `@devlog/ai/src/cli/`\n- Local processing and file export\n- Mixed concerns in AI package\n\n**Target State**:\n- Dedicated `@devlog/cli` package\n- Client-server architecture: CLI extracts → ChatHub API receives\n- Clean separation: `@devlog/ai` = processing logic, `@devlog/cli` = user interface + local extraction\n- Better discoverability and installation experience\n\n**CLI Commands Evolution**:\n- `devlog chat stats` → stream stats to API and display\n- `devlog chat import` → extract and upload to ChatHub API  \n- `devlog chat search` → search via API endpoints"
+    }
+  ],
+  "files": [],
+  "relatedDevlogs": [],
+  "context": {
+    "businessContext": "Having a dedicated CLI package improves discoverability and makes it easier for users to install and use devlog's command-line tools. It also separates concerns better - the AI package focuses on core chat processing logic while the CLI package handles user interaction and local data extraction.",
+    "technicalContext": "Currently @devlog/ai contains CLI commands for chat extraction embedded in src/cli/. The new architecture calls for clients to stream data to server APIs rather than processing locally. The CLI should become a client that extracts chat data from local sources and sends it to ChatHub API endpoints.",
+    "dependencies": [],
+    "decisions": [],
+    "acceptanceCriteria": [
+      "Create new @devlog/cli package structure",
+      "Move CLI functionality from @devlog/ai/src/cli to @devlog/cli/src",
+      "Update CLI to stream data to ChatHub API instead of local processing",
+      "Add @devlog/cli to monorepo workspace configuration",
+      "Update build scripts and CI to include new package",
+      "CLI maintains same user interface but changes backend behavior",
+      "Remove CLI code from @devlog/ai package",
+      "Update documentation to reference new package structure"
+    ],
+    "risks": []
+  },
+  "aiContext": {
+    "currentSummary": "",
+    "keyInsights": [],
+    "openQuestions": [],
+    "relatedPatterns": [],
+    "suggestedNextSteps": [],
+    "lastAIUpdate": "2025-07-24T14:17:07.399Z",
+    "contextVersion": 1
+  }
+}