|
| 1 | +--- |
| 2 | +title: Satellite Commands |
| 3 | +description: Command queue system for satellite orchestration including command types, priorities, and event payload patterns. |
| 4 | +sidebarTitle: Commands |
| 5 | +--- |
| 6 | + |
| 7 | +The backend uses a command queue system to orchestrate satellites, enabling dynamic MCP server management through prioritized commands with automatic retry logic. |
| 8 | + |
| 9 | +## Command Queue Architecture |
| 10 | + |
| 11 | +Commands are created by the backend and stored in the `satelliteCommands` table. Satellites poll for pending commands, execute them, and report results back to the backend. |
| 12 | + |
| 13 | +### Command Flow |
| 14 | + |
| 15 | +``` |
| 16 | +Backend → Creates Command → satelliteCommands Table → Satellite Polls → Executes → Reports Result → Backend Updates Status |
| 17 | +``` |
| 18 | + |
| 19 | +**Key Components:** |
| 20 | +- **Command Queue**: PostgreSQL table with priority-based ordering |
| 21 | +- **Polling**: Satellites poll every 2-60 seconds based on priority |
| 22 | +- **Execution**: Satellite processes command and performs action |
| 23 | +- **Result Reporting**: Satellite reports success/failure back to backend |
| 24 | + |
| 25 | +## Command Types |
| 26 | + |
| 27 | +The system supports 5 command types defined in the `command_type` enum: |
| 28 | + |
| 29 | +| Command Type | Purpose | Satellite Action | |
| 30 | +|---|---|---| |
| 31 | +| `configure` | Trigger configuration refresh | Fetch fresh config, apply changes | |
| 32 | +| `spawn` | Start MCP server process | Launch HTTP proxy or stdio process | |
| 33 | +| `kill` | Stop MCP server process | Terminate process gracefully | |
| 34 | +| `restart` | Restart MCP server | Stop and start process | |
| 35 | +| `health_check` | Verify server health | Call tools/list to check connectivity | |
| 36 | + |
| 37 | +### Configure Commands |
| 38 | + |
| 39 | +**Most Common**: Used for all MCP installation lifecycle events (create/update/delete). |
| 40 | + |
| 41 | +**How It Works:** |
| 42 | +1. Backend creates `configure` command with event metadata |
| 43 | +2. Satellite fetches fresh configuration from backend |
| 44 | +3. DynamicConfigManager compares old vs. new config |
| 45 | +4. Identifies added/removed/modified servers |
| 46 | +5. Takes appropriate action (spawn/restart/terminate) |
| 47 | + |
| 48 | +**Key Insight**: The satellite doesn't parse individual event types. All `configure` commands trigger a full configuration refresh where changes are detected by comparing configurations. |
| 49 | + |
| 50 | +## Command Priorities |
| 51 | + |
| 52 | +Commands are processed based on priority: |
| 53 | + |
| 54 | +| Priority | Use Case | Polling Interval | |
| 55 | +|---|---|---| |
| 56 | +| `immediate` | New installations, deletions | 2 seconds | |
| 57 | +| `high` | Server recovery, critical updates | 10 seconds | |
| 58 | +| `normal` | Configuration updates | 30 seconds | |
| 59 | +| `low` | Maintenance tasks | 60 seconds | |
| 60 | + |
| 61 | +Higher priority commands are retrieved first when satellite polls. |
| 62 | + |
| 63 | +## Command Payload Structure |
| 64 | + |
| 65 | +All commands include a JSON payload with event-specific data: |
| 66 | + |
| 67 | +```typescript |
| 68 | +interface CommandPayload { |
| 69 | + event?: string; // Event type for logging/tracking |
| 70 | + installation_id?: string; // MCP installation identifier |
| 71 | + team_id?: string; // Team context |
| 72 | + server_name?: string; // Legacy server identifier |
| 73 | + // Additional command-specific fields |
| 74 | +} |
| 75 | +``` |
| 76 | + |
| 77 | +## Command Event Types |
| 78 | + |
| 79 | +All `configure` commands include an `event` field in the payload for tracking and logging: |
| 80 | + |
| 81 | +| Event Type | When Sent | Purpose | |
| 82 | +|---|---|---| |
| 83 | +| `mcp_installation_created` | New MCP installation | Satellite refreshes config, spawns new servers | |
| 84 | +| `mcp_installation_updated` | Config changes (args/env/headers) | Satellite refreshes config, restarts affected servers | |
| 85 | +| `mcp_installation_deleted` | Installation removed | Satellite refreshes config, terminates removed servers | |
| 86 | +| `mcp_recovery` | Server recovered from failure | Satellite rediscovers tools for recovered server | |
| 87 | + |
| 88 | +### Event Field Usage |
| 89 | + |
| 90 | +**Current Implementation**: The satellite does NOT parse the `event` field. All configure commands trigger a full configuration refresh where the satellite: |
| 91 | + |
| 92 | +1. Fetches fresh config from backend |
| 93 | +2. Compares with current running config |
| 94 | +3. Identifies added/removed/modified servers |
| 95 | +4. Takes appropriate action |
| 96 | + |
| 97 | +**Purpose of Event Field**: |
| 98 | +- Database record keeping |
| 99 | +- Structured logging |
| 100 | +- Future extensibility |
| 101 | +- Developer debugging |
| 102 | + |
| 103 | +**Example payload** for deletion: |
| 104 | +```json |
| 105 | +{ |
| 106 | + "event": "mcp_installation_deleted", |
| 107 | + "installation_id": "U3hCfHenbK5kH2_8ehSGx", |
| 108 | + "team_id": "4vj7igb2fcwzmko" |
| 109 | +} |
| 110 | +``` |
| 111 | + |
| 112 | +## SatelliteCommandService API |
| 113 | + |
| 114 | +The backend provides convenience methods for creating commands: |
| 115 | + |
| 116 | +### notifyMcpInstallation() |
| 117 | + |
| 118 | +Creates immediate priority configure commands when a new MCP server is installed. |
| 119 | + |
| 120 | +```typescript |
| 121 | +await satelliteCommandService.notifyMcpInstallation( |
| 122 | + installationId, |
| 123 | + teamId, |
| 124 | + userId |
| 125 | +); |
| 126 | +``` |
| 127 | + |
| 128 | +**Payload**: `event: 'mcp_installation_created'` |
| 129 | + |
| 130 | +### notifyMcpUpdate() |
| 131 | + |
| 132 | +Creates immediate priority configure commands when MCP configuration is updated. |
| 133 | + |
| 134 | +```typescript |
| 135 | +await satelliteCommandService.notifyMcpUpdate( |
| 136 | + installationId, |
| 137 | + teamId, |
| 138 | + userId |
| 139 | +); |
| 140 | +``` |
| 141 | + |
| 142 | +**Payload**: `event: 'mcp_installation_updated'` |
| 143 | + |
| 144 | +### notifyMcpDeletion() |
| 145 | + |
| 146 | +Creates immediate priority configure commands when an MCP installation is deleted. |
| 147 | + |
| 148 | +```typescript |
| 149 | +await satelliteCommandService.notifyMcpDeletion( |
| 150 | + installationId, |
| 151 | + teamId, |
| 152 | + userId |
| 153 | +); |
| 154 | +``` |
| 155 | + |
| 156 | +**Payload**: `event: 'mcp_installation_deleted'` |
| 157 | + |
| 158 | +### notifyMcpRecovery() |
| 159 | + |
| 160 | +Creates high priority configure commands when a server recovers from failure. |
| 161 | + |
| 162 | +```typescript |
| 163 | +await satelliteCommandService.notifyMcpRecovery( |
| 164 | + installationId, |
| 165 | + teamId |
| 166 | +); |
| 167 | +``` |
| 168 | + |
| 169 | +**Payload**: `event: 'mcp_recovery'` |
| 170 | + |
| 171 | +## Critical Pattern |
| 172 | + |
| 173 | +**ALWAYS use the correct convenience method**: |
| 174 | + |
| 175 | +- Installation created → `notifyMcpInstallation()` |
| 176 | +- Installation updated → `notifyMcpUpdate()` |
| 177 | +- Installation deleted → `notifyMcpDeletion()` |
| 178 | +- Server recovered → `notifyMcpRecovery()` |
| 179 | + |
| 180 | +**NEVER** call the wrong method for an operation! For example, don't call `notifyMcpInstallation()` for delete operations. |
| 181 | + |
| 182 | +## Command Lifecycle |
| 183 | + |
| 184 | +Commands progress through the following states: |
| 185 | + |
| 186 | +| Status | Description | |
| 187 | +|---|---| |
| 188 | +| `pending` | Command created, awaiting satellite poll | |
| 189 | +| `acknowledged` | Satellite retrieved command | |
| 190 | +| `executing` | Satellite processing command | |
| 191 | +| `completed` | Command executed successfully | |
| 192 | +| `failed` | Command execution failed | |
| 193 | + |
| 194 | +### Retry Logic |
| 195 | + |
| 196 | +Failed commands are automatically retried: |
| 197 | + |
| 198 | +- **Max Retries**: 3 attempts |
| 199 | +- **Retry Delay**: Exponential backoff |
| 200 | +- **Expiration**: Commands expire after 5 minutes |
| 201 | + |
| 202 | +## Database Schema |
| 203 | + |
| 204 | +**Table**: `satelliteCommands` |
| 205 | + |
| 206 | +**Key Fields**: |
| 207 | +- `id`: Command UUID |
| 208 | +- `satellite_id`: Target satellite |
| 209 | +- `command_type`: One of 5 command types |
| 210 | +- `priority`: Command priority level |
| 211 | +- `payload`: JSON command data (includes event type) |
| 212 | +- `status`: Current execution state |
| 213 | +- `target_team_id`: Team context |
| 214 | +- `correlation_id`: Request tracing |
| 215 | +- `retry_count`: Retry attempts |
| 216 | +- `error_message`: Failure details |
| 217 | +- `result`: Execution result |
| 218 | + |
| 219 | +## Command Processing on Satellite |
| 220 | + |
| 221 | +When satellites receive commands: |
| 222 | + |
| 223 | +**For `configure` commands**: |
| 224 | +1. Fetch fresh configuration from backend |
| 225 | +2. Compare with current running config |
| 226 | +3. Identify changes (added/removed/modified servers) |
| 227 | +4. Execute appropriate actions: |
| 228 | + - **Added**: Spawn new processes |
| 229 | + - **Modified**: Restart affected processes |
| 230 | + - **Removed**: Terminate processes |
| 231 | + |
| 232 | +**For `spawn` commands**: |
| 233 | +1. Parse server configuration |
| 234 | +2. For stdio: Launch subprocess with JSON-RPC |
| 235 | +3. For HTTP/SSE: Create proxy tracker entry |
| 236 | +4. Discover tools via tools/list call |
| 237 | + |
| 238 | +**For `kill` commands**: |
| 239 | +1. Locate running process |
| 240 | +2. Send SIGTERM for graceful shutdown |
| 241 | +3. Wait for timeout (10 seconds) |
| 242 | +4. Send SIGKILL if needed |
| 243 | + |
| 244 | +**For `restart` commands**: |
| 245 | +1. Execute kill sequence |
| 246 | +2. Wait for process termination |
| 247 | +3. Execute spawn sequence |
| 248 | + |
| 249 | +**For `health_check` commands**: |
| 250 | +1. Call tools/list on target server |
| 251 | +2. Verify response |
| 252 | +3. Report health status |
| 253 | + |
| 254 | +## Example Usage |
| 255 | + |
| 256 | +### Creating Commands for Installation Deletion |
| 257 | + |
| 258 | +```typescript |
| 259 | +// In route handler |
| 260 | +const satelliteCommandService = new SatelliteCommandService(db, logger); |
| 261 | +const commands = await satelliteCommandService.notifyMcpDeletion( |
| 262 | + installationId, |
| 263 | + teamId, |
| 264 | + userId |
| 265 | +); |
| 266 | + |
| 267 | +logger.info({ |
| 268 | + installationId, |
| 269 | + commandsCreated: commands.length, |
| 270 | + satelliteIds: commands.map(c => c.satellite_id) |
| 271 | +}, 'Satellite commands created for deletion'); |
| 272 | +``` |
| 273 | + |
| 274 | +### Creating Commands for Configuration Update |
| 275 | + |
| 276 | +```typescript |
| 277 | +// After updating environment variables |
| 278 | +const satelliteCommandService = new SatelliteCommandService(db, logger); |
| 279 | +const commands = await satelliteCommandService.notifyMcpUpdate( |
| 280 | + installationId, |
| 281 | + teamId, |
| 282 | + userId |
| 283 | +); |
| 284 | +``` |
| 285 | + |
| 286 | +## Monitoring Commands |
| 287 | + |
| 288 | +### Check Command Status |
| 289 | + |
| 290 | +```sql |
| 291 | +SELECT id, command_type, status, priority, created_at, updated_at |
| 292 | +FROM "satelliteCommands" |
| 293 | +WHERE satellite_id = 'sat-123' |
| 294 | +ORDER BY created_at DESC |
| 295 | +LIMIT 10; |
| 296 | +``` |
| 297 | + |
| 298 | +### View Command Payload |
| 299 | + |
| 300 | +```sql |
| 301 | +SELECT id, command_type, payload::json->>'event' as event_type, payload |
| 302 | +FROM "satelliteCommands" |
| 303 | +WHERE id = 'cmd-456'; |
| 304 | +``` |
| 305 | + |
| 306 | +### Monitor Failed Commands |
| 307 | + |
| 308 | +```sql |
| 309 | +SELECT id, command_type, status, retry_count, error_message |
| 310 | +FROM "satelliteCommands" |
| 311 | +WHERE status = 'failed' |
| 312 | +AND retry_count >= max_retries; |
| 313 | +``` |
| 314 | + |
| 315 | +## Related Documentation |
| 316 | + |
| 317 | +- [Satellite Communication](/development/backend/satellite/communication) - Overall communication architecture |
| 318 | +- [Satellite Events](/development/backend/satellite/events) - Event system for real-time updates |
| 319 | +- [Database Management](/development/backend/database/) - Schema and migrations |
0 commit comments