Commit 5425da3
authored
feat: LLM conversation reassembly and viewer UI (#10)
* feat: MITM TLS interception for HTTP content inspection
Add TLS man-in-the-middle capability so greyproxy can decrypt and log
HTTPS request/response content flowing through the proxy.
- Add `greyproxy cert generate/install` CLI for CA certificate management
- Auto-inject MITM cert paths into HTTP/SOCKS5 handler metadata on startup
- Enable sniffing by default in greyproxy.yml for both proxy services
- Add OnHTTPRoundTrip callback to Sniffer for decrypted traffic hooks
- Wire [MITM] log output in both HTTP and SOCKS5 handlers
- Fix GenerateCertificate to auto-detect key type (ECDSA/Ed25519/RSA)
instead of hardcoding SHA256WithRSA
* feat: Phase 1 observability — capture and display MITM HTTP transactions
Add end-to-end HTTP transaction recording for MITM-intercepted requests:
- DB: migration for http_transactions table with indexes
- Models: HttpTransaction, HttpTransactionJSON, HttpTransactionCreateInput
- CRUD: create, get, query with filtering (container, destination, method, date range)
- API: GET /api/transactions (list) and GET /api/transactions/:id (detail with body)
- UI: Traffic page with HTMX table, filters, pagination, expandable row details
- Hook: GlobalHTTPRoundTripHook in sniffer + bridge via gostx.SetGlobalMitmHook
- Wire: program.go connects MITM hook to DB storage and EventBus
- Fix: auther srcAddrKey type mismatch — use xctx.SrcAddrFromContext for correct
client IP resolution (was causing unknown-unknown container names)
- Fix: remove custom lt/gt template funcs that shadowed Go builtins and broke
int64 comparisons in traffic table (only first row rendered)
- Tests: API handler tests (9), HTMX route tests (10), CRUD tests (5), plugin tests
* docs: add vibedocs with MITM proposal as 001-mitm.md
* feat: Phase 2 MITM - pending HTTP request approval, URL pattern matching, and UI
* feat: auto-detect Linux distro for CA cert install
Detect whether the system uses update-ca-trust (Arch, Fedora, RHEL) or
update-ca-certificates (Debian, Ubuntu) and run the install automatically
with sudo, falling back to manual instructions on failure.
* fix: increase the body capture to 2MB
Most content was truncated. But i need to see what's the required size
to support the 1M context from opus
* feat: support content decompression
* feat: experiment accross reassembly of http
* feat: step-by-step turn rendering, tool result merging, subagent linking, and system prompt capture
Restructure conversation turns from flat concatenated fields into ordered
steps that preserve the actual back-and-forth flow (assistant -> tool calls
-> tool results -> assistant). Tool results are now attached directly to
their corresponding tool calls. Subagent conversations are linked from
parent Agent tool calls. Full system prompt is captured in output JSON.
* feat: track conversation viewer and auto-copy to output directory
Move viewer.html to cmd/assembleconv/ so it's version-controlled.
The assembler now copies it into the output dir as index.html on each run.
* feat: incremental conversation assembler with API server and viewer
assemble2.py reads directly from greyproxy.db (bypassing the export step),
tracks a watermark for incremental processing, stores results in
conversation.db, and serves a REST API with a live web viewer.
* feat: native conversation dissector with Anthropic support and dashboard UI
Port the Python conversation assembler (assemble2.py) into Go as a native
greyproxy feature. Introduces a Wireshark-inspired dissector framework for
provider-specific HTTP body parsing, with Anthropic as the first dissector.
- Dissector framework: interface + registry in internal/greyproxy/dissector/
- Anthropic dissector: parses Messages API requests, SSE responses, extracts
session IDs, models, messages, system prompts, and tool calls
- SSE parser shared across providers
- Database: conversations, turns, processing_state tables (migration 7),
conversation_id FK on http_transactions (migration 8)
- Assembly engine: subscribes to EventTransactionNew, debounces, groups by
session, classifies threads, builds turns with tool result merging,
recovers SSE responses, links subagents. Backfills on startup.
- REST API: /api/conversations, /api/conversations/:id, /api/conversations/:id/subagents
- Conversations page: split-pane layout with sidebar list and detail panel,
collapsible system prompts, thinking blocks, tool calls, subagent navigation,
live updates via WebSocket
- Test fixtures from real Anthropic API transactions
* feat: improve conversation UI with markdown rendering, tool I/O display, URL state, and subagent linking
* feat: subagent navigation, assembler versioning, and settings maintenance UI
- Add "View subagent →" button on Agent tool calls linking to subagent conversations
- Add "← Back to parent" navigation when viewing a subagent conversation
- Improve linked subagents section with first prompt preview and model info
- Auto-scroll to top when navigating between conversations
- Add assembler version tracking with auto-rebuild on version change
- Add Settings > Advanced > Maintenance section with manual rebuild button
- Add Cache-Control: no-store on conversation detail HTMX responses
* refactor: remove HTTP pending requests and HTTP-level rule fields
Rules are back to domain/IP-level only (container, destination, port).
Removes method_pattern, path_pattern, content_action from rules, the
pending_http_requests table, and all associated API/UI/CRUD code.
Migrations renumbered from 8 to 6.
* feat: unified Activity view replacing separate Logs and Traffic pages
Merge connection logs and HTTP transactions into a single time-ordered
Activity page with UNION ALL query. Connections are deduplicated when
HTTP transactions exist for the same destination, with rule info
recovered via correlated subquery. Adds Live/Paused toggle for
real-time updates, DevTools-style header display with Raw toggle,
and full URL in expandable detail panes.
* feat: replace CONN badge with lock/unlock icons for decoded vs undecoded traffic
Use closed padlock + "TCP" for undecoded connections and open padlock +
HTTP method for decoded traffic. Rename filter options to "Decoded (HTTP)"
and "Undecoded (TCP)" to clarify why some traffic is not inspected.
* feat: add Live/Paused toggle to conversations page and preserve scroll position
Replace the static Refresh button with the same Live/Paused toggle pattern
used in the Activity page. When paused, incoming events are buffered and shown
as a badge count. Also preserve the conversation list scroll position on refresh.
* chore: remove prototype scripts and planning documents
Remove assemble2.py, viewer2.html (superseded by native Go assembler),
and root-level planning/research markdown files that are dev artifacts.
* chore: add Python and conversation.db exclusions to .gitignore
Add __pycache__/, *.pyc, *.pyo patterns and /conversation.db to
prevent development artifacts from being committed.
* fix: escape SQL LIKE wildcards in session ID matching
Session IDs containing % or _ were treated as LIKE wildcards,
causing loadTransactionsForSessions to match unrelated rows.
Add escapeLikePattern helper and ESCAPE '\' clause to the query.
Tests demonstrate the vulnerability (unescaped % matches all rows)
and verify the fix (escaped % matches none).
* fix: add mutex to conversation assembler to prevent race conditions
RebuildAllConversations (called from HTTP API) and processNewTransactions
(called from event loop and debounce timer) could run concurrently,
causing TOCTOU races on the last_processed_id cursor.
Add sync.Mutex with a locked internal method to serialize all processing.
* fix: handle errors instead of silently swallowing them in assembler
- Log warnings for rows.Scan failures instead of silent continue
- Log debug for json.Unmarshal failures on request bodies
- Log warnings for invalid strconv parsing of assembler_version and
last_processed_id
- Handle time.Parse failures gracefully in heuristic grouping
(keep entries in same group instead of using zero time)
- Add truncateUTF8 helper to avoid splitting multi-byte characters
when truncating thinking previews, system prompt summaries, and
subagent first prompts
* fix: eliminate data race in assembler debounce timer
The pending boolean was read/written from both the main event loop
goroutine and the time.AfterFunc callback without synchronization.
Replace with a channel-based trigger: the timer sends a signal on a
buffered channel, and the main select loop receives it to call
processNewTransactions. This keeps all state access on the main
goroutine.
* feat: MITM TLS visibility, cert management, global toggle, and skip reason tracking
Add end-to-end MITM interception visibility: skip reason tracking (no_cert,
mitm_disabled, mitm_error, non_http_after_tls, etc.), certificate management
API/UI, global MITM toggle in settings, and activity view dedup fixes.
Key fixes:
- RequestLog model was missing MitmSkipReason field (written but never read)
- Handlers now set mitm_error when HandleTLS returns an error
- UpdateLatestLogMitmSkipReason matches resolved_hostname (fixes IP vs SNI mismatch)
- Activity dedup handles IP-based connection logs vs hostname-based HTTP transactions
- Integration tests for HandleTLS covering 5 MITM scenarios
* feat: MITM TLS visibility, cert management, global toggle, skip reason tracking, and integration tests
Add end-to-end MITM interception visibility: skip reason tracking (no_cert,
mitm_disabled, mitm_error, non_http_after_tls, etc.), certificate management
API/UI, global MITM toggle in settings, and activity view fixes.
Key fixes:
- RequestLog model was missing MitmSkipReason field (written but never read)
- Handlers now set mitm_error when HandleTLS returns an error
- UpdateLatestLogMitmSkipReason matches resolved_hostname (fixes IP vs SNI mismatch)
- Activity dedup handles IP-based connection logs vs hostname-based HTTP transactions
- Integration tests for HandleTLS covering 5 MITM scenarios
* feat: compact collapsible tool call cards in conversation UI
- Tool calls now show as single-line cards with icon, name, and smart
summary (file path, command description, pattern, etc.)
- Click to expand and see full input/result with truncation controls
- Add expand/collapse all tools button in conversation header
- Extract tool_summary at dissection time from full input before the
300-char truncation, fixing broken previews for Edit/Write
- Unify Live button style between activity and conversations pages
- Fix activity Live button hover jump (overflow-hidden on ping dot)
- Flatten step layout: remove per-HTTP-request borders and padding
* fix: remove button hover translateY transform
* fix: handle new Anthropic metadata.user_id JSON object format for session extraction
The Anthropic API changed metadata.user_id from a plain string
("user_HASH_account_UUID_session_UUID") to a JSON object with a
session_id field. This broke session ID extraction, leaving all
new transactions unlinked to conversations.
* chore: remove vibedocs directory
* chore: remove exportlogs/assembleconv experiment
* feat: cert install improvements and cert uninstall command
- Print destination path on successful cert install
- Check if cert is already installed, require -f to overwrite
- Add `greyproxy cert uninstall` to remove CA from OS trust store
- `greyproxy uninstall` now also removes the CA certificate
* fix: remove environment variable hints section from settings UI
* chore: ignore local log/script
* fix: show all connections in activity log and eagerly report MITM skip reason
Remove activity dedup logic that hid connection rows when HTTP
transactions existed for the same destination. Old MITM-era transactions
were permanently suppressing connection rows even after MITM was disabled.
Fire OnMitmSkip callback immediately when MITM is skipped (before the
blocking pipe), so long-lived connections get their mitm_skip_reason
updated right away instead of waiting for connection close.
* feat: show notice that MITM toggle only affects new connections
* fix: redact real API key from testdata fixtures1 parent 07d32a8 commit 5425da3
File tree
60 files changed
+11613
-91
lines changed- cmd/greyproxy
- internal
- gostx
- handler
- http
- socks/v5
- internal/util
- sniffing
- tls
- recorder
- greyproxy
- api
- dissector
- testdata
- plugins
- static/css
- ui
- templates
- partials
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
60 files changed
+11613
-91
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
34 | 39 | | |
35 | 40 | | |
36 | 41 | | |
| |||
42 | 47 | | |
43 | 48 | | |
44 | 49 | | |
| 50 | + | |
45 | 51 | | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
249 | 249 | | |
250 | 250 | | |
251 | 251 | | |
| 252 | + | |
| 253 | + | |
252 | 254 | | |
253 | 255 | | |
254 | 256 | | |
255 | 257 | | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
256 | 261 | | |
257 | 262 | | |
258 | 263 | | |
| |||
284 | 289 | | |
285 | 290 | | |
286 | 291 | | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
287 | 297 | | |
288 | 298 | | |
289 | 299 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
145 | 148 | | |
146 | 149 | | |
147 | 150 | | |
| |||
159 | 162 | | |
160 | 163 | | |
161 | 164 | | |
| 165 | + | |
162 | 166 | | |
163 | 167 | | |
164 | 168 | | |
| |||
0 commit comments