Skip to content

Commit d995b78

Browse files
authored
[HDX-3919] Add @hyperdx/cli package — terminal TUI, source map upload, and agent-friendly commands (#2043)
## Summary Adds `packages/cli` (`@hyperdx/cli`) — a unified CLI for HyperDX that provides an interactive TUI for searching/tailing logs and traces, source map upload (migrated from `hyperdx-js`), and agent-friendly commands for programmatic access. Ref: HDX-3919 Ref: HDX-3920 ### CLI Commands ``` hdx tui -s <url> # Interactive TUI (main command) hdx sources -s <url> # List sources with ClickHouse schemas hdx sources -s <url> --json # JSON output for agents / scripts hdx dashboards -s <url> # List dashboards with tile summaries hdx dashboards -s <url> --json # JSON output for agents / scripts hdx query --source "Logs" --sql "SELECT count() FROM default.otel_logs" hdx upload-sourcemaps -k <key> # Upload source maps (ported from hyperdx-js) hdx auth login -s <url> # Sign in (interactive or -e/-p flags) hdx auth status # Show auth status hdx auth logout # Clear saved session ``` ### Key Features **Interactive TUI (`hdx tui`)** - Table view with dynamic columns derived from query results (percentage-based widths) - Follow mode (live tail) enabled by default, auto-pauses when detail panel is open - Vim-like keybindings: j/k navigation, G/g jump, Ctrl+D/U half-page scroll - `/` for Lucene search, `s` to edit SELECT clause in `$EDITOR`, `t` to edit time range - Tab to cycle between sources and saved searches - `w` to toggle line wrap **Detail Panel (3 tabs, full-screen with Ctrl+D/U scrolling)** - **Overview** — Structured view: Top Level Attributes, Log/Span Attributes, Resource Attributes - **Column Values** — Full `SELECT *` row data with `__hdx_*` aliased columns - **Trace** — Waterfall chart (port of `DBTraceWaterfallChart` DAG builder) with correlated log events, j/k span navigation, inverse highlight, Event Details section **Agent-friendly commands** - `hdx sources --json` — Full source metadata with ClickHouse `CREATE TABLE` DDL, expression mappings, and correlated source IDs. Detailed `--help` describes the JSON schema for LLM consumption. Schema queries run in parallel. - `hdx dashboards --json` — Dashboard metadata with simplified tile summaries (name, type, source, sql). Resolves source names for human-readable output. - `hdx query --source <name> --sql <query>` — Raw SQL execution against any source's ClickHouse connection. Supports `--format` for ClickHouse output formats (JSON, JSONEachRow, CSV, etc.). **Source map upload (`hdx upload-sourcemaps`)** - Ported from `hyperdx-js/packages/cli` to consolidate on a single `@hyperdx/cli` package - Authenticates via service account API key (`-k` / `HYPERDX_SERVICE_KEY` env var) - Globs `.js` and `.js.map` files, handles Next.js route groups - Uploads to presigned URLs in parallel with retry (3 attempts) and progress - Modernized: native `fetch` (Node 22+), ESM-compatible, proper TypeScript types ### Architecture ``` packages/cli/ ├── src/ │ ├── cli.tsx # Commander CLI: tui, sources, dashboards, query, │ │ # upload-sourcemaps, auth │ ├── App.tsx # Ink app shell (login → source picker → EventViewer) │ ├── sourcemaps.ts # Source map upload logic (ported from hyperdx-js) │ ├── api/ │ │ ├── client.ts # ApiClient + ProxyClickhouseClient │ │ └── eventQuery.ts # Query builders (renderChartConfig, raw SQL) │ ├── components/ │ │ ├── EventViewer/ # Main TUI (9 files) │ │ │ ├── EventViewer.tsx # Orchestrator (state, hooks, render shell) │ │ │ ├── types.ts # Shared types & constants │ │ │ ├── utils.ts # Row formatting functions │ │ │ ├── SubComponents.tsx # Header, TabBar, SearchBar, Footer, HelpScreen, TableHeader │ │ │ ├── TableView.tsx # Table rows rendering │ │ │ ├── DetailPanel.tsx # Detail panel (overview/columns/trace tabs) │ │ │ ├── useEventData.ts # Data fetching hook │ │ │ └── useKeybindings.ts # Input handler hook │ │ ├── TraceWaterfall/ # Trace chart (6 files) │ │ │ ├── TraceWaterfall.tsx # Orchestrator + render │ │ │ ├── types.ts # SpanRow, SpanNode, props │ │ │ ├── utils.ts # Duration/status/bar helpers │ │ │ ├── buildTree.ts # DAG builder (port of DBTraceWaterfallChart) │ │ │ └── useTraceData.ts # Data fetching hook │ │ ├── RowOverview.tsx │ │ ├── ColumnValues.tsx │ │ ├── LoginForm.tsx │ │ └── SourcePicker.tsx │ ├── shared/ # Ported from packages/app (@source annotated) │ └── utils/ # Config, editor, log silencing ├── AGENTS.md ├── CONTRIBUTING.md └── README.md ``` ### Tech Stack - **Ink v6.8.0** (React 19 for terminals) + Commander.js - **@clickhouse/client** via ProxyClickhouseClient (routes through `/clickhouse-proxy`) - **@hyperdx/common-utils** for query generation (`renderChartConfig`, `chSqlToAliasMap`) - **glob v13** for source map file discovery - **tsup** for bundling (all deps bundled via `noExternal: [/.*/]`, zero runtime deps) - **Bun 1.3.11** for standalone binary compilation - Session stored at `~/.config/hyperdx/cli/session.json` ### CI/CD (`release.yml`) - CLI binaries compiled for macOS ARM64, macOS x64, and Linux x64 - GitHub Release created with download instructions - Version-change gate: skips release if `cli-v{version}` tag already exists - `softprops/action-gh-release` pinned to full SHA (v2.6.1) for supply chain safety - Bun pinned to `1.3.11` for reproducible builds - npm publishing handled by changesets ### Keybindings | Key | Action | |---|---| | `j/k` | Navigate rows (or spans in Trace tab) | | `l/Enter` | Expand row detail | | `h/Esc` | Close detail / blur search | | `G/g` | Jump to newest/oldest | | `Ctrl+D/U` | Scroll half-page (table, detail panels, Event Details) | | `/` | Search (global or detail filter) | | `Tab` | Cycle sources/searches or detail tabs | | `s` | Edit SELECT clause in $EDITOR | | `t` | Edit time range in $EDITOR | | `f` | Toggle follow mode (live tail) | | `w` | Toggle line wrap | | `?` | Help screen | ### Demo #### Main Search View <img width="1004" height="1014" alt="image" src="https://github.com/user-attachments/assets/bb6a7f00-38c9-4281-9915-c71b65d852f8" /> #### Event Details Overview <img width="1003" height="1024" alt="image" src="https://github.com/user-attachments/assets/57025fa5-fddb-452a-9320-93465538d5b2" /> #### Trace Waterfall <img width="1004" height="1029" alt="image" src="https://github.com/user-attachments/assets/3443c898-ea0d-47f3-acc5-edb7cdd31946" />
1 parent 5de23e1 commit d995b78

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+6765
-6
lines changed

.changeset/add-hyperdx-cli.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@hyperdx/cli": minor
3+
---
4+
5+
Add @hyperdx/cli package — terminal CLI for searching, tailing, and inspecting logs and traces from HyperDX with interactive TUI, trace waterfall, raw SQL queries, dashboard listing, and sourcemap uploads.

.github/workflows/release.yml

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -418,6 +418,104 @@ jobs:
418418
"${IMAGE}:${VERSION}-arm64"
419419
done
420420
421+
# ---------------------------------------------------------------------------
422+
# CLI – compile standalone binaries and upload as GitHub Release assets
423+
# npm publishing is handled by changesets in the check_changesets job above.
424+
# This job only compiles platform-specific binaries and creates a GH Release.
425+
# ---------------------------------------------------------------------------
426+
release-cli:
427+
name: Release CLI Binaries
428+
needs: [check_changesets, check_version]
429+
if: needs.check_version.outputs.should_release == 'true'
430+
runs-on: ubuntu-24.04
431+
steps:
432+
- name: Checkout
433+
uses: actions/checkout@v4
434+
- name: Setup node
435+
uses: actions/setup-node@v4
436+
with:
437+
node-version-file: '.nvmrc'
438+
cache-dependency-path: 'yarn.lock'
439+
cache: 'yarn'
440+
- name: Setup Bun
441+
uses: oven-sh/setup-bun@v2
442+
with:
443+
bun-version: '1.3.11'
444+
- name: Install dependencies
445+
run: yarn install
446+
- name: Get CLI version
447+
id: cli_version
448+
run: |
449+
CLI_VERSION=$(node -p "require('./packages/cli/package.json').version")
450+
echo "version=${CLI_VERSION}" >> $GITHUB_OUTPUT
451+
echo "CLI version: ${CLI_VERSION}"
452+
- name: Check if CLI release already exists
453+
id: check_cli_release
454+
env:
455+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
456+
run: |
457+
if gh release view "cli-v${{ steps.cli_version.outputs.version }}" > /dev/null 2>&1; then
458+
echo "Release cli-v${{ steps.cli_version.outputs.version }} already exists. Skipping."
459+
echo "exists=true" >> $GITHUB_OUTPUT
460+
else
461+
echo "Release does not exist. Proceeding."
462+
echo "exists=false" >> $GITHUB_OUTPUT
463+
fi
464+
- name: Compile CLI binaries
465+
if: steps.check_cli_release.outputs.exists == 'false'
466+
working-directory: packages/cli
467+
run: |
468+
yarn compile:linux
469+
yarn compile:macos
470+
yarn compile:macos-x64
471+
- name: Create GitHub Release
472+
if: steps.check_cli_release.outputs.exists == 'false'
473+
uses: softprops/action-gh-release@153bb8e04406b158c6c84fc1615b65b24149a1fe # v2.6.1
474+
with:
475+
tag_name: cli-v${{ steps.cli_version.outputs.version }}
476+
name: '@hyperdx/cli v${{ steps.cli_version.outputs.version }}'
477+
body: |
478+
## @hyperdx/cli v${{ steps.cli_version.outputs.version }}
479+
480+
### Installation
481+
482+
**npm (recommended):**
483+
```bash
484+
npm install -g @hyperdx/cli
485+
```
486+
487+
**Or run directly with npx:**
488+
```bash
489+
npx @hyperdx/cli tui -s <your-hyperdx-api-url>
490+
```
491+
492+
**Manual download (standalone binary, no Node.js required):**
493+
```bash
494+
# macOS Apple Silicon
495+
curl -L https://github.com/hyperdxio/hyperdx/releases/download/cli-v${{ steps.cli_version.outputs.version }}/hdx-darwin-arm64 -o hdx
496+
# macOS Intel
497+
curl -L https://github.com/hyperdxio/hyperdx/releases/download/cli-v${{ steps.cli_version.outputs.version }}/hdx-darwin-x64 -o hdx
498+
# Linux x64
499+
curl -L https://github.com/hyperdxio/hyperdx/releases/download/cli-v${{ steps.cli_version.outputs.version }}/hdx-linux-x64 -o hdx
500+
501+
chmod +x hdx && sudo mv hdx /usr/local/bin/
502+
```
503+
504+
### Usage
505+
506+
```bash
507+
hdx auth login -s <your-hyperdx-api-url>
508+
hdx tui
509+
```
510+
draft: false
511+
prerelease: false
512+
files: |
513+
packages/cli/dist/hdx-linux-x64
514+
packages/cli/dist/hdx-darwin-arm64
515+
packages/cli/dist/hdx-darwin-x64
516+
env:
517+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
518+
421519
# ---------------------------------------------------------------------------
422520
# Downstream notifications
423521
# ---------------------------------------------------------------------------
@@ -553,6 +651,7 @@ jobs:
553651
publish-otel-collector,
554652
publish-local,
555653
publish-all-in-one,
654+
release-cli,
556655
notify_helm_charts,
557656
notify_ch,
558657
notify_clickhouse_clickstack,

packages/cli/.gitignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Build output
2+
dist/
3+
4+
# Bun build artifacts
5+
*.bun-build

packages/cli/AGENTS.md

Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
# @hyperdx/cli Development Guide
2+
3+
## What is @hyperdx/cli?
4+
5+
A terminal CLI for searching, tailing, and inspecting logs and traces from
6+
HyperDX. It provides both an interactive TUI (built with Ink — React for
7+
terminals) and a non-interactive streaming mode for piping.
8+
9+
The CLI connects to the HyperDX API server and queries ClickHouse directly
10+
through the API's `/clickhouse-proxy` endpoint, using the same query generation
11+
logic (`@hyperdx/common-utils`) as the web frontend.
12+
13+
## CLI Commands
14+
15+
```
16+
hdx tui -s <url> # Interactive TUI (main command)
17+
hdx stream -s <url> --source "Logs" # Non-interactive streaming to stdout
18+
hdx sources -s <url> # List available sources
19+
hdx auth login -s <url> # Sign in (interactive or -e/-p flags)
20+
hdx auth status # Show auth status (reads saved session)
21+
hdx auth logout # Clear saved session
22+
```
23+
24+
The `-s, --server <url>` flag is required for commands that talk to the API. If
25+
omitted, the CLI falls back to the server URL saved in the session file from a
26+
previous `hdx auth login`.
27+
28+
## Architecture
29+
30+
```
31+
src/
32+
├── cli.tsx # Entry point — Commander CLI with commands:
33+
│ # tui, stream, sources, auth (login/logout/status)
34+
│ # Also contains the standalone LoginPrompt component
35+
├── App.tsx # App shell — state machine:
36+
│ # loading → login → pick-source → EventViewer
37+
├── api/
38+
│ ├── client.ts # ApiClient (REST + session cookies)
39+
│ │ # ProxyClickhouseClient (routes through /clickhouse-proxy)
40+
│ └── eventQuery.ts # Query builders:
41+
│ # buildEventSearchQuery (table view, uses renderChartConfig)
42+
│ # buildTraceSpansSql (waterfall trace spans)
43+
│ # buildTraceLogsSql (waterfall correlated logs)
44+
│ # buildFullRowSql (SELECT * for row detail)
45+
├── components/
46+
│ ├── EventViewer.tsx # Main TUI view — table, search, detail panel with tabs
47+
│ ├── TraceWaterfall.tsx # Trace waterfall chart with j/k navigation + event details
48+
│ ├── RowOverview.tsx # Structured overview (top-level attrs, event attrs, resource attrs)
49+
│ ├── ColumnValues.tsx # Shared key-value renderer (used by Column Values tab + Event Details)
50+
│ ├── LoginForm.tsx # Email/password login form (used inside TUI App)
51+
│ └── SourcePicker.tsx # Arrow-key source selector
52+
└── utils/
53+
├── config.ts # Session persistence (~/.config/hyperdx/cli/session.json)
54+
├── editor.ts # $EDITOR integration for time range and select clause editing
55+
└── silenceLogs.ts # Suppresses console.debug/warn/error from common-utils
56+
```
57+
58+
## Key Components
59+
60+
### EventViewer (`components/EventViewer.tsx`)
61+
62+
The main TUI component (~1100 lines). Handles:
63+
64+
- **Table view**: Dynamic columns derived from query results, percentage-based
65+
widths, `overflowX="hidden"` for truncation
66+
- **Search**: Lucene query via `/` key, submits on Enter
67+
- **Follow mode**: Slides time range forward every 2s, pauses when detail panel
68+
is open, restores on close
69+
- **Detail panel**: Three tabs (Overview / Column Values / Trace), cycled via
70+
Tab key. Detail search via `/` filters content within the active tab.
71+
- **Select editor**: `s` key opens `$EDITOR` with the current SELECT clause.
72+
Custom selects are stored per source ID.
73+
- **State management**: ~20 `useState` hooks. Key states include `events`,
74+
`expandedRow`, `detailTab`, `isFollowing`, `customSelectMap`,
75+
`traceSelectedIndex`.
76+
77+
### TraceWaterfall (`components/TraceWaterfall.tsx`)
78+
79+
Port of the web frontend's `DBTraceWaterfallChart`. Key details:
80+
81+
- **Tree building**: Single-pass DAG builder over time-sorted rows. Direct port
82+
of the web frontend's logic — do NOT modify without checking
83+
`DBTraceWaterfallChart` first.
84+
- **Correlated logs**: Fetches log events via `buildTraceLogsSql` and merges
85+
them into the span tree (logs attach as children of the span with matching
86+
SpanId, using `SpanId-log` suffix to avoid key collisions).
87+
- **j/k navigation**: `selectedIndex` + `onSelectedIndexChange` props controlled
88+
by EventViewer. `effectiveIndex` falls back to `highlightHint` when no j/k
89+
navigation has occurred.
90+
- **Event Details**: `SELECT *` fetch for the selected span/log, rendered via
91+
the shared `ColumnValues` component. Uses stable scalar deps
92+
(`selectedNodeSpanId`, `selectedNodeTimestamp`, `selectedNodeKind`) to avoid
93+
infinite re-fetch loops.
94+
- **Duration formatting**: Dynamic units — `1.2s`, `3.5ms`, `45.2μs`, `123ns`.
95+
- **Highlight**: `inverse` on label and duration text for the selected row. Bar
96+
color unchanged.
97+
98+
### RowOverview (`components/RowOverview.tsx`)
99+
100+
Port of the web frontend's `DBRowOverviewPanel`. Three sections:
101+
102+
1. **Top Level Attributes**: Standard OTel fields (TraceId, SpanId, SpanName,
103+
ServiceName, Duration, StatusCode, etc.)
104+
2. **Span/Log Attributes**: Flattened from `source.eventAttributesExpression`,
105+
shown with key count header
106+
3. **Resource Attributes**: Flattened from
107+
`source.resourceAttributesExpression`, rendered as chips with
108+
`backgroundColor="#3a3a3a"` and cyan key / white value
109+
110+
### ColumnValues (`components/ColumnValues.tsx`)
111+
112+
Shared component for rendering key-value pairs from a row data object. Used by:
113+
114+
- Column Values tab in the detail panel
115+
- Event Details section in the Trace tab's waterfall
116+
117+
Supports `searchQuery` filtering and `wrapLines` toggle.
118+
119+
## Web Frontend Alignment
120+
121+
This package mirrors several web frontend components. **Always check the
122+
corresponding web component before making changes** to ensure behavior stays
123+
consistent:
124+
125+
| CLI Component | Web Component | Notes |
126+
| ---------------- | ----------------------- | -------------------------------- |
127+
| `TraceWaterfall` | `DBTraceWaterfallChart` | Tree builder is a direct port |
128+
| `RowOverview` | `DBRowOverviewPanel` | Same sections and field list |
129+
| Trace tab logic | `DBTracePanel` | Source resolution (trace/log) |
130+
| Detail panel | `DBRowSidePanel` | Tab structure, highlight hint |
131+
| Event query | `DBTraceWaterfallChart` | `getConfig()``buildTrace*Sql` |
132+
133+
Key expression mappings from the web frontend's `getConfig()`:
134+
135+
- `Timestamp``displayedTimestampValueExpression` (NOT
136+
`timestampValueExpression`)
137+
- `Duration``durationExpression` (raw, not seconds like web frontend)
138+
- `Body``bodyExpression` (logs) or `spanNameExpression` (traces)
139+
- `SpanId``spanIdExpression`
140+
- `ParentSpanId``parentSpanIdExpression` (traces only)
141+
142+
## Keybindings (TUI mode)
143+
144+
| Key | Action |
145+
| ------------- | ------------------------------------------ |
146+
| `j` / `` | Move selection down |
147+
| `k` / `` | Move selection up |
148+
| `l` / `Enter` | Expand row detail |
149+
| `h` / `Esc` | Close detail / blur search |
150+
| `G` | Jump to newest |
151+
| `g` | Jump to oldest |
152+
| `/` | Search (global in table, filter in detail) |
153+
| `Tab` | Cycle sources/searches or detail tabs |
154+
| `Shift+Tab` | Cycle backwards |
155+
| `s` | Edit SELECT clause in $EDITOR |
156+
| `t` | Edit time range in $EDITOR |
157+
| `f` | Toggle follow mode (live tail) |
158+
| `w` | Toggle line wrap |
159+
| `?` | Toggle help screen |
160+
| `q` | Quit |
161+
162+
In the **Trace tab**, `j`/`k` navigate spans/logs in the waterfall instead of
163+
the main table.
164+
165+
## Development
166+
167+
```bash
168+
# Run in dev mode (tsx, no compile step)
169+
cd packages/cli
170+
yarn dev tui -s http://localhost:8000
171+
172+
# Type check
173+
npx tsc --noEmit
174+
175+
# Bundle with tsup
176+
yarn build
177+
178+
# Compile standalone binary (current platform)
179+
yarn compile
180+
181+
# Cross-compile
182+
yarn compile:macos # macOS ARM64
183+
yarn compile:macos-x64 # macOS x64
184+
yarn compile:linux # Linux x64
185+
```
186+
187+
## Key Patterns
188+
189+
### Session Management
190+
191+
Session is stored at `~/.config/hyperdx/cli/session.json` with mode `0o600`.
192+
Contains `apiUrl` and `cookies[]`. The `ApiClient` constructor loads the saved
193+
session and checks if the stored `apiUrl` matches the requested one.
194+
195+
### ClickHouse Proxy Client
196+
197+
`ProxyClickhouseClient` extends `BaseClickhouseClient` from common-utils. It:
198+
199+
- Routes queries through `/clickhouse-proxy` (sets `pathname`)
200+
- Injects session cookies for auth
201+
- Passes `x-hyperdx-connection-id` header
202+
- Disables basic auth (`set_basic_auth_header: false`)
203+
- Forces `content-type: text/plain` to prevent Express body parser issues
204+
205+
### Source Expressions
206+
207+
Sources have many expression fields that map to ClickHouse column names. Key
208+
ones used in the CLI:
209+
210+
- `timestampValueExpression` — Primary timestamp (often `TimestampTime`,
211+
DateTime)
212+
- `displayedTimestampValueExpression` — High-precision timestamp (often
213+
`Timestamp`, DateTime64 with nanoseconds). **Use this for waterfall queries.**
214+
- `traceIdExpression`, `spanIdExpression`, `parentSpanIdExpression`
215+
- `bodyExpression`, `spanNameExpression`, `serviceNameExpression`
216+
- `durationExpression` + `durationPrecision` (3=ms, 6=μs, 9=ns)
217+
- `eventAttributesExpression`, `resourceAttributesExpression`
218+
- `logSourceId`, `traceSourceId` — Correlated source IDs
219+
220+
### useInput Handler Ordering
221+
222+
The `useInput` callback in EventViewer has a specific priority order. **Do not
223+
reorder these checks**:
224+
225+
1. `?` toggles help (except when search focused)
226+
2. Any key closes help when showing
227+
3. `focusDetailSearch` — consumes all keys except Esc/Enter
228+
4. `focusSearch` — consumes all keys except Tab/Esc
229+
5. Trace tab j/k — when detail panel open and Trace tab active
230+
6. General j/k, G/g, Enter/Esc, Tab, etc.
231+
7. Single-key shortcuts: `w`, `f`, `/`, `s`, `t`, `q`
232+
233+
### Dynamic Table Columns
234+
235+
When `customSelect` is set (via `s` key), columns are derived from the query
236+
result keys. Otherwise, hardcoded percentage-based columns are used per source
237+
kind. The `getDynamicColumns` function distributes 60% evenly among non-last
238+
columns, with the last column getting the remainder.
239+
240+
### Follow Mode
241+
242+
- Enabled by default on startup
243+
- Slides `timeRange` forward every 2s, triggering a replace fetch
244+
- **Paused** when detail panel opens (`wasFollowingRef` saves previous state)
245+
- **Restored** when detail panel closes
246+
247+
### Custom Select Per Source
248+
249+
`customSelectMap: Record<string, string>` stores custom SELECT overrides keyed
250+
by `source.id`. Each source remembers its own custom select independently.

0 commit comments

Comments
 (0)