Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
250 changes: 250 additions & 0 deletions .ai/dashboard-ui.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,250 @@
# Dashboard UI (React/TypeScript)

## Tech Stack

- **React 18** with TypeScript 4.5, built via **Craco** (CRA override)
- **State**: React Context + `useReducer` (no Redux)
- **Routing**: `react-router-dom` v6
- **WebSocket**: `react-use-websocket`
- **Charts**: ECharts (`echarts-for-react`)
- **Tables**: `@tanstack/react-table` + `@tanstack/react-virtual`
- **Graphs**: React Flow (`reactflow`) + dagre layout
- **Styling**: Tailwind CSS 3 with custom CSS variable themes
- **Storybook**: Available for component development (`yarn storybook`)
- **Node**: >= 20 required

## Source Tree

```
ui/dashboard/src/
├── index.tsx Entry point (provider tree)
├── App.tsx Routes + DashboardProvider
├── hooks/ State management & side effects
│ ├── useDashboard.tsx Root provider composition (7 nested providers)
│ ├── useDashboardState.tsx Main reducer (IDashboardContext)
│ ├── useDashboardExecution.tsx WebSocket lifecycle + snapshot loading
│ ├── useDashboardWebSocket.ts WebSocket connection (react-use-websocket)
│ ├── useDashboardWebSocketEventHandler.ts Event buffering (500ms flush)
│ ├── useDashboardInputs.tsx Input/filter state
│ ├── useDashboardSearchPath.tsx Search path state
│ ├── useDashboardDatetimeRange.tsx DateTime range state
│ ├── useDashboardPanelDetail.tsx Side panel state
│ ├── useDashboardSearch.tsx Global search state
│ ├── useTheme.tsx Light/dark theme
│ ├── useBreakpoint.tsx Responsive breakpoints
│ └── useAnalytics.tsx Analytics tracking
├── components/
│ ├── dashboards/ Dashboard-specific components
│ │ ├── charts/ AreaChart, BarChart, ColumnChart, DonutChart, etc.
│ │ ├── flows/ Sankey, Flow
│ │ ├── graphs/ ForceDirectedGraph, Graph
│ │ ├── hierarchies/ Tree, Hierarchy
│ │ ├── inputs/ DateInput, SelectInput, ComboInput, etc.
│ │ ├── layout/ Dashboard, Container, Panel, Grid
│ │ ├── grouping/ Benchmark, CheckPanel, DetectionBenchmark
│ │ ├── Card/, Table/, Text/, Image/, Error/
│ │ └── index.ts Component registry (getComponent/registerComponent)
│ └── DashboardHeader, DashboardList, DashboardSearch, etc.
├── utils/
│ ├── registerComponents.ts Registers all panel types in registry
│ ├── dashboardEventHandlers.ts WebSocket event processing + schema migration
│ ├── state.ts State builders
│ └── data.ts, color.ts, url.ts, snapshot.ts, ...
├── types/ TypeScript type definitions
├── constants/ Schema versions, icon mappings
└── styles/ Tailwind config, CSS themes
```

## Provider Hierarchy (outermost to innermost)

```
BrowserRouter
└─ ThemeProvider (light/dark)
└─ ErrorBoundary
└─ BreakpointProvider (responsive)
└─ AnalyticsProvider
└─ DashboardProvider (composes 7 inner providers):
├─ DashboardThemeProvider
├─ DashboardSearchProvider
├─ DashboardStateProvider ← main reducer
├─ DashboardInputsProvider
├─ DashboardSearchPathProvider
├─ DashboardDatetimeRangeProvider
├─ DashboardPanelDetailProvider
└─ DashboardExecutionProvider ← WebSocket
```

## State Management

`useDashboardState.tsx` defines the main reducer with `IDashboardContext` state type.

Key state fields:
- `dataMode`: `"live"` | `"cli_snapshot"` | `"cloud_snapshot"`
- `state`: `"running"` | `"complete"` | `"error"`
- `panelsMap`: All panel data keyed by name
- `panelsLog`: Execution logs per panel
- `dashboard`: Current dashboard definition
- `selectedDashboard`: Currently selected dashboard
- `dashboards` / `dashboardsMap`: Available dashboards
- `snapshot`: Execution snapshot for completed runs
- `execution_id`: Current execution UUID
- `progress`: Execution progress (0-100)
- `metadata`: Server metadata
- `error`: Current error state

Key dispatch actions (in `DashboardActions` enum):
- `SERVER_METADATA` - Server metadata received
- `DASHBOARD_METADATA` - Dashboard-specific metadata
- `AVAILABLE_DASHBOARDS` - List of runnable dashboards
- `EXECUTION_STARTED` - Execution begins, panel structure received
- `EXECUTION_COMPLETE` - All panels done, snapshot ready
- `CONTROLS_UPDATED` - Batch of control updates
- `LEAF_NODES_COMPLETE` - Batch of completed leaf nodes
- `LEAF_NODES_UPDATED` - Batch of updated leaf nodes

## WebSocket Event Handling

### Event Buffering (`useDashboardWebSocketEventHandler.ts`)

Rapid events from server are buffered to prevent UI thrashing:
- `CONTROL_COMPLETE` / `CONTROL_ERROR` events → buffered in array
- `LEAF_NODE_COMPLETE` events → buffered with timestamp
- `LEAF_NODE_UPDATED` events → buffered
- Buffer flushed every **500ms** via `setInterval`
- Other events (`execution_started`, `execution_complete`) → dispatched immediately

### WebSocket Connection (`useDashboardWebSocket.ts`)

Uses `react-use-websocket` library.

URL resolution:
- **Development**: `ws://localhost:9033/ws`
- **Production**: Derives from `window.location` (http→ws, https→wss)

Reconnection: max 10 attempts, 3000ms interval.

### Socket Actions (client → server)

```typescript
SocketActions = {
CLEAR_DASHBOARD: "clear_dashboard",
GET_AVAILABLE_DASHBOARDS: "get_available_dashboards",
GET_SERVER_METADATA: "get_server_metadata",
SELECT_DASHBOARD: "select_dashboard",
INPUT_CHANGED: "input_changed",
}
```

### Server Events (server → client)

- `available_dashboards` - List of runnable dashboards
- `execution_started` - Dashboard execution began; includes panel metadata and layout
- `leaf_node_updated` - Panel result ready; includes data/status
- `leaf_node_complete` - Panel execution finished
- `execution_complete` - All panels complete; includes full snapshot
- `execution_error` - Runtime error
- `control_complete` / `control_error` - Individual control results
- `workspace_error` - Mod parse/load error

## Routes

```
/ Live dashboard view (WebSocket streaming)
/:dashboard_name Select specific dashboard
/snapshot/:dashboard_name View saved snapshot (read-only, no re-execution)
```

Query params on snapshot route encode: inputs, datetime range, search_path_prefix.

## Theming

Two CSS variable themes defined in `styles/index.css`:
- `.theme-steampipe-default` (light)
- `.theme-steampipe-dark` (dark)

Custom color variables:
- Control colors: `--color-alert`, `--color-ok`, `--color-info`, `--color-skip`, `--color-severity`
- Layout: `--color-dashboard`, `--color-dashboard-panel`
- Text: `--color-foreground`, `--color-foreground-light`, `-lighter`, `-lightest`
- Tables: `--color-table-border`, `--color-table-divide`, `--color-table-head`
- Scale: `--color-black-scale-1` through `--color-black-scale-8`

Tailwind plugins: `@tailwindcss/forms`, `@tailwindcss/typography`.

## Component Registry

Components registered dynamically in `utils/registerComponents.ts` into a map in `components/dashboards/index.ts`:

```typescript
const componentsMap = {};
const getComponent = (key: string) => componentsMap[key];
const registerComponent = (key: string, component) => { componentsMap[key] = component; };
```

The layout renderer looks up components by panel type string (e.g., `"card"`, `"chart"`, `"table"`). This allows extensibility without hardcoded imports.

Registered types: Panel, Container, Dashboard, all chart types (Area, Bar, Column, Donut, Heatmap, Line, Pie), Flow, Sankey, Graph, ForceDirectedGraph, Tree, Hierarchy, all input types, Benchmark, DetectionBenchmark, Table, Text, Image, Card, Error.

## Build & Dev Workflow

```bash
cd ui/dashboard
yarn install
yarn start # Dev server on http://localhost:3000 (proxies to Go backend on :9033)
yarn test # Jest + React Testing Library
yarn storybook # Component playground on http://localhost:6006
yarn build # Production build → build/
```

### Craco Configuration (`craco.config.js`)

- **WebAssembly**: `experiments.asyncWebAssembly` enabled
- **Path alias**: `@powerpipe` → `src/`
- **Node polyfills**: buffer, crypto, path, stream, vm (via `ProvidePlugin`)
- **Circular dependency detection**: `CircularDependencyPlugin` fails build on circular imports in `/src`

### Schema Versions

Event schema versions tracked in `utils/dashboardEventHandlers.ts`:
```
EXECUTION_SCHEMA_VERSION_20220614
EXECUTION_SCHEMA_VERSION_20220929
EXECUTION_SCHEMA_VERSION_20221222
EXECUTION_SCHEMA_VERSION_20240130
EXECUTION_SCHEMA_VERSION_20240607
EXECUTION_SCHEMA_VERSION_20241125
```

Schema migration functions handle backwards compatibility between versions.

## Key Type Definitions (`types/index.ts`)

- `DashboardDataMode`: `"live"` | `"cli_snapshot"` | `"cloud_snapshot"`
- `DashboardRunState`: `"running"` | `"complete"` | `"error"`
- `DashboardPanelType`: `"dashboard"` | `"card"` | `"table"` | `"chart"` | `"input"` | `"graph"` | `"hierarchy"` | `"flow"` | `"benchmark"` | `"control"` | `"detection"` | `"image"` | `"text"` | `"error"` | `"with"` | `"edge"`
- `PanelLog`: `{ error, executionTime, isDependency, prefix, status, timestamp, title }`
- `ReceivedSocketMessagePayload`: `{ action: string, [key: string]: any }`

## Data Flow

```
User Action (click dashboard, change input)
React Component / Event Handler
WebSocket send (react-use-websocket)
Go backend receives, executes query/dashboard
Go backend sends event via WebSocket
useDashboardWebSocket receives message
useDashboardWebSocketEventHandler buffers (500ms)
Dispatch action to useDashboardState reducer
State updated, Context re-renders subscribed components
Component renders with new data (ECharts, ReactFlow, etc.)
```
134 changes: 134 additions & 0 deletions .ai/environment-reference.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
# Environment Variables & Configuration Reference

## Powerpipe Environment Variables

| Variable | Purpose | Default |
|----------|---------|---------|
| `POWERPIPE_LISTEN` | Server listen: `"local"` or `"network"` | `local` |
| `POWERPIPE_PORT` | Server port | `9033` |
| `POWERPIPE_BENCHMARK_TIMEOUT` | Benchmark timeout (seconds) | `300` |
| `POWERPIPE_DASHBOARD_TIMEOUT` | Dashboard query timeout (seconds) | `120` |
| `POWERPIPE_DISPLAY_WIDTH` | CLI output width | Auto-detect |
| `POWERPIPE_INSTALL_DIR` | Installation directory | `~/.powerpipe` |
| `POWERPIPE_MOD_LOCATION` | Mod directory override | Current dir |
| `POWERPIPE_DATABASE` | Default database connection string | - |
| `POWERPIPE_MAX_PARALLEL` | Max parallel DB connections | `10` |
| `POWERPIPE_QUERY_TIMEOUT` | Per-query timeout (seconds) | - |
| `POWERPIPE_SNAPSHOT_LOCATION` | Snapshot upload target | - |
| `POWERPIPE_UPDATE_CHECK` | Enable version check on startup | `true` |
| `POWERPIPE_TELEMETRY` | Telemetry level | `info` |
| `POWERPIPE_CACHE_ENABLED` | Enable query caching | `true` |
| `POWERPIPE_CACHE_TTL` | Cache TTL (seconds) | - |
| `POWERPIPE_CACHE_MAX_TTL` | Cache max TTL (seconds) | - |
| `POWERPIPE_MEMORY_MAX_MB` | Go memory limit (MB) | `1024` |
| `POWERPIPE_MEMORY_MAX_MB_PLUGIN` | Plugin memory limit (MB) | - |
| `POWERPIPE_LOG_LEVEL` | Log level (see below) | `off` |
| `POWERPIPE_CONFIG_PATH` | Colon-separated config search paths | - |
| `POWERPIPE_CONFIG_DUMP` | Debug: dump config as JSON (undocumented) | - |

### Pipes Cloud Variables

| Variable | Purpose |
|----------|---------|
| `PIPES_HOST` | Pipes cloud host |
| `PIPES_TOKEN` | Pipes authentication token |
| `PIPES_INSTALL_DIR` | Pipes installation directory |

### Input Variables

Set HCL variables via `PP_VAR_` prefix:
```bash
PP_VAR_region=us-east-1
PP_VAR_max_age=30
```

### Deprecated Variables

| Variable | Replacement |
|----------|-------------|
| `STEAMPIPE_DIAGNOSTICS_LEVEL` | `PIPES_DIAGNOSTICS_LEVEL` |

## Configuration Precedence (highest to lowest)

1. CLI flags (`--flag`)
2. Environment variables (`POWERPIPE_*`)
3. Explicit workspace profile (`--workspace-profile`)
4. Default workspace profile
5. Config files (`.ppc`)
6. Viper defaults (set in `internal/cmdconfig/mappings.go`)

## Log Levels

Set via `POWERPIPE_LOG_LEVEL` (actual env var name from `app_specific.EnvLogLevel`).

| Value | slog Level | Notes |
|-------|------------|-------|
| `trace` | Custom trace | Most verbose |
| `debug` | `slog.LevelDebug` | |
| `info` | `slog.LevelInfo` | |
| `warn` | `slog.LevelWarn` | |
| `error` | `slog.LevelError` | |
| `off` | Discard | **Default** - no logging |

Logger implementation: Go `log/slog` with JSON handler to stderr. Sensitive values automatically redacted via `sanitize.Instance.SanitizeKeyValue()`.

Performance tracing: `utils.LogTime("label")` logs timestamped markers.

## File Extensions

| Extension | Purpose |
|-----------|---------|
| `.pp` | Mod and resource files (primary) |
| `.sp` | Legacy Steampipe format (still supported) |
| `.ppvars` | Variable files |
| `.spvars` | Legacy variable files |
| `.auto.ppvars` | Auto-loaded variable files |
| `.ppc` | Config files |
| `.powerpipeignore` | Workspace ignore patterns |
| `.mod.cache.json` | Dependency lock file |

## Database Connection String Formats

| Format | Backend |
|--------|---------|
| `steampipe://profile/schema` | Steampipe (PostgreSQL) |
| `postgres://user:pass@host/db` | PostgreSQL |
| `mysql://user:pass@host/db` | MySQL |
| `duckdb:///path/to/db.duckdb` | DuckDB (file) |
| `duckdb://` | DuckDB (in-memory) |
| `sqlite:///path/to/db.sqlite` | SQLite |

## Exit Codes

| Code | Constant | Meaning |
|------|----------|---------|
| 0 | `ExitCodeSuccessful` | Success |
| 1 | `ExitCodeControlsAlarm` | Check/benchmark: alarms found, no errors |
| 2 | `ExitCodeControlsError` | Check/benchmark: control errors found |
| 21 | `ExitCodeSnapshotCreationFailed` | Snapshot creation failed |
| 22 | `ExitCodeSnapshotUploadFailed` | Snapshot upload failed |
| 41 | `ExitCodeQueryExecutionFailed` | Query execution failed |
| 62 | `ExitCodeModInstallFailed` | Mod installation failed |
| 250 | `ExitCodeInitializationFailed` | Workspace/DB initialization failed |
| 251 | `ExitCodeBindPortUnavailable` | Server port binding failed |
| 252 | `ExitCodeNoModFile` | No mod.pp found |
| 254 | `ExitCodeInsufficientOrWrongInputs` | Invalid user input |
| 255 | `ExitCodeUnknownErrorPanic` | Panic recovery (unhandled crash) |

## Error Handling Patterns

- **Panic recovery**: `main.go` has a deferred `recover()` that catches panics, logs error via `error_helpers.ShowError()`, and exits with code 255
- **InitData errors**: Commands call `NewInitData[T]()`, then check `initData.Result.HasError()`. Errors from workspace loading, dependency install, and DB client creation are collected in `Result.ErrorAndWarnings`
- **Error display**: `error_helpers.ShowError(ctx, err)` and `error_helpers.FailOnError()` from pipe-fittings
- **Goroutine errors**: In `dashboardexecute`, child errors propagate via channels to parent nodes. No panic/recover in execution goroutines

## Build Variables (injected via ldflags)

| Variable | Dev Value | Release Value |
|----------|-----------|---------------|
| `main.version` | `0.0.0-dev-{branch}.{timestamp}` | Semver from git tag |
| `main.commit` | `none` | Git commit hash |
| `main.date` | `unknown` | Build timestamp |
| `main.builtBy` | `local` | `goreleaser` |

Dev mode detection: `cmdconfig.IsLocal()` returns `true` when `builtBy == "local"`.
Loading
Loading