feat: fast and full autocomplete

[knudtty]

Summary

• Adds metadataMaterializedViews config to log and trace sources, storing key rollup and KV rollup table references directly on the source instead of hardcoding them by table name.
• Uses rollups by default in autocomplete field and key fetching methods, allowing for much better autocomplete
• tcFromSource() populates metadataMVs on TableConnection from the source's metadataMaterializedViews, so all downstream consumers (field discovery, autocomplete, filters) pick it up automatically.
• Source inference (inferTableSourceConfig) auto-detects rollup tables by checking if {tableName}_key_rollup_15m and {tableName}_kv_rollup_15m exist in ClickHouse before adding the config.
• Adds a MetadataMaterializedViewsFormSection UI to the log and trace source forms for manual configuration.
• Updates all default source configs (.env.development, docker-compose.yml, entry.local.base.sh) to include metadataMaterializedViews for the standard OTel log and trace sources.

Screenshots or video

export-1776301077703.mp4

How to test locally or on Vercel

1. Run this locally. Clear your existing volumes to start from scratch.
2. Fire it up via yarn dev
3. Verify logs and traces have the key and kv rollups as part of the sources
4. Try out autocomplete on either the logs our traces sources

References

Closes HDX-3931

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Apr 17, 2026 8:53pm

[github-actions]

🔴 Tier 4 — Critical

Touches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD.

Why this tier:

• Critical-path files (4):
  • docker/hyperdx/entry.local.base.sh
  • docker/otel-collector/schema/seed/00006_otel_logs_rollups.sql
  • docker/otel-collector/schema/seed/00007_otel_traces_rollups.sql
  • packages/api/src/routers/external-api/v2/sources.ts
• Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

• Production files changed: 23
• Production lines changed: 1422 (+ 442 in test files, excluded from tier calculation)
• Branch: aaron/feat-autocomplete
• Author: knudtty

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[changeset-bot]

🦋 Changeset detected

Latest commit: 41fe0b8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/common-utils	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor
@hyperdx/otel-collector	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

PR Review

• ❌ Value autocomplete is disabled for sources without metadataMVs → In useAllKeyValues, enabled: !!rollupParams blocks all queries when tableConnection.metadataMVs is undefined (since fieldToRollupParams returns null without it). The fallback paths inside queryFn (metadata.getMapValues) are dead code — the query never fires. This is a regression: users on sources without rollup tables lose value autocomplete entirely. Fix: split into two hooks or set enabled to !!searchField and let the queryFn handle the routing between rollup and fallback.

• ⚠️ Stale cursor position in tokenInfo → useAutoCompleteOptions reads inputRef?.current?.selectionStart inside a useMemo (flagged with a non-existent react-hooks/refs eslint-disable). The useMemo only re-runs when value or inputRef changes, so cursor position is stale whenever the user moves their cursor without typing. This causes wrong-token autocomplete suggestions. The eslint-disable comment suppresses nothing real. Consider passing cursorPos as explicit state (from an onSelect/onKeyUp handler) rather than reading the ref inside useMemo.

• ⚠️ inferTableSourceConfig fires 2 extra ClickHouse queries for every OTel table → The IIFE doing Promise.all([getTableMetadata(..._key_rollup_15m...), getTableMetadata(..._kv_rollup_15m...)]) runs for every OTel log/trace schema check, even when rollup tables don't exist. For users without rollup tables, these queries always fail (silently). Consider checking only when an explicit flag or known table list indicates rollups are present, or at least collapse to a single existence check.

• ⚠️ Overly broad object patch in inferConfig effect → The new guard if (value && typeof value === 'object' && !Array.isArray(value)) { setValue(...); return; } applies to ALL object-valued fields (e.g., from, metricTables), bypassing resetField. This could silently prevent dirty-state tracking or validation for other nested objects. Scope it specifically to metadataMaterializedViews by key name.

• ⚠️ useAllKeyValues only fetches from tcs[0] → When multiple table connections are active, only the first is used for key-value autocomplete. This is intentional per the code, but worth a comment since it silently ignores values from other connected tables.

[github-actions]

E2E Test Results

✅ All tests passed • 146 passed • 3 skipped • 1077s

Status	Count
✅ Passed	146
❌ Failed	0
⚠️ Flaky	4
⏭️ Skipped	3

Tests ran across 4 shards in parallel.

View full report →