Skip to content

feat: fast and full autocomplete#2128

Open
knudtty wants to merge 7 commits intomainfrom
aaron/feat-autocomplete
Open

feat: fast and full autocomplete#2128
knudtty wants to merge 7 commits intomainfrom
aaron/feat-autocomplete

Conversation

@knudtty
Copy link
Copy Markdown
Contributor

@knudtty knudtty commented Apr 16, 2026

Summary

  • Adds metadataMaterializedViews config to log and trace sources, storing key rollup and KV rollup table references directly on the source instead of hardcoding them by table name.
  • Uses rollups by default in autocomplete field and key fetching methods, allowing for much better autocomplete
  • tcFromSource() populates metadataMVs on TableConnection from the source's metadataMaterializedViews, so all downstream consumers (field discovery, autocomplete, filters) pick it up automatically.
  • Source inference (inferTableSourceConfig) auto-detects rollup tables by checking if {tableName}_key_rollup_15m and {tableName}_kv_rollup_15m exist in ClickHouse before adding the config.
  • Adds a MetadataMaterializedViewsFormSection UI to the log and trace source forms for manual configuration.
  • Updates all default source configs (.env.development, docker-compose.yml, entry.local.base.sh) to include metadataMaterializedViews for the standard OTel log and trace sources.

Screenshots or video

export-1776301077703.mp4

How to test locally or on Vercel

  1. Run this locally. Clear your existing volumes to start from scratch.
  2. Fire it up via yarn dev
  3. Verify logs and traces have the key and kv rollups as part of the sources
  4. Try out autocomplete on either the logs our traces sources

References

Closes HDX-3931

@vercel
Copy link
Copy Markdown

vercel bot commented Apr 16, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hyperdx-oss Ready Ready Preview, Comment Apr 17, 2026 8:53pm

Request Review

@knudtty knudtty requested review from a team and pulpdrew and removed request for a team April 16, 2026 01:36
@github-actions github-actions bot added the review/tier-4 Critical — deep review + domain expert sign-off label Apr 16, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 16, 2026

🔴 Tier 4 — Critical

Touches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD.

Why this tier:

  • Critical-path files (4):
    • docker/hyperdx/entry.local.base.sh
    • docker/otel-collector/schema/seed/00006_otel_logs_rollups.sql
    • docker/otel-collector/schema/seed/00007_otel_traces_rollups.sql
    • packages/api/src/routers/external-api/v2/sources.ts
  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 23
  • Production lines changed: 1422 (+ 442 in test files, excluded from tier calculation)
  • Branch: aaron/feat-autocomplete
  • Author: knudtty

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@changeset-bot
Copy link
Copy Markdown

changeset-bot bot commented Apr 16, 2026

🦋 Changeset detected

Latest commit: 41fe0b8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Minor
@hyperdx/api Minor
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@knudtty knudtty changed the title Aaron/feat autocomplete feat: fast and full autocomplete Apr 16, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 16, 2026

PR Review

  • Value autocomplete is disabled for sources without metadataMVs → In useAllKeyValues, enabled: !!rollupParams blocks all queries when tableConnection.metadataMVs is undefined (since fieldToRollupParams returns null without it). The fallback paths inside queryFn (metadata.getMapValues) are dead code — the query never fires. This is a regression: users on sources without rollup tables lose value autocomplete entirely. Fix: split into two hooks or set enabled to !!searchField and let the queryFn handle the routing between rollup and fallback.

  • ⚠️ Stale cursor position in tokenInfouseAutoCompleteOptions reads inputRef?.current?.selectionStart inside a useMemo (flagged with a non-existent react-hooks/refs eslint-disable). The useMemo only re-runs when value or inputRef changes, so cursor position is stale whenever the user moves their cursor without typing. This causes wrong-token autocomplete suggestions. The eslint-disable comment suppresses nothing real. Consider passing cursorPos as explicit state (from an onSelect/onKeyUp handler) rather than reading the ref inside useMemo.

  • ⚠️ inferTableSourceConfig fires 2 extra ClickHouse queries for every OTel table → The IIFE doing Promise.all([getTableMetadata(..._key_rollup_15m...), getTableMetadata(..._kv_rollup_15m...)]) runs for every OTel log/trace schema check, even when rollup tables don't exist. For users without rollup tables, these queries always fail (silently). Consider checking only when an explicit flag or known table list indicates rollups are present, or at least collapse to a single existence check.

  • ⚠️ Overly broad object patch in inferConfig effect → The new guard if (value && typeof value === 'object' && !Array.isArray(value)) { setValue(...); return; } applies to ALL object-valued fields (e.g., from, metricTables), bypassing resetField. This could silently prevent dirty-state tracking or validation for other nested objects. Scope it specifically to metadataMaterializedViews by key name.

  • ⚠️ useAllKeyValues only fetches from tcs[0] → When multiple table connections are active, only the first is used for key-value autocomplete. This is intentional per the code, but worth a comment since it silently ignores values from other connected tables.

@knudtty knudtty removed the request for review from pulpdrew April 16, 2026 01:40
@knudtty knudtty marked this pull request as draft April 16, 2026 01:41
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 16, 2026

E2E Test Results

All tests passed • 146 passed • 3 skipped • 1077s

Status Count
✅ Passed 146
❌ Failed 0
⚠️ Flaky 4
⏭️ Skipped 3

Tests ran across 4 shards in parallel.

View full report →

@knudtty knudtty marked this pull request as ready for review April 17, 2026 18:11
@knudtty knudtty requested review from a team and bot-hyperdx and removed request for a team and bot-hyperdx April 17, 2026 20:52
@knudtty knudtty requested review from dhable and removed request for a team April 17, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant