feat: optimized default otel-logs schema by knudtty · Pull Request #2125 · hyperdxio/hyperdx

knudtty · 2026-04-15T20:17:13Z

Summary

After weeks of benchmarking, this optimized logs schema proves to beat the existing schema in almost all queries. The primary key is now prepended with toStartOfFiveMinutes, which groups rows together well enough that TimestampTime is not needed. Additionally, full text search is added for all indexes as it shows superior performance with minimal ingest overhead.

How to test locally

If you have a .volumes directory locally, (re)move it
yarn dev
Poke around and make sure everything is working. Check the schema to ensure its the new one

References

Linear Issue: Closes HDX-4034

vercel · 2026-04-15T20:17:18Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Apr 17, 2026 6:18pm

changeset-bot · 2026-04-15T20:17:21Z

🦋 Changeset detected

Latest commit: 2ddf9fd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/otel-collector	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-04-15T20:17:26Z

🔴 Tier 4 — Critical

Touches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD.

Why this tier:

Critical-path files (7):
- docker/clickhouse/local/init-db-e2e.sh
- docker/hyperdx/Dockerfile
- docker/hyperdx/entry.local.base.sh
- docker/otel-collector/schema/seed/00002_otel_logs.sql
- docker/otel-collector/schema/seed/00002_otel_logs_compat.sql
- packages/otel-collector/cmd/migrate/main.go
- packages/otel-collector/cmd/migrate/main_test.go
Cross-layer change: touches frontend (packages/app) + backend (packages/api)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

Production files changed: 25
Production lines changed: 376 (+ 16 in test files, excluded from tier calculation)
Branch: aaron/new-otel-schema
Author: knudtty

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

github-actions · 2026-04-15T20:20:26Z

PR Review

⚠️ No upgrade migration for existing installations → CREATE TABLE IF NOT EXISTS silently skips schema changes for users with existing otel_logs tables. Upgrading users keep the old schema (with TimestampTime, bloom filters, old ORDER BY) and get no benefit. Consider documenting that a fresh volume is required, or add an ALTER TABLE migration path.
⚠️ hyperdx_sessions inconsistency across DEFAULT_SOURCES → In entry.local.base.sh, the main DEFAULT_SOURCES keeps "timestampValueExpression":"TimestampTime" for sessions, but the BETA_CH_OTEL_JSON_SCHEMA_ENABLED branch uses "timestampValueExpression":"Timestamp" for sessions. If hyperdx_sessions still has TimestampTime this is fine, but the divergence is unexplained and could confuse the next editor.
⚠️ text(tokenizer = 'array') on TraceId string column → The array tokenizer is designed for array-type columns (like mapKeys()/mapValues() results). Using it on a plain String like TraceId may work for exact-match (the whole string becomes one token), but this is non-obvious and the behavior isn't the same as the bloom_filter(0.001) it replaces. A brief comment explaining the benchmarked rationale would help avoid future "why array?" questions.
✅ Version detection + compat schema fallback (getClickHouseVersion / swapLogsSchemaForCompat) is well-structured and fully unit-tested.
✅ TTL fix (toDateTime(Timestamp)) is correct — DateTime64 can't be used directly in TTL expressions.

github-actions · 2026-04-15T20:26:01Z

E2E Test Results

✅ All tests passed • 146 passed • 3 skipped • 1121s

Status	Count
✅ Passed	146
❌ Failed	0
⚠️ Flaky	4
⏭️ Skipped	3

Tests ran across 4 shards in parallel.

View full report →

knudtty · 2026-04-15T21:31:08Z

Edit: This is completed

I still need to add some logic that only adds FTS if the CH version is > 26.2. Will require some modification to the migration logic

26.2

wrn14897 · 2026-04-17T22:13:41Z

+  INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE text(tokenizer = 'array'),
+  INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE text(tokenizer = 'array'),
+  INDEX idx_log_attr_value mapValues(LogAttributes) TYPE text(tokenizer = 'array'),
+  INDEX idx_lower_body lower(Body) TYPE text(tokenizer = 'splitByNonAlpha')


What about adding FTS on all fields like

ADD INDEX fulltext_lower_v1_idx concatWithSeparator(';', TraceId, SpanId, TraceFlags, SeverityText, SeverityNumber, ServiceName, Body, ResourceSchemaUrl, ResourceAttributes, ScopeSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, LogAttributes) TYPE text(tokenizer = 'splitByNonAlpha', preprocessor = lower(concatWithSeparator(';', TraceId, SpanId, TraceFlags, SeverityText, SeverityNumber, ServiceName, Body, ResourceSchemaUrl, ResourceAttributes, ScopeSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, LogAttributes))) GRANULARITY 100000000

Add change the implicit search field to concatWithSeparator(';', TraceId, SpanId, TraceFlags, SeverityText, SeverityNumber, ServiceName, Body, ResourceSchemaUrl, ResourceAttributes, ScopeSchemaUrl, ScopeName, ScopeVersion, ScopeAttributes, LogAttributes). This is going to massively improve DX since developers won’t always need to look up the attributes. Any thoughts?

wrn14897 · 2026-04-17T22:21:50Z

@@ -1 +1 @@
-SELECT SeverityText FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'normalize-severity' AND ResourceAttributes['test-id'] = 'text-case' ORDER BY TimestampTime FORMAT CSV
+SELECT SeverityText FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'normalize-severity' AND ResourceAttributes['test-id'] = 'text-case' ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) FORMAT CSV


For smoke tests, I’d suggest adding another ClickHouse instance with an older version (< 26.2) and ensuring the schema and SELECT queries still work correctly.

wrn14897 · 2026-04-17T22:28:06Z

 ENABLE_SWAGGER=true
 DEFAULT_CONNECTIONS=[{"name":"Local ClickHouse","host":"http://localhost:${HDX_DEV_CH_HTTP_PORT}","username":"default","password":""}]
-DEFAULT_SOURCES=[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"TimestampTime","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"Traces","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","_id":"682586a8b1f81924e628e808","id":"682586a8b1f81924e628e808"},"connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","sessionSourceId":"Sessions"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","metricSourceId":"Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"JSON Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"JSON Traces","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"JSON Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"JSON Logs","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"JSON Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum"},"connection":"Local ClickHouse","logSourceId":"JSON Logs","traceSourceId":"JSON Traces"}]
+DEFAULT_SOURCES=[{"from":{"databaseName":"default","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"Traces","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"Logs","sessionSourceId":"Sessions","metricSourceId":"Metrics"},{"from":{"databaseName":"default","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum","_id":"682586a8b1f81924e628e808","id":"682586a8b1f81924e628e808"},"connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","sessionSourceId":"Sessions"},{"from":{"databaseName":"default","tableName":"hyperdx_sessions"},"kind":"session","timestampValueExpression":"TimestampTime","name":"Sessions","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","logSourceId":"Logs","traceSourceId":"Traces","metricSourceId":"Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_logs"},"kind":"log","timestampValueExpression":"Timestamp","name":"JSON Logs","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"Body","serviceNameExpression":"ServiceName","bodyExpression":"Body","eventAttributesExpression":"LogAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,SeverityText,Body","severityTextExpression":"SeverityText","traceIdExpression":"TraceId","spanIdExpression":"SpanId","connection":"Local ClickHouse","traceSourceId":"JSON Traces","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":"otel_traces"},"kind":"trace","timestampValueExpression":"Timestamp","name":"JSON Traces","displayedTimestampValueExpression":"Timestamp","implicitColumnExpression":"SpanName","serviceNameExpression":"ServiceName","eventAttributesExpression":"SpanAttributes","resourceAttributesExpression":"ResourceAttributes","defaultTableSelectExpression":"Timestamp,ServiceName,StatusCode,round(Duration/1e6),SpanName","traceIdExpression":"TraceId","spanIdExpression":"SpanId","durationExpression":"Duration","durationPrecision":9,"parentSpanIdExpression":"ParentSpanId","spanNameExpression":"SpanName","spanKindExpression":"SpanKind","statusCodeExpression":"StatusCode","statusMessageExpression":"StatusMessage","connection":"Local ClickHouse","logSourceId":"JSON Logs","metricSourceId":"JSON Metrics"},{"from":{"databaseName":"otel_json","tableName":""},"kind":"metric","timestampValueExpression":"TimeUnix","name":"JSON Metrics","resourceAttributesExpression":"ResourceAttributes","metricTables":{"gauge":"otel_metrics_gauge","histogram":"otel_metrics_histogram","sum":"otel_metrics_sum"},"connection":"Local ClickHouse","logSourceId":"JSON Logs","traceSourceId":"JSON Traces"}]


A reminder that we also need to update helm-chart before the next release https://github.com/ClickHouse/ClickStack-helm-charts/blob/a7862878b89aa30d459fbda602087cb76defd33a/charts/clickstack/values.yaml#L103

vinzee · 2026-04-17T23:26:04Z

-  INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
-  INDEX idx_lower_body lower(Body) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
+  INDEX idx_trace_id TraceId TYPE text(tokenizer = 'array'),
+  INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE text(tokenizer = 'array'),


given that the direct read optimization does not kick in for queries like (Map['key'] IN ('val')), is there a general benefit of making text indices the default for Maps? context: #2134

would be nice to see a performance comparison here. (specially around storage and read-latency)

vinzee · 2026-04-17T23:37:48Z

 CREATE TABLE IF NOT EXISTS ${DATABASE}.e2e_otel_logs
 (
  \`Timestamp\` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
-  \`TimestampTime\` DateTime DEFAULT toDateTime(Timestamp),


wondering the rationale behind removing this? as I understand, toDateTime gives second granularity and was deemed to be more performant for partitioning/ordering. did that change?

knudtty added 2 commits April 15, 2026 14:55

feat: optimized default otel-logs schema

93c530d

fix: upgrade clickhouse version; update DEFAULT_SOURCES

32fdb05

github-actions bot added the review/tier-4 Critical — deep review + domain expert sign-off label Apr 15, 2026

vercel bot deployed to Preview April 15, 2026 20:17 View deployment

fix: remove TimestampTime from order by in smoke tests

db5a56d

vercel bot deployed to Preview April 15, 2026 20:33 View deployment

fix: e2e tests

1258457

vercel bot deployed to Preview April 15, 2026 21:05 View deployment

knudtty requested review from a team, bot-hyperdx and dhable and removed request for a team and bot-hyperdx April 16, 2026 01:16

feat: only use full text index if clickhouse version is greater than

9ad1d75

26.2

vercel bot deployed to Preview April 16, 2026 15:30 View deployment

chore: add testing for compatibility schema

508f369

vercel bot deployed to Preview April 16, 2026 16:31 View deployment

knudtty force-pushed the aaron/new-otel-schema branch from d845174 to 8177a12 Compare April 17, 2026 18:11

vercel bot deployed to Preview April 17, 2026 18:14 View deployment

feat: add EventName column (from upstream otel-collector-contrib)

2ddf9fd

knudtty force-pushed the aaron/new-otel-schema branch from 8177a12 to 2ddf9fd Compare April 17, 2026 18:15

vercel bot deployed to Preview April 17, 2026 18:18 View deployment

wrn14897 reviewed Apr 17, 2026

View reviewed changes

vinzee reviewed Apr 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: optimized default otel-logs schema#2125

feat: optimized default otel-logs schema#2125
knudtty wants to merge 7 commits intomainfrom
aaron/new-otel-schema

knudtty commented Apr 15, 2026

Uh oh!

vercel bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

changeset-bot bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 15, 2026 •

edited

Loading

Uh oh!

knudtty commented Apr 15, 2026 •

edited

Loading

Uh oh!

wrn14897 Apr 17, 2026

Uh oh!

wrn14897 Apr 17, 2026

Uh oh!

wrn14897 Apr 17, 2026

Uh oh!

vinzee Apr 17, 2026 •

edited

Loading

Uh oh!

vinzee Apr 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -1 +1 @@
		SELECT SeverityText FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'normalize-severity' AND ResourceAttributes['test-id'] = 'text-case' ORDER BY TimestampTime FORMAT CSV
		SELECT SeverityText FROM otel_logs WHERE ResourceAttributes['suite-id'] = 'normalize-severity' AND ResourceAttributes['test-id'] = 'text-case' ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) FORMAT CSV

Conversation

knudtty commented Apr 15, 2026

Summary

How to test locally

References

Uh oh!

vercel bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Tier 4 — Critical

Uh oh!

github-actions bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review

Uh oh!

github-actions bot commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Results

Uh oh!

knudtty commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wrn14897 Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

wrn14897 Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

wrn14897 Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

vinzee Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinzee Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vercel bot commented Apr 15, 2026 •

edited

Loading

changeset-bot bot commented Apr 15, 2026 •

edited

Loading

github-actions bot commented Apr 15, 2026 •

edited

Loading

github-actions bot commented Apr 15, 2026 •

edited

Loading

github-actions bot commented Apr 15, 2026 •

edited

Loading

knudtty commented Apr 15, 2026 •

edited

Loading

vinzee Apr 17, 2026 •

edited

Loading