chore: split json otel collector to enable both during dev #1247

knudtty · 2025-10-07T21:19:45Z

Gets us closer to a staging instance of json

changeset-bot · 2025-10-07T21:19:49Z

⚠️ No Changeset found

Latest commit: c74aea8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

vercel · 2025-10-07T21:19:53Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
hyperdx-v2-oss-app	Ready	Preview	Comment	Nov 4, 2025 9:17pm

github-actions · 2025-10-07T21:24:59Z

E2E Test Results

✅ All tests passed • 39 passed • 3 skipped • 296s

Status	Count
✅ Passed	39
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	3

View full report →

claude · 2025-10-09T15:58:17Z

Pull Request Review: Split JSON OTel Collector for Dev Environment

This PR introduces a second OTel collector instance (otel-collector-json) to run alongside the standard collector during development.

Strengths

Clear separation of concerns using feature flags
Backward compatible with existing workflow
Well-structured conditional logic

Critical Issues

1. Server Startup Logic Bug (HIGH PRIORITY)
Location: packages/api/src/server.ts:49
The condition !config.IS_DEV || !config.IS_OPAMP_ONLY will start the app server in production even when IS_OPAMP_ONLY=true. Should be !(config.IS_DEV && config.IS_OPAMP_ONLY).

2. Missing Tests (HIGH PRIORITY)
No unit tests for new config flags (IS_OPAMP_ONLY, IS_JSON_OPAMP), server startup logic combinations, or OpAMP controller conditional config.

3. Missing Documentation (MEDIUM PRIORITY)
New environment variables lack JSDoc comments explaining their purpose and usage.

Other Issues

OPAMP_PORT Validation (LOW)
Location: packages/api/src/config.ts:32
If OPAMP_PORT is not set, parseInt returns NaN. Add validation or default value.

Redundant Conditional (LOW)
Location: packages/api/src/opamp/controllers/opampController.ts:263
The spread ...(otlpExporter ? otlpExporter : {}) is redundant since otlpExporter is always defined.

Hardcoded Port (MEDIUM)
Location: docker-compose.dev.yml:65
JSON OpAMP URL uses port 4321 but default is 4320. Add comment explaining why.

API Key Handling (INFO)
Location: packages/api/src/opamp/controllers/opampController.ts:160
Consider logging warning when no API key available in dev mode.

Performance Note
Running two collectors doubles resource usage. Document when to use JSON vs standard collector.

Concurrency Colors (LOW)
package.json has 4 colors but 5 processes. Add 5th color.

Recommendations

Must Fix:

Fix boolean logic in server.ts:49
Add unit tests for new functionality
Document new environment variables

Should Fix:
4. Add OPAMP_PORT validation
5. Document resource implications
6. Add integration tests

Nice to Have:
7. Clean up redundant conditionals
8. Improve naming consistency
9. Fix concurrency script colors

Overall

Well-structured PR that achieves its goal. Main blockers are the logic bug and missing tests. Once addressed, ready to merge!

Reviewed per CLAUDE.md conventions

knudtty · 2025-10-15T20:23:18Z

@claude

claude · 2025-10-15T20:23:32Z

PR Review

Summary

This PR adds a second OTel collector instance (otel-collector-json) to enable testing the JSON schema feature in development alongside the standard collector.

Issues Found

⚠️ Hardcoded API key in config file (docker/otel-collector/custom.config.yaml:23) → While this matches .env.development, hardcoded credentials in config files are a security risk. Consider using environment variable substitution (e.g., ${HYPERDX_API_KEY}) to avoid accidental credential leaks if this file is ever used outside dev.
⚠️ Uncommented OTEL_SUPERVISOR_LOGS in main collector (docker-compose.dev.yml:36) → This enables verbose logging for the primary collector. Confirm this is intentional for dev, or should remain commented by default.
ℹ️ Missing sessionSourceId in JSON sources (packages/api/.env.development) → The new JSON Logs and JSON Traces sources do not reference a sessionSourceId like the standard sources do. Verify if this is intentional or if sessions should be excluded from JSON testing.

Minor Notes

Port conflict risk: Ensure port 14318 does not collide with other local services
The nop exporter addition in opampController.ts:70 correctly supports the new dev.json.config.yaml pipeline

Recommendation: Address the hardcoded credential issue before merge. Other items are minor and can be addressed if needed.

wrn14897

I think adding OTEL_AGENT_FEATURE_GATE_ARG should be sufficient. We can let developers decide whether to switch over and test the JSON columns to minimize code changes. If we are going to test JSON features more, imo we should just enable it by default

dhable

Two comments about the code and a nit pick.

packages/api/src/server.ts

packages/api/src/opamp/controllers/opampController.ts

dhable · 2025-10-27T15:02:51Z

docker-compose.dev.yml

      ch-server:
        condition: service_healthy
+  otel-collector-json:
+    # image: otel/opentelemetry-collector-contrib:0.120.0


nit: remove the commented line

packages/api/src/opamp/controllers/opampController.ts

Pushed changes to use otel custom config instead

wrn14897 · 2025-11-04T02:35:05Z

reimplemented the data forking using otel config override (b2f7e48)

wrn14897 · 2025-11-04T02:44:29Z

docker-compose.dev.yml

-      # Uncomment to enable JSON schema in ClickHouse
-      # Be sure to also set BETA_CH_OTEL_JSON_SCHEMA_ENABLED to 'true' in ch-server
-      # OTEL_AGENT_FEATURE_GATE_ARG: '--feature-gates=clickhouse.json'
+      OTEL_SUPERVISOR_LOGS: 'true'


IMO, we should enable supervisor logs (easier debugging)

wrn14897 · 2025-11-04T02:45:03Z

docker-compose.dev.yml

+    environment:
+      CLICKHOUSE_ENDPOINT: 'tcp://ch-server:9000?dial_timeout=10s'
+      CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT: 'ch-server:9363'
+      HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE: 'otel_json'


hardcode the target database

wrn14897 · 2025-11-04T02:46:57Z

packages/api/src/opamp/controllers/opampController.ts

      },
    },
    exporters: {
+      nop: null,


used for disabling rrweb export (json-otel-collector)

wrn14897 · 2025-11-04T02:50:03Z

docker/otel-collector/custom.config.yaml

+  otlphttp/json:
+    endpoint: http://otel-collector-json:4318
+    headers:
+      authorization: "super-secure-ingestion-api-key"


hyperdx/packages/api/.env.development

Line 26 in b2f7e48

INGESTION_API_KEY="super-secure-ingestion-api-key"

wrn14897

It works on my local environment, but I’d appreciate another pair of eyes!

vercel bot deployed to Preview October 7, 2025 21:23 View deployment

vercel bot deployed to Preview October 9, 2025 16:01 View deployment

knudtty requested review from dhable and wrn14897 October 15, 2025 18:09

vercel bot deployed to Preview October 15, 2025 20:06 View deployment

vercel bot deployed to Preview October 15, 2025 20:25 View deployment

wrn14897 previously requested changes Oct 23, 2025

View reviewed changes

dhable reviewed Oct 27, 2025

View reviewed changes

wrn14897 reviewed Oct 27, 2025

View reviewed changes

packages/api/src/opamp/controllers/opampController.ts Show resolved Hide resolved

wrn14897 reviewed Oct 27, 2025

View reviewed changes

packages/api/src/opamp/controllers/opampController.ts Outdated Show resolved Hide resolved

wrn14897 force-pushed the aaron/json-dev-split branch from f6dedd2 to 7eea492 Compare November 4, 2025 02:31

vercel bot deployed to Preview November 4, 2025 02:36 View deployment

wrn14897 force-pushed the aaron/json-dev-split branch from 7eea492 to d2d2dfa Compare November 4, 2025 02:39

vercel bot deployed to Preview November 4, 2025 02:41 View deployment

feat: fork in-app telemetry to JSON tables

b2f7e48

wrn14897 force-pushed the aaron/json-dev-split branch from d2d2dfa to b2f7e48 Compare November 4, 2025 02:43

wrn14897 reviewed Nov 4, 2025

View reviewed changes

vercel bot deployed to Preview November 4, 2025 02:46 View deployment

wrn14897 reviewed Nov 4, 2025

View reviewed changes

wrn14897 approved these changes Nov 4, 2025

View reviewed changes

wrn14897 added the automerge label Nov 4, 2025

Merge branch 'main' into aaron/json-dev-split

c74aea8

kodiakhq bot merged commit 19c5085 into main Nov 4, 2025
8 of 9 checks passed

kodiakhq bot deleted the aaron/json-dev-split branch November 4, 2025 21:16

vercel bot deployed to Preview November 4, 2025 21:17 View deployment

chore: split json otel collector to enable both during dev #1247

chore: split json otel collector to enable both during dev #1247

Uh oh!

Conversation

knudtty commented Oct 7, 2025 • edited by wrn14897 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

vercel bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Results

Uh oh!

claude bot commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Review: Split JSON OTel Collector for Dev Environment

Strengths

Critical Issues

Other Issues

Recommendations

Overall

Uh oh!

knudtty commented Oct 15, 2025

Uh oh!

claude bot commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review

Summary

Issues Found

Minor Notes

Uh oh!

wrn14897 left a comment

Choose a reason for hiding this comment

Uh oh!

dhable left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dhable Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wrn14897 commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wrn14897 Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

wrn14897 Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

wrn14897 Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

wrn14897 Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

wrn14897 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

knudtty commented Oct 7, 2025 •

edited by wrn14897

Loading

changeset-bot bot commented Oct 7, 2025 •

edited

Loading

vercel bot commented Oct 7, 2025 •

edited

Loading

github-actions bot commented Oct 7, 2025 •

edited

Loading

claude bot commented Oct 9, 2025 •

edited

Loading

claude bot commented Oct 15, 2025 •

edited

Loading

wrn14897 commented Nov 4, 2025 •

edited

Loading