test: add OTEL-to-ClickHouse ingestion assertions to integration smoke test#200
Merged
test: add OTEL-to-ClickHouse ingestion assertions to integration smoke test#200
Conversation
Add synthetic OTLP trace/log submissions and ClickHouse row-count assertions so integration smoke tests fail when collector ingestion is not persisted. Increase smoke test timeout in workflow to allow propagation time. Made-with: Cursor
|
Avoid direct OTLP curl payload injection and instead verify ClickHouse persistence from telemetry generated by HyperDX app activity, which is compatible with the chart's collector setup in CI. Made-with: Cursor
Use telemetrygen to send traces and logs to the collector (gRPC with HTTP fallback) before checking ClickHouse row growth so the integration assertion validates end-to-end ingestion with valid OTLP traffic. Made-with: Cursor
Use OTLP HTTP/protobuf on port 4318 for synthetic telemetry generation to match the chart's configured endpoint and avoid gRPC transport failures during integration runs. Made-with: Cursor
Set CUSTOM_OTELCOL_CONFIG_FILE and provide an OTEL relay config that routes OTLP traces/logs to ClickHouse in chart integration tests, allowing ingestion assertions to validate persisted data. Made-with: Cursor
Set OPAMP_SERVER_URL to an empty value in test overrides so the clickstack collector uses its built-in standalone ClickHouse pipelines, avoiding supervisor remote-config startup failures in CI. Made-with: Cursor
Reduce the smoke test TIMEOUT from 600s to 300s so ingestion assertion failures are detected sooner in CI. Made-with: Cursor
wrn14897
reviewed
Mar 25, 2026
| kubectl port-forward service/$RELEASE_NAME-$CHART_NAME-app 3000:3000 -n $NAMESPACE & | ||
| pf_pid=$! | ||
| sleep 10 | ||
| pf_pid=$(start_port_forward "service/$RELEASE_NAME-$CHART_NAME-app" "3000" "3000" "hyperdx-ui") |
Collaborator
There was a problem hiding this comment.
Any chance we could add a Playwright test to make sure the log line appears on the search page?
Verify the HyperDX app works end-to-end after OTEL ingestion by registering a user through the UI and searching for the smoke-test log line on /search. This proves registration, default source setup, and the ClickHouse query path all function correctly. Made-with: Cursor
wrn14897
reviewed
Mar 25, 2026
.github/workflows/chart-test.yml
Outdated
| - name: Set up Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '20' |
Collaborator
There was a problem hiding this comment.
can we use node 24? 20 is too old
wrn14897
reviewed
Mar 25, 2026
scripts/e2e/package.json
Outdated
| { | ||
| "private": true, | ||
| "devDependencies": { | ||
| "@playwright/test": "^1.52.0" |
Address PR review feedback on version choices. Made-with: Cursor
wrn14897
approved these changes
Mar 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/smoke-test.shwith ClickHouse query helpers, port-forward lifecycle cleanup, and ingestion pollingtelemetrygenand send them to the in-cluster collector (4318)default.otel_tracesanddefault.otel_logsrow counts increase after telemetry is sentTIMEOUT=300in the integration workflow so failures surface soonerhyperdx.config.OPAMP_SERVER_URL: ""in CI test values so the collector runs in standalone mode with built-in ClickHouse export pipelinesTest
bash -n scripts/smoke-test.shHelm Chart Integration Testpassed on this PR (test-helm-chart)helm-unittestpassed on this PRNew Test Assertions
CLICKHOUSE_APP_PASSWORDfromclickstack-secretand query ClickHouse over8123asappdefault.otel_tracesanddefault.otel_logsWhy This Would Catch
6f29e730856cb5bcc30138dd168794bfdb17441dThat commit fixed a grant-shape issue where split
appgrants could result in missingSELECTprivileges (SELECT ON default.*/SELECT ON system.*) because only the first grant was effectively applied.The new integration assertions actively query
default.otel_tracesanddefault.otel_logsas theappuser before and after telemetry ingestion. If the old broken grant shape were reintroduced, thoseSELECTchecks would fail (or remain unreadable), causing the smoke test and PR checks to fail automatically.Made with Cursor