Skip to content

Conversation

@kschelonka
Copy link
Collaborator

@kschelonka kschelonka commented Jan 13, 2026

Description

Local setup for opentelemetry collection, export, and dashboards.

Sentry setup for opentelemetry compatibility

Convert prom-client metrics to otel metrics for HIBP requests

Example: query for a trace and failure metrics (dev environment)

How to test (local)

  • Pull this branch
  • Follow instructions to start repository using docker compose
  • Start the app with npm run dev
  • Update your .env.local file with the new OTEL env vars in .env.example
  • Make requests to generate trace and metrics -- /api/v1/hibp/notify endpoint has custom metrics enabled
  • Visit localhost:3000 (grafana) to see metrics and traces; the README has more specific info

How to test (dev env)

Via dev site

  • Visit the monitor dev url and log in with your mozilla credentials
  • Click around the site to start generating traces; non-error traces will be sampled at a low rate
  • Use the explore pane to search for traces

Via curl

  • configure kubectl monitor-dev namespace and port-forward the monitor-api-endpoint service (see docs for more info
  • Make requests the api/v1/hibp/notify endpoint to generate traces and metrics (hibp_notify_requests_total, hibp_notify_request_failures_total) (you will need to pull the dev hibp notify token in order to make successful requests)

Local setup for opentelemetry collection, export, and dashboards.

Sentry setup for opentelemetry compatibility

Convert prom-client metrics to otel metrics for HIBP requests
@kschelonka kschelonka marked this pull request as ready for review January 26, 2026 20:27
@kschelonka kschelonka requested review from Vinnl and codemist January 26, 2026 20:38
* (e.g. used by OTEL_RESOURCE_ATTRIBUTES env var:
* 'k8s.container.name=monitor-api-endpoint,k8s.namespace.name=monitor-stage...')
* */
export function parseKVList(str?: string): Record<string, string> {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should this be in a util type folder?

import * as Sentry from "@sentry/nextjs";

export async function register() {
console.log(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: would there be any risk of this function getting invoked more than once on reload or how the runtime boots?

- Prometheus (scrapes metrics for grafana; in GCP environment we use Google-Managed Prometheus)
- Grafana (visualization)

To view metrics locally, visit [Grafana](http://localhost:3000/d/monitor-dashboard/monitor?orgId=1). Some default dashboard panels are seeded. To see traces, navigate to the [Explore] pane in Grafana and select the Tempo datasource. Note that the data won't propagate immediately, so wait a minute if you're not seeing expected activity show up.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

praise: Thank you for this write-up! It was helpful in explaining the process and made the setup pretty painless.

await import("./instrumentation.node");
const { nodeSDKBuilder } = await import("./instrumentation.node");
console.log("[INSTRUMENTATION] Initializing OTEL...");
await nodeSDKBuilder();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this have a try/catch?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants