feat(otel): opentelemetry instrumentation #6408

kschelonka · 2026-01-13T21:39:22Z

Description

Local setup for opentelemetry collection, export, and dashboards.

Sentry setup for opentelemetry compatibility

Convert prom-client metrics to otel metrics for HIBP requests

Example: query for a trace and failure metrics (dev environment)

How to test (local)

Pull this branch
Follow instructions to start repository using docker compose
Start the app with npm run dev
Update your .env.local file with the new OTEL env vars in .env.example
Make requests to generate trace and metrics -- /api/v1/hibp/notify endpoint has custom metrics enabled
Visit localhost:3000 (grafana) to see metrics and traces; the README has more specific info

How to test (dev env)

Via dev site

Visit the monitor dev url and log in with your mozilla credentials
Click around the site to start generating traces; non-error traces will be sampled at a low rate
Use the explore pane to search for traces

Via curl

configure kubectl monitor-dev namespace and port-forward the monitor-api-endpoint service (see docs for more info
Make requests the api/v1/hibp/notify endpoint to generate traces and metrics (hibp_notify_requests_total, hibp_notify_request_failures_total) (you will need to pull the dev hibp notify token in order to make successful requests)

Local setup for opentelemetry collection, export, and dashboards. Sentry setup for opentelemetry compatibility Convert prom-client metrics to otel metrics for HIBP requests

Avoid double-counting db queries

codemist · 2026-01-26T21:43:38Z

src/config.ts

+ * (e.g. used by OTEL_RESOURCE_ATTRIBUTES env var:
+ * 'k8s.container.name=monitor-api-endpoint,k8s.namespace.name=monitor-stage...')
+ * */
+export function parseKVList(str?: string): Record<string, string> {


nit: should this be in a util type folder?

I co-located it where it was used like the other utility methods here (getEnvEnum, etc.). I don't think it is likely to be reused by anything else. Do you want them all moved somewhere?

Any reason to not name this getEnvKvlist and then use that in config.otel.resourceAttributes directly, rather than turning that into a string and deferring parsing to the call site?

codemist · 2026-01-26T22:31:47Z

src/instrumentation.ts

 import * as Sentry from "@sentry/nextjs";
-
 export async function register() {
+  console.log(


nit: would there be any risk of this function getting invoked more than once on reload or how the runtime boots?

good question, I will have to read next documentation

Only once according to the docs https://nextjs.org/docs/app/api-reference/file-conventions/instrumentation#register-optional

codemist · 2026-01-26T22:36:25Z

README.md

+- Prometheus (scrapes metrics for grafana; in GCP environment we use Google-Managed Prometheus)
+- Grafana (visualization)
+
+To view metrics locally, visit [Grafana](http://localhost:3000/d/monitor-dashboard/monitor?orgId=1). Some default dashboard panels are seeded. To see traces, navigate to the [Explore] pane in Grafana and select the Tempo datasource. Note that the data won't propagate immediately, so wait a minute if you're not seeing expected activity show up.


praise: Thank you for this write-up! It was helpful in explaining the process and made the setup pretty painless.

src/instrumentation.ts

Vinnl

It would probably be good if someone with more relevant experience took a look too, since I don't know half of these tools, and barely know the other half. So I mainly focused on running this myself.

Not everything worked for me, although I should also note that I'm using devenv instead of Docker, and thus had to align that with the Docker setup. (I did try Docker as well, but it was missing a dependency that I think gets installed when you run npm run dev (but when you ran it, it added the MacOS version) and didn't know how to add it.) I've opened a PR into this branch that adds some modifications that makes some values overridable to enable devenv use: #6440.

I do see data in the Grafana dashboard show up:

But if I go to the Explore page and select Tempo (which I've never heard of), as described in the readme, I see nothing:

Not sure what I'm doing wrong there...

Vinnl · 2026-01-28T14:41:28Z

src/config.ts

+ * (e.g. used by OTEL_RESOURCE_ATTRIBUTES env var:
+ * 'k8s.container.name=monitor-api-endpoint,k8s.namespace.name=monitor-stage...')
+ * */
+export function parseKVList(str?: string): Record<string, string> {


Any reason to not name this getEnvKvlist and then use that in config.otel.resourceAttributes directly, rather than turning that into a string and deferring parsing to the call site?

Vinnl · 2026-01-28T14:44:38Z

src/instrumentation.node.ts


 declare global {
-  var metrics: Readonly<MetricsState>;
+  var __metrics: Readonly<MetricsState> | undefined;


Thought: Suddenly wondering if this is something for which Node's AsyncLocalContext would make sense? Though I suppose this works fine, so 🤷

Can you explain more?

groovecoder

High-level review looks good. I didn't see metrics showing up in the pre-built monitor-dashboard in the local grafana, but I saw the data from Prometheus and Tempo show up so I assume it's working.

chore(instrumentation): use cached singleton vs. global metrics

kschelonka added 6 commits January 26, 2026 12:09

feat(otel): opentelemetry instrumentation

3a94e50

Local setup for opentelemetry collection, export, and dashboards. Sentry setup for opentelemetry compatibility Convert prom-client metrics to otel metrics for HIBP requests

chore: dynamic import for config.ts only in node

0374c86

feat(otel): ingest OTEL_RESOURCE_ATTRIBUTES

cff11b6

chore(otel): disable instrumentation-pg in favor of knex

d28e0f0

Avoid double-counting db queries

chore: fix test input string

cb0a7c4

chore(deps): group otel dependencies for dependabot

14c9c0a

kschelonka force-pushed the feat/otel-test branch from 128612b to 14c9c0a Compare January 26, 2026 20:10

kschelonka added 2 commits January 26, 2026 12:11

chore(docs): move o11y section

3f3e34e

chore(deps): update package-lock.json

9ecbac3

kschelonka marked this pull request as ready for review January 26, 2026 20:27

kschelonka requested review from Vinnl and codemist January 26, 2026 20:38

kschelonka force-pushed the feat/otel-test branch from c183730 to 9ecbac3 Compare January 26, 2026 20:58

codemist reviewed Jan 26, 2026

View reviewed changes

src/instrumentation.ts Show resolved Hide resolved

Vinnl reviewed Jan 28, 2026

View reviewed changes

groovecoder approved these changes Jan 28, 2026

View reviewed changes

kschelonka added 2 commits January 28, 2026 16:37

chore: remove duplicative gcp detector

e166140

chore(config): invoke key-value parsing directly in config

0326731

chore(instrumentation): use cached singleton vs. global metrics

kschelonka force-pushed the feat/otel-test branch from 3989c39 to 0326731 Compare January 29, 2026 01:11

feat(otel): opentelemetry instrumentation #6408

Are you sure you want to change the base?

feat(otel): opentelemetry instrumentation #6408

Conversation

kschelonka commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How to test (local)

How to test (dev env)

Via dev site

Via curl

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kschelonka Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kschelonka Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Vinnl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

groovecoder left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kschelonka commented Jan 13, 2026 •

edited

Loading

kschelonka Jan 27, 2026 •

edited

Loading

kschelonka Jan 29, 2026 •

edited

Loading