fix(otel): enrich http traces

[MasterPtato]

No description provided.

[MasterPtato]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• fix(pegboard): cache none runner pool errors too #3899
• fix(engine-runner): handle shutdown close codes correctly #3896
• chore(gas): add overview and history docs #3879
• fix(otel): enrich http traces #3878 👈 (View in Graphite)
• fix(guard): clean up guard #3862
• fix: add ready_chunks to worker bumps, bumps per tick metric #3855
• fix(epoxy): add metrics #3847
• chore: clean up errors #3846
• fix(pb): have restart policy actors sleep after pending for too long #3837
• fix: set failure reason regardless of crash policy #3835
• feat: add metrics to pb, guard #3815
• fix: clear all pending signals after workflow complete #3798
• fix: remove pending actors metric, fix actor error tracker, engine runner error print #3797
• chore(engine): add support for epoxy debugging #3794
• main

---

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

PR Review: fix(otel): enrich http traces

Summary

This PR adds OpenTelemetry span attributes to HTTP traces in both the API builder middleware and guard proxy service to improve observability. The changes add standard HTTP semantic attributes like method, path, route, and status code to spans.

Code Quality ✅

Strengths:

• Clean implementation following OpenTelemetry patterns
• Consistent approach across both middleware and proxy service
• Proper use of structured attributes instead of embedding data in span names
• Correctly added tracing-opentelemetry dependency to guard-core
• Good span lifecycle management by storing current_span in a variable

Issues to Address:

1. Non-standard attribute naming

   • Location: engine/packages/api-builder/src/middleware.rs:71, engine/packages/guard-core/src/proxy_service.rs:2039
   • Issue: Using http.path instead of standard OpenTelemetry semantic conventions
   • Recommendation: According to OpenTelemetry HTTP semantic conventions v1.23+, use:
     • url.path for the path component (e.g., /api/users)
     • url.full for the complete URI if needed
   • Impact: Non-standard attributes may not be recognized by observability tools and dashboards expecting standard conventions

2. Redundant span data

   • Location: engine/packages/api-builder/src/middleware.rs:70-71
   • Issue: The span already records method and uri as tracing fields during span creation (lines 65-66), then sets them again as OpenTelemetry attributes
   • Analysis: While not incorrect, this duplicates data. Tracing fields and OTel attributes serve different purposes (fields for tracing display, attributes for OTel exporters)
   • Recommendation: This is acceptable if intentional for cross-tool compatibility, but consider documenting why both are needed

3. Missing attribute validation

   • Location: All set_attribute calls
   • Issue: No length limits on string conversions that could result in very large trace payloads
   • Recommendation: Consider truncating extremely long URIs (e.g., those with large query strings) to prevent bloating trace data. Most observability backends have payload limits.

Best Practices 📋

Follows CLAUDE.md conventions:

• ✅ Uses structured logging (tracing with separate fields)
• ✅ Imports added at the top of files
• ✅ No formatting changes made
• ✅ Proper dependency management using workspace dependencies
• ✅ Lowercase log messages

OpenTelemetry Semantic Convention Status:

• http.request.method → ✅ Correct (stable in v1.23+)
• http.path → ❌ Should be url.path (as of v1.23+)
• http.route → ✅ Correct (stable in v1.23+)
• http.response.status_code → ✅ Correct (stable in v1.23+)

Potential Bugs 🐛

None identified - The implementation is safe and straightforward. All attribute values are properly typed (strings and i64).

Performance Considerations ⚡

Current impact: Minimal

• String allocations for attributes are small and infrequent (once per request)
• No blocking operations introduced
• Attribute setting is efficient in the tracing-opentelemetry layer

Potential concern:

• Very long URIs (e.g., with large base64-encoded data in query params) could increase trace payload size
• Consider adding length validation if this becomes an issue

Security Concerns 🔒

Low risk, but consider:

• URIs may contain sensitive data (auth tokens, API keys in query params)
• Ensure your OpenTelemetry collector/backend pipeline sanitizes sensitive data
• Consider adding attribute value sanitization for known sensitive patterns

Test Coverage 🧪

Not included in PR:

• No new tests added
• Recommendation: Consider adding integration tests that verify:
  1. Span attributes are correctly set on successful requests
  2. Span attributes are correctly set on error responses
  3. Attributes follow expected format and types

Example test structure:

#[tokio::test]
async fn test_http_span_attributes() {
    // Setup tracing subscriber with in-memory exporter
    // Make HTTP request through middleware
    // Assert span contains expected attributes with correct values
}

Recommendations

High Priority:

1. Update http.path to url.path to match OpenTelemetry semantic conventions v1.23+
2. Consider updating the full URI attribute to url.full if needed

Medium Priority:
3. Add integration tests for span attribute verification
4. Consider adding URI sanitization for sensitive data

Low Priority:
5. Document why both tracing fields and OTel attributes are used (if intentional)
6. Consider adding length limits for URI attributes

Overall Assessment

Status: ✅ Approve with suggestions

This is a solid improvement to observability. The main issue is using non-standard attribute names (http.path instead of url.path), which should be corrected to ensure compatibility with standard OpenTelemetry tooling and dashboards. The implementation is otherwise clean and follows Rivet's conventions well.

[pkg-pr-new]

More templates

• actor-actions
• example-ai-agent
• example-ai-and-user-generated-actors-freestyle
• chat-room
• example-cloudflare-workers
• example-cloudflare-workers-hono
• example-cloudflare-workers-inline-client
• cross-actor-actions
• example-cursors
• example-cursors-raw-websocket
• custom-serverless
• example-sqlite
• example-elysia
• example-experimental-durable-streams-ai-agent
• hello-world
• example-hono
• hono-react
• example-kitchen-sink
• multi-region
• native-websockets
• next-js
• example-raw-fetch-handler
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-react
• scheduling
• state
• example-stream
• example-trpc

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@3878

@rivetkit/db

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/db@3878

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@3878

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@3878

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@3878

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@3878

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@3878

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@3878

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@3878

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@3878

commit: 116ea58