fix(epoxy): add metrics

[MasterPtato]

Fixes RVT-5331

[MasterPtato]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• fix(pegboard): cache none runner pool errors too #3899
• fix(engine-runner): handle shutdown close codes correctly #3896
• chore(gas): add overview and history docs #3879
• fix(otel): enrich http traces #3878
• fix(guard): clean up guard #3862
• fix: add ready_chunks to worker bumps, bumps per tick metric #3855
• fix(epoxy): add metrics #3847 👈 (View in Graphite)
• chore: clean up errors #3846
• fix(pb): have restart policy actors sleep after pending for too long #3837
• fix: set failure reason regardless of crash policy #3835
• feat: add metrics to pb, guard #3815
• fix: clear all pending signals after workflow complete #3798
• fix: remove pending actors metric, fix actor error tracker, engine runner error print #3797
• chore(engine): add support for epoxy debugging #3794
• main

---

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[linear]

RVT-5331 Add metrics to epoxy

[claude]

PR Review: Add metrics to Epoxy

Summary

This PR adds comprehensive Prometheus metrics to the Epoxy consensus system. The implementation covers consensus operations, message handling, quorum tracking, replica state, cluster health, and error conditions.

✅ Strengths

1. Comprehensive coverage - Metrics cover all critical aspects of the EPaxos consensus protocol
2. Well-organized - Metrics are logically grouped with clear MARK comments
3. Proper metric types - Correct use of counters, gauges, and histograms
4. Good label cardinality - Labels are well-chosen without creating cardinality explosions
5. Consistent naming - All metrics follow the epoxy_* naming convention

🔍 Issues & Suggestions

Critical Issues

1. Error classification mismatch in proposals (ops/propose.rs:94-99)

The proposal status metric incorrectly classifies CommandError as an error:

metrics::PROPOSALS_TOTAL
    .with_label_values(&[if let ProposalResult::Committed = res {
        "ok"
    } else {
        "err"
    }])
    .inc();

According to the code comments at ops/propose.rs:18-20:

Command errors indicate that a proposal succeeded but the command did not apply.
Proposals that have command errors are still written to the log but have no effect.

Issue: CommandError represents a successful consensus operation with an expected application-level failure (e.g., optimistic locking failure). Tagging it as "err" will make it appear as if consensus failed, which is misleading for monitoring.

Recommendation: Change the status label to distinguish three cases:

metrics::PROPOSALS_TOTAL
    .with_label_values(&[match &res {
        ProposalResult::Committed => "committed",
        ProposalResult::CommandError(_) => "command_error",
        ProposalResult::ConsensusFailed => "consensus_failed",
    }])
    .inc();

This provides better observability by separating:

• committed - Full success
• command_error - Consensus succeeded but command didn't apply (expected in optimistic concurrency scenarios)
• consensus_failed - Actual consensus failure

2. Commit error classification (replica/messages/commit.rs:43-45)

Similar issue with commit metrics:

metrics::COMMIT_TOTAL
    .with_label_values(&[if cmd_err.is_none() { "ok" } else { "cmd_err" }])
    .inc();

While this is better than the proposal metric, it's inconsistent. The commit always succeeds at the consensus level even if there's a command error. Consider using labels like "success" / "command_error" to make it clear this is tracking successful commits, with command errors being a subset.

Medium Priority

3. Missing error details in request metrics (replica/message_request.rs:35-36)

metrics::REQUESTS_TOTAL
    .with_label_values(&[request_type, if res.is_ok() { "ok" } else { "err" }])
    .inc();

Issue: All errors are bucketed together. For a distributed consensus system, different error types have different operational implications (network failures, ballot rejections, validation failures, etc.).

Recommendation: Consider adding error type information, though be careful about cardinality. At minimum, you could add a separate counter for specific critical error types.

4. Quorum metric placement (http_client.rs:95-104)

The quorum metric is recorded after the function returns successful responses, but there's no failure path metric if an error occurs before reaching that point.

Issue: If fanout_to_replicas returns an error (e.g., at line 106), the metric won't be recorded, creating a blind spot in monitoring.

Recommendation: Use a defer-like pattern or move the metric to a finally block to ensure it's always recorded. You could use a pattern like:

let result = /* ... */;
let status = if result.is_ok() { "ok" } else { "error" };
metrics::QUORUM_ATTEMPTS_TOTAL
    .with_label_values(&[quorum_type.to_string().as_str(), status])
    .inc();
result

Minor Issues

5. Inconsistent result label naming

The PR uses different label values for similar concepts:

• PROPOSALS_TOTAL: "ok" / "err"
• PRE_ACCEPT_TOTAL: "ok" / "invalid_ballot"
• ACCEPT_TOTAL: "ok" / "invalid_ballot"
• COMMIT_TOTAL: "ok" / "cmd_err"
• REQUESTS_TOTAL: "ok" / "err"
• QUORUM_ATTEMPTS_TOTAL: "ok" / "insufficient_responses"

Recommendation: Standardize on either "ok"/"error" or "success"/"failure" for binary outcomes, and use specific descriptive labels for other cases.

6. Missing histogram for quorum operations

You have PROPOSAL_DURATION (histogram) but QUORUM_ATTEMPTS_TOTAL only has a counter. Quorum latency would be valuable for debugging slow consensus.

Suggestion: Consider adding a QUORUM_DURATION histogram to track how long quorum operations take.

7. Ballot gauge semantics (replica/ballot.rs:53-54)

metrics::BALLOT_EPOCH.set(current_ballot.epoch as i64);
metrics::BALLOT_NUMBER.set(current_ballot.ballot as i64);

These are only updated in increment_ballot, not when ballots are read or validated. This means the metrics might not reflect the true current state if ballots are updated through other paths.

Verification needed: Ensure increment_ballot is the only place ballots can change, or add metric updates to other ballot modification points.

Code Quality

8. Good use of existing reqwest client (http_client.rs:146)

✅ Correctly uses rivet_pools::reqwest::client().await? instead of creating a new client. This follows the CLAUDE.md optimization guidelines.

9. Proper logging style

✅ Log messages follow the guidelines:

• Lowercase: "updating config" (line update_config.rs:13)
• Structured logging with ? for debug: ?err patterns
• Complete sentences

10. Import organization

✅ Imports are properly placed at the top of files, not inline within functions (per CLAUDE.md style guide).

Performance Considerations

• Metric overhead: The metrics add minimal overhead (microseconds per operation). The consensus protocol involves network I/O and database transactions, so metric recording is negligible in comparison.
• Label cardinality: Current label cardinality is low and safe. The highest cardinality is request_type which has ~11 values, well within Prometheus best practices.

Security Considerations

No security concerns identified. Metrics don't expose sensitive data.

Test Coverage

The PR doesn't include tests for the metrics. While metrics themselves are typically not unit tested, consider:

1. Adding integration tests that verify key metrics are recorded during consensus operations
2. Documenting expected metric behavior in the package README

Documentation

Suggestion: Consider adding a METRICS.md file documenting:

• What each metric measures
• Expected values/ranges
• Alerting thresholds
• Example PromQL queries for common operational questions

Overall Assessment

This is a solid addition to the Epoxy package that will greatly improve observability. The main issues are around error classification semantics, which could lead to misleading dashboards if not addressed. Once the critical issues are fixed, this will be ready to merge.

Recommendation: Request changes to fix the error classification issues before merging.

---

Generated by Claude Code

[pkg-pr-new]

More templates

• actor-actions
• example-ai-agent
• example-ai-and-user-generated-actors-freestyle
• chat-room
• example-cloudflare-workers
• example-cloudflare-workers-hono
• example-cloudflare-workers-inline-client
• cross-actor-actions
• example-cursors
• example-cursors-raw-websocket
• custom-serverless
• example-sqlite
• example-elysia
• example-experimental-durable-streams-ai-agent
• hello-world
• example-hono
• hono-react
• example-kitchen-sink
• multi-region
• native-websockets
• next-js
• example-raw-fetch-handler
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-react
• scheduling
• state
• example-stream
• example-trpc

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@3847

@rivetkit/db

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/db@3847

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@3847

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@3847

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@3847

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@3847

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@3847

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@3847

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@3847

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@3847

commit: b007d1a