chore(engine): add support for epoxy debugging

[NathanFlurry]

No description provided.

[railway-app]

🚅 Deployed to the rivet-pr-3794 environment in rivet-frontend

Service	Status	Web	Updated (UTC)
frontend-cloud-staging	✅ Success (View Logs)	Web	Jan 10, 2026 at 2:03 am
frontend-inspector	✅ Success (View Logs)	Web	Jan 10, 2026 at 2:01 am
frontend-cloud	✅ Success (View Logs)	Web	Jan 10, 2026 at 2:01 am
website	✅ Success (View Logs)	Web	Jan 10, 2026 at 2:01 am

---

🚅 Environment rivet-pr-3794 in rivet-frontend-website-preview has no services deployed.

[NathanFlurry]

• feat: add metrics to pb, guard #3815
• fix: clear all pending signals after workflow complete #3798
• fix: remove pending actors metric, fix actor error tracker, engine runner error print #3797
• chore(engine): add support for epoxy debugging #3794 👈 (View in Graphite)
• main

---

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

[claude]

PR Review: Add Support for Epoxy Debugging

Overview

This PR adds comprehensive debugging capabilities for the Epoxy distributed consensus system, including replica state inspection, key-level debugging, and KV operations through both CLI commands and API endpoints.

Code Quality & Best Practices

✅ Strengths

1. Good code organization - Clear separation between API handlers (api-peer/internal.rs) and CLI commands (engine/commands/epoxy.rs)
2. Comprehensive functionality - Covers replica debugging, key inspection, fanout operations, and KV get/set
3. Documentation - Good doc comments explaining what each endpoint does
4. Error handling - Proper use of anyhow::Context throughout
5. Follows repository patterns - Uses existing utilities like rivet_pools::reqwest::client() correctly

🔍 Issues & Suggestions

1. Inconsistent error handling style (engine/packages/api-peer/src/internal.rs:452-476)

The fanout function uses fully qualified std::result::Result::Ok/Err instead of relying on use statements. Consider adding a type alias or using pattern matching directly for consistency.

2. Sequential fanout requests (engine/packages/api-peer/src/internal.rs:441-477)

The get_epoxy_key_debug_fanout function makes HTTP requests sequentially in a for loop. This can be slow when there are many replicas. If there are 10 replicas and each takes 100ms to respond, this will take ~1 second instead of ~100ms.

Recommendation: Use futures::future::join_all or tokio::spawn to make requests concurrently.

3. Missing timeout configuration

The fanout and HTTP request operations do not specify timeouts. If a replica is unresponsive, these could hang indefinitely. Consider adding request timeouts.

4. Base64 import inconsistency

Some files use use base64::Engine; then call base64::engine::general_purpose::STANDARD.decode(), while others inline the import. Use the Engine trait pattern consistently.

5. Debug format for status string (engine/packages/api-peer/src/internal.rs:361)

The {:?} debug format may not be user-friendly for a debugging interface. Consider implementing a Display trait or document that this is intentional.

Security Concerns

⚠️ Moderate Risk: Unauthenticated internal API endpoints

The new endpoints are exposed on the internal API peer:

• /epoxy/replica/debug
• /epoxy/replica/key/{key}
• /epoxy/replica/kv/{key} (with PUT method)

Concern: These endpoints allow:

1. Reading arbitrary key-value data from the epoxy system
2. Writing/modifying values through consensus (set_epoxy_kv)
3. Accessing internal cluster state information

Questions:

• Are these endpoints protected by network segmentation (internal-only network)?
• Should there be authentication/authorization for write operations?
• Is there rate limiting to prevent abuse?

Recommendation:

• Document the security model (internal-only access assumed?)
• Consider adding an audit log for write operations
• Add authentication if these could be exposed beyond trusted networks

Performance Considerations

1. ✅ Good: Reuses reqwest client from pool (rivet_pools::reqwest::client())
2. ⚠️ Concern: Sequential fanout requests (see issue [SVC-2479] Send cluster events to PostHog #2 above)
3. ✅ Good: Uses UDB transactions appropriately

Test Coverage

⚠️ No tests included in this PR

While this is a debugging/tooling feature, consider adding:

• Unit tests for encode_key() function with various tuple formats
• Integration tests for the CLI commands
• Tests for error cases (invalid base64, missing replicas, etc.)

Additional Observations

Dependency Management ✅

• Correctly added epoxy-protocol as a workspace dependency
• Alphabetized dependencies in Cargo.toml (follows conventions)

Code Style ✅

• Follows Rust formatting conventions
• Uses structured logging appropriately
• Lowercase log messages (follows CLAUDE.md)

Summary

This is a well-structured PR that adds useful debugging capabilities for Epoxy. The main concerns are:

1. Performance: Sequential fanout requests should be parallelized
2. Security: Document/implement authentication for internal endpoints
3. Testing: Add test coverage for key encoding and edge cases
4. Minor: Inconsistent error handling style in fanout logic

The code quality is generally good and follows repository conventions. With the above improvements, this would be production-ready.

Recommendations Priority

High Priority:

• Parallelize fanout HTTP requests
• Document security model for internal API endpoints
• Add request timeouts

Medium Priority:

• Add basic test coverage
• Standardize error handling style

Low Priority:

• Improve status string formatting
• Add usage examples to documentation

[pkg-pr-new]

More templates

• actor-actions
• example-ai-agent
• example-ai-and-user-generated-actors-freestyle
• chat-room
• chat-room-next-js
• example-cloudflare-workers
• example-cloudflare-workers-hono
• example-cloudflare-workers-inline-client
• cross-actor-actions
• example-cursors
• example-cursors-raw-websocket
• custom-serverless
• example-sqlite
• example-elysia
• example-experimental-durable-streams-ai-agent
• example-hono
• hono-react
• example-kitchen-sink
• multi-region
• native-websockets
• next-js
• example-node
• example-raw-fetch-handler
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-react
• scheduling
• state
• example-stream
• example-trpc

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@3794

@rivetkit/db

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/db@3794

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@3794

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@3794

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@3794

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@3794

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@3794

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@3794

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@3794

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@3794

commit: d789061

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
rivet-cloud	Ready	Preview, Comment	Jan 10, 2026 2:01am
rivetkit-serverless	Error		Jan 10, 2026 2:01am

2 Skipped Deployments

Project	Deployment	Review	Updated (UTC)
rivet-inspector	Ignored	Preview	Jan 10, 2026 2:01am
rivet-site	Ignored	Preview	Jan 10, 2026 2:01am