feat(seatbelt): introduce chaos injection middleware

[martintmk]

Add chaos injection middleware

Introduces Injection, a new middleware under seatbelt::chaos::injection that replaces service output with a user-provided value at a configurable probability. Gated behind the chaos-injection feature flag.

API

use seatbelt::chaos::injection::Injection;

// Callback-based: receives consumed input
Injection::layer("my_injection", &context)
    .rate(0.1)
    .output_with(|input, _args| format!("fault for {input}"));

// Fixed value (cloned each time)
Injection::layer("my_injection", &context)
    .rate(0.1)
    .output("injected error".to_string());

Highlights

• Type-state builder enforces rate + output are set before use
• Dual Service impls — layered::Service and tower_service::Service
• rate() clamps to [0.0, 1.0] instead of panicking
• output_with() receives consumed input for context-aware fault generation
• InjectionConfig with serde support for file-based configuration
• Telemetry — resilience.event counter (chaos.injection) + structured log
• enable_if / disable for conditional activation

Changes

• New feature: chaos-injection (depends on fastrand)
• New module: src/chaos/injection/ (8 files)
• New example: examples/chaos_injection.rs
• New integration tests: tests/chaos_injection.rs (18 tests via rstest, layered + tower)
• Updated lib.rs, context.rs, metrics.rs feature gates

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.0%. Comparing base (9377895) to head (32709e6).

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #335    +/-   ##
========================================
  Coverage   100.0%   100.0%            
========================================
  Files         207      210     +3     
  Lines       15403    15552   +149     
========================================
+ Hits        15403    15552   +149     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds a new chaos injection middleware to the seatbelt crate, enabling controlled fault/output injection into service pipelines for resilience testing.

Changes:

• Introduces seatbelt::chaos::injection middleware (layer + service + config + docs) with optional Tower Service support.
• Adds an example (examples/chaos_injection.rs) and integration tests covering layered + tower execution paths.
• Extends telemetry/metrics plumbing and feature gating to include the new middleware (and ensures resilience event counter is available for more middleware combinations).

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
crates/seatbelt/tests/chaos_injection.rs	Integration tests for chaos injection behavior (layered + tower).
crates/seatbelt/src/metrics.rs	Expands cfg gating so resilience event counter is available for additional middleware/features.
crates/seatbelt/src/lib.rs	Documents chaos testing, adds chaos module export, updates feature gating.
crates/seatbelt/src/context.rs	Allows telemetry helper creation when chaos-injection is enabled.
crates/seatbelt/src/chaos/mod.rs	Adds the new top-level chaos module and docs.
crates/seatbelt/src/chaos/injection/telemetry.rs	Defines the chaos injection telemetry event name.
crates/seatbelt/src/chaos/injection/service.rs	Implements the injection middleware and telemetry emission + unit tests.
crates/seatbelt/src/chaos/injection/mod.rs	Public module docs and re-exports for injection.
crates/seatbelt/src/chaos/injection/layer.rs	Builder/type-state layer for configuring injection.
crates/seatbelt/src/chaos/injection/config.rs	Serializable config for injection middleware + tests.
crates/seatbelt/src/chaos/injection/callbacks.rs	Callback wrapper type for injected output factory.
crates/seatbelt/src/chaos/injection/args.rs	Callback args type for injected output factory.
crates/seatbelt/examples/chaos_injection.rs	Example demonstrating injection usage.
crates/seatbelt/Cargo.toml	Adds chaos-injection feature and registers the new example.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 23 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[geeknoid]

Very nice.

What would you think of a variation on this which is "pretend endpoint N is completely dead for the next N seconds". This would provide another tool to test out recovery logic.

[martintmk]

Very nice.

What would you think of a variation on this which is "pretend endpoint N is completely dead for the next N seconds". This would provide another tool to test out recovery logic.

This is already possible with a custom InjectionLayer::output_with that can contain complex logic to extract the some key from the input and fail the execution for a certain period. For now, I keep the actual middleware straightforward.

[geeknoid]

Maybe add an example then of how to simulate an extended outage?

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.