feat: add production resilience features

[mlevkov]

Summary

Adds production-level resilience features including circuit breaker pattern, Prometheus metrics export, request timeout propagation, and fixes for edge cases in IP extraction and validation.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement
• CI/CD changes

Changes Made

• Circuit Breaker Pattern (src/iggy_client/circuit_breaker.rs)

  • Three-state circuit breaker: Closed → Open → HalfOpen
  • Configurable failure/success thresholds and open duration
  • Integrated with IggyClientWrapper.with_reconnect() for fail-fast during outages
  • New AppError::CircuitOpen error type (returns 503)

• Prometheus Metrics (src/metrics.rs)

  • Counters: iggy_messages_sent_total, iggy_messages_polled_total, iggy_connection_reconnects_total, iggy_circuit_breaker_opens_total
  • Histograms: iggy_request_duration_seconds, iggy_send_duration_seconds, iggy_poll_duration_seconds
  • Gauges: iggy_connection_status, iggy_circuit_breaker_state
  • Configurable via METRICS_PORT (default: 9090, 0 = disabled)

• Request Timeout Propagation (src/middleware/timeout.rs)

  • Clients can specify X-Request-Timeout header (milliseconds)
  • Bounded: 100ms minimum, 5 minutes maximum
  • RequestTimeoutExt trait for easy extraction in handlers

• Bug Fixes

  • Empty/whitespace-only X-Forwarded-For headers now return "unknown" instead of empty string
  • Prevents creating separate rate-limit buckets for malformed headers

• Validation Improvements

  • Added MAX_CONSUMER_ID (1 billion) upper bound validation
  • Catches likely misconfigurations with garbage data

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed

Test commands run:

cargo fmt --check
cargo clippy -- -D warnings
cargo test --lib  # 130 tests passing

Checklist

Code Quality

• Code follows project style guidelines (cargo fmt)
• No new Clippy warnings (cargo clippy -- -D warnings)
• Public APIs have documentation comments
• Error handling is appropriate (no unwrap in production code)

Testing

• Tests cover the happy path
• Tests cover error cases
• All existing tests pass

Documentation

• CLAUDE.md updated (if architectural changes)
• README updated (if user-facing changes)
• Code comments explain "why" not "what"

Security

• No secrets or credentials committed
• Input validation added where needed
• No new security vulnerabilities introduced

New Configuration Options

Variable	Default	Description
CIRCUIT_BREAKER_FAILURE_THRESHOLD	5	Failures before opening circuit
CIRCUIT_BREAKER_SUCCESS_THRESHOLD	2	Successes in half-open to close
CIRCUIT_BREAKER_OPEN_DURATION_SECS	30	How long circuit stays open
METRICS_PORT	9090	Prometheus metrics port (0 = disabled)

Additional Notes

• Circuit breaker only counts connection-related errors (not validation errors or timeouts with a healthy connection)
• Metrics are recorded even without explicit init_metrics() call (noop if not initialized)
• Request timeout middleware is applied before the request ID layer in the middleware stack