Add Observability Stack and Comprehensive Documentation

[mlevkov]

Summary

Adds a complete Grafana-based observability stack for monitoring Iggy and comprehensive documentation guides for event-driven architecture, partitioning, and durable storage.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Performance improvement
• CI/CD changes

Changes Made

Observability Stack

• Prometheus (port 9090): Metrics collection with 15-day retention, scraping Iggy at /metrics
• Grafana (port 3001): Pre-configured dashboards with Prometheus datasource (admin/admin)
• Iggy Web UI (port 3050): Dashboard for managing streams, topics, messages, and users (iggy/iggy)
• Pre-built Iggy Overview dashboard with server status, request rates, message throughput, and latency percentiles
• Updated docker-compose.yaml with full stack configuration

Documentation

• docs/guide.md: Comprehensive event-driven architecture guide covering streams/topics/partitions, consumer groups, error handling patterns (idempotency, DLQ), and production patterns (outbox, saga)
• docs/partitioning-guide.md: Deep dive into partition keys, ordering guarantees, and partition selection strategies
• docs/durable-storage-guide.md: Storage architecture, fsync configuration, backup/archiving to S3, recovery procedures, and production recommendations
• docs/README.md: Documentation index with quick navigation by topic
• Updated README.md and CLAUDE.md to reference the docs directory
• Updated CHANGELOG.md

Housekeeping

• Added .claude/settings.local.json to .gitignore
• Dependency updates in Cargo.lock

Testing

• Unit tests added/updated
• Integration tests added/updated
• Manual testing performed

Test commands run:

# Verified observability stack starts correctly
docker-compose up -d

# Verified all services accessible:
# - Iggy HTTP API: http://localhost:3000
# - Iggy Web UI: http://localhost:3050
# - Prometheus: http://localhost:9090
# - Grafana: http://localhost:3001

Checklist

Code Quality

• Code follows project style guidelines (cargo fmt)
• No new Clippy warnings (cargo clippy -- -D warnings)
• Public APIs have documentation comments
• Error handling is appropriate (no unwrap in production code)

Testing

• Tests cover the happy path
• Tests cover error cases
• All existing tests pass

Documentation

• CLAUDE.md updated (if architectural changes)
• README updated (if user-facing changes)
• Code comments explain "why" not "what"

Security

• No secrets or credentials committed
• Input validation added where needed
• No new security vulnerabilities introduced

Related Issues

None

Screenshots (if applicable)

N/A - Infrastructure and documentation changes only

Additional Notes

Quick Start

# Start the full observability stack
docker-compose up -d

# Access points:
# - Iggy HTTP API: http://localhost:3000
# - Iggy Web UI: http://localhost:3050 (iggy/iggy)
# - Prometheus: http://localhost:9090
# - Grafana: http://localhost:3001 (admin/admin)

Files Changed

• 13 files changed, +3,965 lines, -86 lines