Add Observability Stack and Comprehensive Documentation (#6)

* feat(observability): add Grafana and Prometheus monitoring stack

- Add Prometheus service with Iggy metrics scraping (15s interval)
- Add Grafana service with auto-provisioned datasource and dashboard
- Configure Iggy server to expose /metrics endpoint
- Create pre-built Iggy Overview dashboard with:
  - Server status indicator
  - HTTP request rate graph
  - Message throughput visualization
  - Request latency percentiles (p50, p95)
  - Active streams counter
- Add shared Docker network for service discovery
- Update CLAUDE.md with observability documentation

Access points:
- Iggy HTTP: localhost:3000
- Prometheus: localhost:9090
- Grafana: localhost:3001 (admin/admin)

* feat(observability): add Iggy Web UI dashboard

- Add apache/iggy-web-ui service on port 3050
- Configure Web UI to connect to Iggy HTTP API
- Update documentation with Web UI access and features

Web UI provides:
- Streams and topics management
- Message browsing and inspection
- User management
- Server health monitoring

Access: http://localhost:3050 (iggy/iggy)

* docs: add event-driven architecture guide and update README

- Add comprehensive guide covering:
  - Streams, topics, and partitions explained
  - Partition keys and ordering guarantees
  - Consumer groups and scaling patterns
  - Message retention and expiry configuration
  - Error handling strategies (at-least-once, DLQ, idempotency)
  - Production patterns (outbox, saga, versioning)
  - Complete code examples for producers and consumers
  - CLI and HTTP API quick reference

- Update README with:
  - Observability stack documentation (Grafana, Prometheus, Web UI)
  - Service access URLs and credentials
  - Architecture diagram with observability layer
  - Prometheus metrics and OpenTelemetry configuration
  - Link to new guide in documentation section

* docs: add partitioning guide and enhance main guide

- Add comprehensive partitioning-guide.md covering:
  - What partitions are and why they exist
  - The ordering problem and partition key solutions
  - Domain-specific partition key examples
  - Common partitioning mistakes and how to avoid them
  - Partition count guidelines and capacity planning
  - Consumer group rebalancing mechanics
  - Real-world scenario walkthroughs
  - Troubleshooting guide

- Enhance docs/guide.md with educational details:
  - Add detailed Database struct explanation with SQL examples
  - Add database schema for idempotency pattern
  - Expand Outbox pattern with problem statement and solution
  - Add outbox table schema and publisher query examples
  - Add Further Reading references to external resources

* docs: enhance partitioning guide with Iggy-specific details

Add comprehensive Iggy-specific content to the partitioning guide:

- Iggy Partitioning Strategies section:
  - Balanced (round-robin) strategy
  - Partition ID (direct assignment) strategy
  - Messages Key (hash-based) strategy with MurmurHash3 explanation
  - Comparison table and use case guidance

- Iggy Partition Storage Architecture section:
  - Physical storage layout (streams/topics/partitions/segments)
  - Segment structure and benefits
  - Offset and time indexes explained
  - Index caching modes

- Iggy Server Configuration section:
  - Partition settings (fsync, checksum, buffer thresholds)
  - Segment settings (size, expiry, caching, archival)
  - Topic settings (max_size, auto-delete)
  - Message deduplication options
  - Configuration trade-offs table

- Custom Partitioners section:
  - Partitioner trait explanation
  - Example: Weighted partitioner for heterogeneous hardware
  - Example: Time-based partitioner for analytics
  - Example: Content-based priority partitioner
  - Integration with IggyClient

- Enhanced Further Reading section:
  - Apache Iggy official resources
  - Distributed systems fundamentals
  - Related technologies (Kafka, MurmurHash, Cassandra)
  - Benchmarking resources

* docs: add durable storage guide and documentation index

- Add comprehensive durable storage guide covering:
  - Storage architecture (append-only log, segments)
  - Durability configuration (fsync settings, trade-offs)
  - Data retention policies (time-based, size-based)
  - Backup and archiving (disk, S3-compatible storage)
  - Recovery procedures and data integrity
  - Performance vs durability trade-offs
  - Production recommendations with config templates

- Add docs/README.md as documentation index with:
  - Quick navigation by topic
  - Links to all available guides
  - External resources and references

- Update README.md and CLAUDE.md to reference docs/ directory

* chore: update dependencies

* chore: add .claude/settings.local.json to gitignore

* docs: update changelog for observability and documentation additions

Author: mlevkov

.gitignore

@@ -31,3 +31,6 @@ fuzz/target/
 fuzz/corpus/
 fuzz/artifacts/
 fuzz/Cargo.lock
+
+# local settings
+.claude/settings.local.json

CHANGELOG.md

@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- **Observability Stack**: Complete Grafana-based monitoring setup
+  - Prometheus metrics collection (port 9090) with 15-day retention
+  - Grafana dashboards (port 3001) with pre-configured Prometheus datasource
+  - Iggy Web UI integration (port 3050) for stream/topic/message management
+  - Pre-built Iggy Overview dashboard (server status, request rates, throughput, latency)
+- **Documentation Guides**:
+  - Event-driven architecture guide (`docs/guide.md`): streams/topics/partitions, consumer groups, error handling patterns, production patterns (outbox, saga, idempotency)
+  - Partitioning guide (`docs/partitioning-guide.md`): partition keys, ordering guarantees, selection strategies
+  - Durable storage guide (`docs/durable-storage-guide.md`): storage architecture, fsync configuration, S3 backup/archiving, recovery procedures
+  - Documentation index (`docs/README.md`) with topic-based navigation
+
+### Changed
+
+- Updated `docker-compose.yaml` with full observability stack configuration
+- Simplified documentation section in README.md to reference `docs/` directory
+
 ## [0.1.0] - 2024-12-01
 
 ### Added

CLAUDE.md

@@ -75,6 +75,17 @@ This application showcases how to build a production-ready message streaming ser
 
 .commitlintrc.json            # Conventional commits configuration
 
+observability/
+├── prometheus/
+│   └── prometheus.yml      # Prometheus scrape configuration
+└── grafana/
+    └── provisioning/
+        ├── datasources/
+        │   └── datasources.yml    # Prometheus datasource
+        └── dashboards/
+            ├── dashboards.yml     # Dashboard provisioning
+            └── iggy-overview.json # Pre-built Iggy dashboard
+
 src/
 ├── main.rs           # Application entry point
 ├── lib.rs            # Library exports
@@ -117,8 +128,19 @@ fuzz/
     └── fuzz_validation.rs # Validation function fuzz tests
 
 deny.toml                 # License and security policy for cargo-deny
+
+docs/                     # Documentation and guides
+├── README.md             # Guide index and navigation
+├── guide.md              # Event-driven architecture guide
+├── partitioning-guide.md # Partitioning strategies
+├── durable-storage-guide.md # Storage and durability configuration
+└── structured-concurrency.md # Task lifecycle management
 ```
 
+## Documentation
+
+See the [docs/](docs/) directory for comprehensive guides covering event-driven architecture, partitioning strategies, durable storage configuration, and more.
+
 ## API Endpoints
 
 ### Health & Status
@@ -247,6 +269,81 @@ RUST_LOG=debug
 RUST_LOG=trace
 ```
 
+## Observability Stack
+
+The project includes a complete Grafana-based observability stack for monitoring Iggy and the sample application.
+
+### Components
+
+| Service | Port | Description |
+|---------|------|-------------|
+| **Iggy** | 3000 | Message streaming server (also serves `/metrics`) |
+| **Iggy Web UI** | 3050 | Dashboard for streams, topics, messages, and users |
+| **Prometheus** | 9090 | Metrics collection and storage |
+| **Grafana** | 3001 | Visualization and dashboards |
+
+### Quick Start with Observability
+
+```bash
+# Start the full stack (Iggy + Prometheus + Grafana)
+docker-compose up -d
+
+# Access the services:
+# - Iggy HTTP API: http://localhost:3000
+# - Iggy Web UI: http://localhost:3050 (iggy/iggy)
+# - Prometheus: http://localhost:9090
+# - Grafana: http://localhost:3001 (admin/admin)
+```
+
+### Iggy Web UI
+
+The Iggy Web UI provides a comprehensive dashboard for managing the Iggy server:
+
+- **Streams & Topics**: Create, browse, and delete streams and topics
+- **Messages**: Browse and inspect messages in topics
+- **Users**: Manage users and permissions
+- **Server Health**: Monitor server status and connections
+
+Access at http://localhost:3050 with credentials `iggy/iggy`.
+
+### Grafana Dashboards
+
+Pre-configured dashboards are automatically provisioned:
+
+- **Iggy Overview**: Server status, request rates, message throughput, latency percentiles
+
+### Prometheus Metrics
+
+Iggy exposes Prometheus-compatible metrics at `/metrics`:
+
+```bash
+# View raw metrics
+curl http://localhost:3000/metrics
+```
+
+### Customizing Dashboards
+
+1. Log into Grafana at http://localhost:3001
+2. Navigate to Dashboards → Iggy folder
+3. Edit existing dashboards or create new ones
+4. Export JSON and save to `observability/grafana/provisioning/dashboards/`
+
+### Adding OpenTelemetry (Optional)
+
+Iggy supports OpenTelemetry for distributed tracing. Add to docker-compose:
+
+```yaml
+environment:
+  - IGGY_TELEMETRY_ENABLED=true
+  - IGGY_TELEMETRY_SERVICE_NAME=iggy
+  - IGGY_TELEMETRY_LOGS_TRANSPORT=grpc
+  - IGGY_TELEMETRY_LOGS_ENDPOINT=http://otel-collector:4317
+  - IGGY_TELEMETRY_TRACES_TRANSPORT=grpc
+  - IGGY_TELEMETRY_TRACES_ENDPOINT=http://otel-collector:4317
+```
+
+---
+
 ## Development
 
 ### Prerequisites
@@ -255,9 +352,9 @@ RUST_LOG=trace
 
 ### Quick Start
 
-1. Start Iggy server:
+1. Start Iggy server (with observability):
    ```bash
-   docker-compose up -d iggy
+   docker-compose up -d iggy prometheus grafana
    ```
 
 2. Run the application: