feat(sprint-10): implement Story 10.1 - Prometheus Metrics Integration (8 points)

SUMMARY:
Implemented comprehensive Prometheus metrics integration for system monitoring,
ML operations tracking, and business metrics collection. All metrics exposed
via /metrics endpoint for Prometheus scraping.

IMPLEMENTATION:

1. Request Metrics Middleware (app/middleware/metrics.py):
   - Request latency histogram with buckets [0.1, 0.5, 1, 2, 5, 10] seconds
   - Request counter by endpoint, method, and status code
   - Active requests gauge tracking concurrent requests
   - Automatic exclusion of /metrics endpoint from tracking

2. ML Metrics Collector (app/services/metrics_collector.py):
   - Training duration histogram with context manager
   - Prediction latency histogram with context manager
   - Model accuracy gauge for performance tracking
   - Dataset size histogram for data profiling

3. Business Metrics Collector (app/services/metrics_collector.py):
   - Active users gauge with time window support (day/week/month)
   - Datasets created counter by user
   - Models trained counter by model type and user
   - Predictions made counter by model

4. Integration (app/main.py):
   - Registered MetricsMiddleware in FastAPI app
   - Added /metrics endpoint for Prometheus scraping
   - Exposed metrics in Prometheus text exposition format

TESTING:
- Created 35 comprehensive tests with 100% code coverage
  - tests/test_middleware/test_metrics.py (16 tests)
  - tests/test_services/test_metrics_collector.py (19 tests)
- All 205 unit tests passing (100% pass rate)

DEPENDENCIES:
- Added prometheus-client>=0.19.0 to pyproject.toml

SPRINT TRACKING:
- Sprint 10 Story 10.1 COMPLETE (8/27 points, 30%)
- Updated SPRINT_10.md with implementation details
- All acceptance criteria met

METRICS NAMING CONVENTIONS:
- http_request_duration_seconds (histogram)
- http_requests_total (counter)
- http_requests_active (gauge)
- ml_training_duration_seconds (histogram)
- ml_prediction_latency_seconds (histogram)
- ml_model_accuracy (gauge)
- ml_dataset_size_rows (histogram)
- business_active_users (gauge)
- business_datasets_created_total (counter)
- business_models_trained_total (counter)
- business_predictions_made_total (counter)

Author: frankbria

SPRINT_10.md

@@ -0,0 +1,222 @@
+# Sprint 10: Monitoring & Documentation
+
+**Sprint Duration**: Oct 9-11, 2025 (3 days - accelerated)
+**Sprint Goal**: Implement comprehensive monitoring with Prometheus and Grafana, complete API documentation with OpenAPI specs, and fill integration test coverage gaps
+**Velocity Target**: 27 story points
+**Points Completed**: 8/27 (30%)
+**Risk Level**: Low (non-breaking improvements)
+**Status**: 🚧 **IN PROGRESS** - Story 10.1 starting
+
+---
+
+## Sprint Overview
+
+### Capacity Planning
+- **Team Size**: 1 developer
+- **Velocity Target**: 27 story points
+- **Focus**: Observability & Documentation
+- **Risk Level**: Low (non-breaking improvements)
+
+### Sprint Goals
+1. ✅ Prometheus metrics integration with system, ML, and business metrics
+2. ⏳ Grafana dashboards for visualization
+3. ⏳ Complete OpenAPI documentation for all endpoints
+4. ⏳ Integration test coverage >80%
+5. ⏳ Monitoring runbook for on-call engineers
+
+---
+
+## Stories
+
+### Story 10.1: Prometheus Metrics Integration (Priority: 🟡, Points: 8)
+
+**Status**: ✅ **COMPLETE**
+**Started**: 2025-10-09
+**Completed**: 2025-10-09
+
+**As a** SRE
+**I want** Prometheus metrics for system and ML operations
+**So that** I can monitor performance and identify issues proactively
+
+**Acceptance Criteria:**
+- [x] System metrics exposed: request latency, error rates, throughput
+- [x] ML metrics exposed: training duration, prediction latency, model accuracy
+- [x] Business metrics exposed: active users, datasets created, models trained
+- [x] Metrics endpoint `/metrics` accessible
+- [x] Metrics follow Prometheus naming conventions
+
+**Technical Tasks:**
+1. Install and configure Prometheus client - 1.5h
+   - Files: `apps/backend/requirements.txt`, `apps/backend/app/middleware/metrics.py` (new)
+   - Add prometheus-client library: `prometheus-client==0.19.0`
+   - Create metrics registry
+   - Configure metrics HTTP handler at `/metrics`
+
+2. Implement request metrics middleware - 2h
+   - File: `apps/backend/app/middleware/metrics.py:20-80`
+   - Track request latency histogram (buckets: 0.1, 0.5, 1, 2, 5, 10s)
+   - Track request count by endpoint and status code
+   - Track active request gauge
+   - Add labels: endpoint, method, status_code
+
+3. Implement ML operation metrics - 2.5h
+   - File: `apps/backend/app/services/metrics_collector.py` (new)
+   - Track training duration histogram
+   - Track prediction latency histogram
+   - Track model accuracy gauge (updated after training)
+   - Track dataset size histogram
+   - Add labels: model_type, problem_type, dataset_id
+
+4. Implement business metrics - 2h
+   - File: `apps/backend/app/services/metrics_collector.py:100-150`
+   - Track active users gauge (from auth)
+   - Track datasets created counter
+   - Track models trained counter
+   - Track predictions made counter
+   - Add time-based labels (day, week, month)
+
+**Dependencies:**
+- Sprint 8: API versioning (metrics paths)
+
+**Risks:**
+- Metrics overhead may impact performance
+- High cardinality labels could cause issues
+
+**Progress:**
+- ✅ Installed prometheus-client library (v0.23.1)
+- ✅ Created request metrics middleware in `app/middleware/metrics.py`
+  - Request latency histogram (buckets: 0.1, 0.5, 1, 2, 5, 10s)
+  - Request count by endpoint, method, and status code
+  - Active requests gauge
+- ✅ Created ML and business metrics collector in `app/services/metrics_collector.py`
+  - ML metrics: training duration, prediction latency, model accuracy, dataset size
+  - Business metrics: active users, datasets created, models trained, predictions made
+- ✅ Registered metrics middleware and `/metrics` endpoint in `app/main.py`
+- ✅ Created comprehensive tests with 100% coverage (35 tests)
+  - `tests/test_middleware/test_metrics.py` (16 tests)
+  - `tests/test_services/test_metrics_collector.py` (19 tests)
+- ✅ All 205 unit tests pass (100% pass rate)
+
+**Story 10.1 Status**: ✅ **COMPLETE**
+
+---
+
+### Story 10.2: Grafana Dashboards (Priority: 🟡, Points: 5)
+
+**Status**: ⏳ **PENDING**
+
+**As a** operator
+**I want** Grafana dashboards for system visualization
+**So that** I can understand system health at a glance
+
+**Acceptance Criteria:**
+- [ ] System health dashboard: latency, errors, throughput
+- [ ] ML operations dashboard: training metrics, model performance
+- [ ] Business metrics dashboard: user activity, usage trends
+- [ ] Dashboards auto-refresh every 30s
+- [ ] Alerts configured for critical thresholds
+
+**Dependencies:**
+- Story 10.1: Prometheus metrics
+
+---
+
+### Story 10.3: OpenAPI Spec Completion (Priority: 🟡, Points: 8)
+
+**Status**: ⏳ **PENDING**
+
+**As an** API consumer
+**I want** complete OpenAPI documentation
+**So that** I can integrate with the API easily
+
+**Acceptance Criteria:**
+- [ ] All endpoints documented with request/response schemas
+- [ ] Authentication documented (JWT bearer token)
+- [ ] Error responses documented with examples
+- [ ] Interactive API docs at `/docs` endpoint
+- [ ] OpenAPI spec downloadable as JSON/YAML
+
+**Dependencies:**
+- Sprint 8: API versioning complete
+
+---
+
+### Story 10.4: Complete Integration Tests (Priority: 🟡, Points: 5)
+
+**Status**: ⏳ **PENDING**
+
+**As a** developer
+**I want** complete integration test coverage
+**So that** service interactions are validated
+
+**Acceptance Criteria:**
+- [ ] S3 integration tests cover upload/download/delete
+- [ ] MongoDB integration tests cover CRUD operations
+- [ ] OpenAI integration tests cover analysis workflows
+- [ ] End-to-end workflow integration tests pass
+- [ ] Integration test coverage >80%
+
+**Dependencies:**
+- Sprint 9.3: Integration test fixtures
+
+---
+
+### Story 10.5: Monitoring Runbook (Priority: 🟢, Points: 1)
+
+**Status**: ⏳ **PENDING**
+
+**As an** on-call engineer
+**I want** monitoring runbook documentation
+**So that** I can respond to incidents effectively
+
+**Acceptance Criteria:**
+- [ ] Alert response procedures documented
+- [ ] Common issue troubleshooting guide
+- [ ] Metric interpretation guide
+- [ ] Escalation procedures defined
+
+**Dependencies:**
+- Stories 10.1 and 10.2 complete
+
+---
+
+## Sprint Validation Gates
+
+- [ ] Prometheus metrics exposed and accurate
+- [ ] 3 Grafana dashboards operational
+- [ ] OpenAPI spec complete at `/docs`
+- [ ] Integration tests passing with >80% coverage
+- [ ] Monitoring runbook reviewed by team
+- [ ] All documentation updated
+
+## Progress Tracking
+
+**Daily Updates:**
+
+### Day 1 (2025-10-09)
+- ✅ Completed Story 10.1: Prometheus Metrics Integration (8 points)
+  - Implemented request metrics middleware with latency, count, and active request tracking
+  - Implemented ML metrics collector for training, prediction, and model metrics
+  - Implemented business metrics collector for users, datasets, models, and predictions
+  - Created `/metrics` endpoint for Prometheus scraping
+  - Wrote 35 comprehensive tests with 100% code coverage
+  - All 205 unit tests passing
+- Next: Story 10.2 (Grafana Dashboards) or Story 10.3 (OpenAPI Spec Completion)
+
+---
+
+## Sprint Retrospective (To be completed)
+
+**What went well:**
+- TBD
+
+**What to improve:**
+- TBD
+
+**Action items for Sprint 11:**
+- TBD
+
+---
+
+**Last Updated**: 2025-10-09
+**Maintained By**: Development team

apps/backend/.coverage

@@ -32,8 +32,11 @@
 from motor.motor_asyncio import AsyncIOMotorClient
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import Response
 
 from app.middleware.api_version import APIVersionMiddleware
+from app.middleware.metrics import MetricsMiddleware, get_metrics
+from prometheus_client import CONTENT_TYPE_LATEST
 from app.api.routes import (
     health,
     user_data,
@@ -127,6 +130,9 @@ async def lifespan(app: FastAPI):
 # ✅ Apply API versioning middleware
 app.add_middleware(APIVersionMiddleware)
 
+# ✅ Apply Prometheus metrics middleware
+app.add_middleware(MetricsMiddleware)
+
 # ✅ Include routers
 # Health check routes at root level (no version prefix)
 app.include_router(health.router, tags=["health"])
@@ -225,3 +231,13 @@ async def lifespan(app: FastAPI):
 @app.get("/")
 async def root():
     return {"message": "Welcome to the Narrative Modeling API"}
+
+
+@app.get("/metrics")
+async def metrics():
+    """
+    Prometheus metrics endpoint.
+
+    Returns metrics in Prometheus text exposition format for scraping.
+    """
+    return Response(content=get_metrics(), media_type=CONTENT_TYPE_LATEST)

apps/backend/app/main.py

@@ -0,0 +1,127 @@
+# apps/backend/app/middleware/metrics.py
+"""
+Prometheus metrics middleware for FastAPI.
+
+Tracks:
+- Request latency histogram (buckets: 0.1, 0.5, 1, 2, 5, 10s)
+- Request count by endpoint, method, and status code
+- Active request gauge
+
+Usage:
+    from app.middleware.metrics import MetricsMiddleware, metrics_registry
+
+    app.add_middleware(MetricsMiddleware)
+
+    @app.get("/metrics")
+    async def metrics():
+        return Response(
+            content=generate_latest(metrics_registry),
+            media_type=CONTENT_TYPE_LATEST
+        )
+"""
+
+import time
+from typing import Callable
+from prometheus_client import (
+    CollectorRegistry,
+    Counter,
+    Histogram,
+    Gauge,
+    generate_latest,
+    CONTENT_TYPE_LATEST,
+)
+from starlette.middleware.base import BaseHTTPMiddleware
+from starlette.requests import Request
+from starlette.responses import Response
+from starlette.types import ASGIApp
+
+# Create a custom registry for application metrics
+metrics_registry = CollectorRegistry()
+
+# Request latency histogram with specified buckets
+request_latency = Histogram(
+    name="http_request_duration_seconds",
+    documentation="HTTP request latency in seconds",
+    labelnames=["method", "endpoint", "status_code"],
+    buckets=(0.1, 0.5, 1.0, 2.0, 5.0, 10.0),
+    registry=metrics_registry,
+)
+
+# Request counter by endpoint, method, and status
+request_count = Counter(
+    name="http_requests_total",
+    documentation="Total HTTP requests",
+    labelnames=["method", "endpoint", "status_code"],
+    registry=metrics_registry,
+)
+
+# Active requests gauge
+active_requests = Gauge(
+    name="http_requests_active",
+    documentation="Number of active HTTP requests",
+    labelnames=["method", "endpoint"],
+    registry=metrics_registry,
+)
+
+
+class MetricsMiddleware(BaseHTTPMiddleware):
+    """
+    Middleware to collect Prometheus metrics for HTTP requests.
+
+    Tracks request latency, count, and active requests for all endpoints
+    except the /metrics endpoint itself to avoid metric pollution.
+    """
+
+    def __init__(self, app: ASGIApp):
+        super().__init__(app)
+
+    async def dispatch(self, request: Request, call_next: Callable) -> Response:
+        # Skip metrics collection for the metrics endpoint itself
+        if request.url.path == "/metrics":
+            return await call_next(request)
+
+        # Extract endpoint path (use route path if available, otherwise URL path)
+        endpoint = request.url.path
+        method = request.method
+
+        # Track active requests
+        active_requests.labels(method=method, endpoint=endpoint).inc()
+
+        # Start timing
+        start_time = time.time()
+
+        try:
+            # Process the request
+            response = await call_next(request)
+            status_code = response.status_code
+        except Exception as e:
+            # Track errors as 500
+            status_code = 500
+            raise
+        finally:
+            # Calculate latency
+            latency = time.time() - start_time
+
+            # Record metrics
+            request_latency.labels(
+                method=method, endpoint=endpoint, status_code=status_code
+            ).observe(latency)
+
+            request_count.labels(
+                method=method, endpoint=endpoint, status_code=status_code
+            ).inc()
+
+            # Decrement active requests
+            active_requests.labels(method=method, endpoint=endpoint).dec()
+
+        return response
+
+
+def get_metrics() -> bytes:
+    """
+    Generate Prometheus metrics in the exposition format.
+
+    Returns:
+        bytes: Metrics in Prometheus text format
+    """
+    return generate_latest(metrics_registry)