Observability: structured logging, metrics dashboard, admin page

[zjy4fun]

Goal

Add production-grade observability so you can see what the system is actually doing — latency, token consumption, error rates, model selection patterns.

Current State

logger.ts does console.log(JSON.stringify(...)) — no persistence, no querying, no visualization.

Tasks

Structured Logging

• Replace console.log with pino (fast, structured, JSON by default)
• Log levels: debug/info/warn/error with consistent fields
• Request-scoped context: trace_id, user_id, session_id attached to every log line
• Log rotation: configure pino-roll or external logrotate

Metrics Storage

• Create metrics table in SQLite: timestamp, event_type, user_id, model, provider, latency_ms, prompt_tokens, completion_tokens, status (success/error), error_code
• Write metrics on every chat request completion (both success and failure)
• Aggregate views: per-hour/per-day rollups for dashboard queries

Admin Dashboard

• GET /admin/metrics/summary — today's stats: total requests, avg latency, total tokens, error rate
• GET /admin/metrics/models — per-model breakdown: usage count, avg latency, total tokens, cost estimate
• GET /admin/metrics/timeline — time-series data for charts (requests/hour, latency/hour)
• GET /admin/metrics/errors — recent errors with trace_id for debugging
• Frontend: simple admin page with charts (use recharts or plain SVG)
  • Request volume over time
  • Latency distribution (p50/p95/p99)
  • Token usage by model
  • Error rate trend
  • Top users by usage

Alerting (stretch)

• Detect anomalies: error rate spike, latency p95 above threshold
• Simple webhook notification (Discord/Slack/Telegram)

Why

A system without observability is a black box. This is what separates "I built a chat app" from "I built and operated a chat service." Every production system needs this. Also, building dashboards with real data is inherently satisfying.

Learning Points

• Structured logging best practices (correlation IDs, log levels)
• Metrics aggregation: counters, histograms, percentiles
• Time-series data visualization
• The difference between logs, metrics, and traces