Claims Monitoring: Eventual To‑Dos (Alerting, Notifications, Dashboards)

[rmusser01]

Context

Stage 3 analytics work is complete. Remaining items are marked partial/out-of-scope in:

• Docs/Product/Completed/Claims_Monitoring_PRD.md

Eventual To‑Dos

• Add automated alert evaluation (scheduler/Alertmanager integration).
• Create alert rules/runbooks for:
  • Unsupported ratio threshold breaches
  • Rebuild heartbeat missing/stale
  • Provider error/latency anomalies
• Expand notification channels beyond webhooks (Slack/email routing).
• Publish Grafana dashboards + panels for:
  • Provider performance
  • Unsupported ratio trends
  • Rebuild queue health
  • Data quality overview
• Add hotspot index/advanced analytics panels.

Notes

• Keep existing API response shapes stable (dashboard/export already in place).
• Prefer minimal operational overhead; integrate with existing metrics/monitoring stack.

Done When

• Alerting is automated and documented with runbooks.
• Dashboards are published and referenced in docs.
• Notification routing supports Slack/email and is configurable.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• Tracker: Running list #599

👤 Suggested Assignees

• rmusser01

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[coderabbitai]

Coding Plan

Approach

Implement automated claims monitoring by adding Grafana-managed alert rules with Alertmanager integration, expanding notification channels to support configurable Slack/email routing with environment-based configuration, and publishing comprehensive Grafana dashboards for provider performance, unsupported ratio trends, rebuild queue health, and data quality analytics.

Observations

The codebase has a mature monitoring infrastructure with Prometheus scraping /metrics, Grafana for visualization with auto-provisioned dashboards and alerts, and a dual metrics system (custom MetricsRegistry plus native prometheus_client). Claims monitoring already has foundational components: claims_alerts_scheduler.py for periodic evaluation, claims_notifications.py supporting Slack/webhook/email channels, and claims_service.py computing unsupported ratios. Grafana alert rules follow YAML patterns in Docs/Deployment/Monitoring/Alerts/, dashboards are stored in Docs/Monitoring/Grafana_Dashboards/, and Alertmanager example config exists at Docs/Operations/monitoring/alertmanager_example.yml. The notification infrastructure supports multiple channels with environment-based configuration.

Assumptions

Assumption 1: Alerting strategy: Grafana Managed Alerts vs Alertmanager-only

Options Considered:

• Use Alertmanager exclusively for all alerting and routing
• Use Grafana managed alerts for Prometheus-based alerts with Alertmanager for notification routing

Chosen Option: 2

Rationale: Grafana managed alerts are already the established pattern (see app-alerts.yml, mcp-alerts.yml, rag-alerts.yml), provide better visibility in the Grafana UI, and integrate cleanly with the existing provisioning setup while still leveraging Alertmanager for external notification routing.

Assumption 2: Notification channel integration approach

Options Considered:

• Extend database-driven per-user configuration to support all channels
• Add environment-based global notification routing configuration alongside existing per-user config
• Rely entirely on Alertmanager for notification routing

Chosen Option: 2

Rationale: Environment-based global configuration aligns with the pattern used by other notification systems (MONITORING_NOTIFY_*, SECURITY_ALERT_*) and allows ops teams to configure routing without database changes, while preserving per-user overrides for specific use cases.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Alert Rules & Metrics Exposure

This phase establishes the Prometheus metrics and Grafana alert rules needed for automated claims monitoring. These tasks are grouped because alerts depend on metrics being properly exposed, and both configure the same monitoring subsystem.

Task 1: Ensure Claims Monitoring Metrics Are Properly Exposed

Verify and extend the claims metrics exposed to Prometheus so that alert rules can query them reliably.

• Review tldw_Server_API/app/core/Claims_Extraction/claims_service.py to confirm _compute_unsupported_ratios() publishes metrics to the registry
• Add missing metrics to the metrics registry if not present: claims_unsupported_ratio (gauge), claims_rebuild_queue_size (gauge), claims_rebuild_last_success_timestamp (gauge), claims_provider_request_duration_seconds (histogram), claims_provider_errors_total (counter)
• Register metrics following the pattern in tldw_Server_API/app/core/Metrics/metrics_manager.py using MetricDefinition with appropriate labels (e.g., provider, tenant)
• Ensure the claims rebuild worker in main.py updates heartbeat/queue metrics on each cycle
• Verify metrics appear at /metrics endpoint with expected labels

Task 2: Create Grafana Alert Rules for Claims Monitoring

Define alert rules in Grafana YAML format for the three alert categories specified in the ticket.

• Create Docs/Deployment/Monitoring/Alerts/claims-alerts.yml following the structure of existing alert files (rag-alerts.yml, mcp-alerts.yml)
• Add alert rule for unsupported ratio threshold: trigger when claims_unsupported_ratio > 0.3 sustained for 10 minutes, severity warning; critical at 0.5
• Add alert rule for rebuild heartbeat: trigger when claims_rebuild_last_success_timestamp is older than 2x the configured interval, severity critical
• Add alert rule for provider errors: trigger when rate(claims_provider_errors_total[5m]) > 1, severity warning
• Add alert rule for provider latency: trigger when p95 of claims_provider_request_duration_seconds exceeds 5s for 5 minutes
• Include appropriate annotations (summary, description, runbook_url) referencing the runbook created in Phase 3
• Set folder to TLDW and evaluation interval to 1m matching existing patterns

Task 3: Configure Alertmanager Routing for Claims Alerts

Extend Alertmanager configuration to route claims alerts to appropriate notification channels.

• Update Docs/Operations/monitoring/alertmanager_example.yml to add a claims-alerts route matching alerts with label alertgroup: claims
• Add receiver configuration for claims alerts supporting Slack webhook and email channels
• Configure severity-based routing: critical alerts to both Slack and PagerDuty, warning alerts to Slack only
• Add grouping configuration to batch related claims alerts (group by alertname, tenant)
• Document the environment variables needed to activate this routing (ALERTMANAGER_SLACK_WEBHOOK_URL, ALERTMANAGER_EMAIL_TO)

🤖 Prompt for AI agents

Set up Prometheus metrics exposure and Grafana alert rules for automated claims
monitoring.

Ensure claims metrics are properly exposed:
- Review claims_service.py to confirm _compute_unsupported_ratios() publishes
metrics
- Add missing metrics: claims_unsupported_ratio (gauge),
claims_rebuild_queue_size (gauge), claims_rebuild_last_success_timestamp
(gauge), claims_provider_request_duration_seconds (histogram),
claims_provider_errors_total (counter)
- Register metrics in metrics_manager.py using MetricDefinition with labels
(provider, tenant)
- Update claims rebuild worker in main.py to update heartbeat/queue metrics
- Verify metrics at /metrics endpoint

Create Grafana alert rules:
- Create claims-alerts.yml in Docs/Deployment/Monitoring/Alerts/ following
existing patterns
- Add alert rules for: unsupported ratio threshold (warning at 0.3, critical at
0.5 for 10min), rebuild heartbeat (critical when timestamp older than 2x
interval), provider errors (warning when rate > 1/5min), provider latency (p95 >
5s for 5min)
- Include annotations (summary, description, runbook_url)
- Set folder to TLDW, evaluation interval to 1m

Configure Alertmanager routing:
- Update alertmanager_example.yml with claims-alerts route matching alertgroup:
claims
- Add receiver for Slack webhook and email channels
- Configure severity-based routing (critical to Slack + PagerDuty, warning to
Slack only)
- Add grouping by alertname and tenant
- Document required environment variables

Phase 2: Notification Channel Configuration

This phase enhances the claims notification system to support configurable Slack/email routing at the system level, complementing the alert-based notifications with application-level notification capabilities.

Task 1: Add Global Notification Channel Configuration

Introduce environment-based configuration for claims alert notification routing that operators can configure without database changes.

• Add environment variables following existing patterns: CLAIMS_ALERTS_SLACK_WEBHOOK_URL, CLAIMS_ALERTS_SLACK_CHANNEL, CLAIMS_ALERTS_EMAIL_RECIPIENTS, CLAIMS_ALERTS_EMAIL_ENABLED, CLAIMS_ALERTS_SLACK_ENABLED
• Create configuration loading in tldw_Server_API/app/core/Claims_Extraction/ that reads these variables with sensible defaults (disabled if not configured)
• Add configuration validation on startup to warn if notification channels are enabled but URLs/recipients are missing
• Document new environment variables in Env_Vars.md under a Claims Monitoring section

Task 2: Enhance Claims Notification Routing Logic

Update the claims notification service to use the global configuration alongside per-user overrides.

• Modify tldw_Server_API/app/core/Claims_Extraction/claims_notifications.py to check global config when per-user config is not set
• Implement channel selection logic: use per-user config if available, fall back to global config, respect enabled/disabled flags
• Add Slack message formatting for claims alerts with severity indicators, metric values, and runbook links
• Ensure email notifications use the existing EmailService from AuthNZ with HTML templates for claims alert digests
• Add delivery status logging and metrics (claims_notifications_sent_total{channel, status})

Task 3: Update Claims Alerts Scheduler Integration

Ensure the existing scheduler properly leverages the enhanced notification routing.

• Review tldw_Server_API/app/services/claims_alerts_scheduler.py to confirm it calls the notification service
• Update scheduler to pass alert context (severity, metric values, threshold breached) to notification functions
• Ensure scheduler respects the new global notification configuration
• Add logging for notification delivery attempts and outcomes

🤖 Prompt for AI agents

Enhance claims notification system with configurable Slack/email routing at
system level.

Add global notification configuration:
- Add environment variables: CLAIMS_ALERTS_SLACK_WEBHOOK_URL,
CLAIMS_ALERTS_SLACK_CHANNEL, CLAIMS_ALERTS_EMAIL_RECIPIENTS,
CLAIMS_ALERTS_EMAIL_ENABLED, CLAIMS_ALERTS_SLACK_ENABLED
- Create configuration loading in Claims_Extraction/ that reads variables with
sensible defaults
- Add startup validation to warn if channels enabled but URLs/recipients missing
- Document environment variables in Env_Vars.md under Claims Monitoring section

Enhance notification routing logic:
- Modify claims_notifications.py to check global config as fallback when
per-user config not set
- Implement channel selection logic prioritizing per-user config, falling back
to global config
- Add Slack message formatting with severity indicators, metric values, and
runbook links
- Use existing EmailService from AuthNZ with HTML templates for claims alert
digests
- Add delivery status logging and metrics (claims_notifications_sent_total)

Update scheduler integration:
- Review claims_alerts_scheduler.py for notification service calls
- Update scheduler to pass alert context (severity, metric values, threshold
breached) to notification functions
- Ensure scheduler respects global notification configuration
- Add logging for notification delivery attempts and outcomes

Phase 3: Dashboards & Documentation

This phase creates the visual monitoring dashboards and operational documentation needed for day-to-day claims monitoring operations.

Task 1: Create Claims Monitoring Grafana Dashboards

Build comprehensive Grafana dashboard JSON files for claims monitoring visualization.

• Create Docs/Monitoring/Grafana_Dashboards/claims-monitoring.json as the main dashboard
• Add Provider Performance panel: table showing request counts, error rates, and p50/p95 latencies per provider using claims_provider_request_duration_seconds histogram
• Add Unsupported Ratio Trends panel: time series graph of claims_unsupported_ratio with threshold annotation lines at warning (0.3) and critical (0.5) levels
• Add Rebuild Queue Health panel: stat panels for queue size, last successful rebuild timestamp, and rebuild rate; include gauge showing time since last heartbeat
• Add Data Quality Overview panel: pie chart or bar chart showing claims by verification status (supported/refuted/NEI)
• Add Hotspot Index panel: heatmap or table identifying media items or tenants with highest unsupported ratios
• Configure dashboard variables for tenant filtering and time range selection
• Set appropriate refresh interval (30s) and default time range (last 6 hours)

Task 2: Create Claims Alert Runbooks

Document operational procedures for responding to each claims monitoring alert.

• Create or update Docs/Operations/Claims_Alerts_Runbook.md with structured runbook format
• Add runbook section for "High Unsupported Ratio": describe what the metric means, investigation steps (check recent ingestion, provider status, content quality), remediation options (pause ingestion, switch providers, manual review queue)
• Add runbook section for "Rebuild Heartbeat Stale": describe worker health checks, common causes (resource exhaustion, database locks, configuration errors), restart procedures
• Add runbook section for "Provider Errors/Latency": describe provider health verification, fallback procedures, escalation paths to provider support
• Include links to relevant dashboards, log queries, and contact information for escalation
• Add severity response time expectations (critical: 15 min, warning: 4 hours)

Task 3: Update Documentation References

Ensure all new monitoring capabilities are discoverable in the documentation.

• Update Docs/Deployment/Monitoring/README.md to reference claims-monitoring dashboard and claims-alerts.yml
• Add claims monitoring section to Docs/Deployment/Monitoring/Metrics_Cheatsheet.md listing all new metrics with descriptions and example queries
• Update Docs/Operations/ index or README to reference the claims alerts runbook
• Add setup instructions for claims alerting to any deployment guides, noting required environment variables
• Create or update Docs/Product/Completed/Claims_Monitoring_PRD.md to mark alerting/dashboard items as complete (if file exists) or document completion status elsewhere

🤖 Prompt for AI agents

Create Grafana dashboards and operational documentation for claims monitoring.

Create Grafana dashboards:
- Create claims-monitoring.json in Docs/Monitoring/Grafana_Dashboards/
- Add panels: Provider Performance (table with request counts, error rates,
p50/p95 latencies), Unsupported Ratio Trends (time series with threshold
annotations at 0.3 and 0.5), Rebuild Queue Health (stat panels for queue size,
last success timestamp, rebuild rate, time since last heartbeat), Data Quality
Overview (pie/bar chart by verification status), Hotspot Index (heatmap/table of
highest unsupported ratios by media/tenant)
- Configure dashboard variables for tenant filtering and time range
- Set refresh interval to 30s, default time range to last 6 hours

Create alert runbooks:
- Create or update Claims_Alerts_Runbook.md in Docs/Operations/
- Add runbook sections for: High Unsupported Ratio (metric explanation,
investigation steps, remediation options), Rebuild Heartbeat Stale (health
checks, common causes, restart procedures), Provider Errors/Latency (health
verification, fallback procedures, escalation paths)
- Include dashboard links, log queries, contact information
- Add severity response times (critical: 15 min, warning: 4 hours)

Update documentation references:
- Update Docs/Deployment/Monitoring/README.md with claims-monitoring dashboard
and claims-alerts.yml
- Add claims monitoring section to Metrics_Cheatsheet.md with metrics
descriptions and example queries
- Update Docs/Operations/ index to reference claims alerts runbook
- Add setup instructions with required environment variables to deployment
guides
- Update Claims_Monitoring_PRD.md to mark alerting/dashboard items complete

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Set up Prometheus metrics exposure and Grafana alert rules for automated claims
monitoring.

Ensure claims metrics are properly exposed:
- Review claims_service.py to confirm _compute_unsupported_ratios() publishes
metrics
- Add missing metrics: claims_unsupported_ratio (gauge),
claims_rebuild_queue_size (gauge), claims_rebuild_last_success_timestamp
(gauge), claims_provider_request_duration_seconds (histogram),
claims_provider_errors_total (counter)
- Register metrics in metrics_manager.py using MetricDefinition with labels
(provider, tenant)
- Update claims rebuild worker in main.py to update heartbeat/queue metrics
- Verify metrics at /metrics endpoint

Create Grafana alert rules:
- Create claims-alerts.yml in Docs/Deployment/Monitoring/Alerts/ following
existing patterns
- Add alert rules for: unsupported ratio threshold (warning at 0.3, critical at
0.5 for 10min), rebuild heartbeat (critical when timestamp older than 2x
interval), provider errors (warning when rate > 1/5min), provider latency (p95 >
5s for 5min)
- Include annotations (summary, description, runbook_url)
- Set folder to TLDW, evaluation interval to 1m

Configure Alertmanager routing:
- Update alertmanager_example.yml with claims-alerts route matching alertgroup:
claims
- Add receiver for Slack webhook and email channels
- Configure severity-based routing (critical to Slack + PagerDuty, warning to
Slack only)
- Add grouping by alertname and tenant
- Document required environment variables
===============================================================================

Task: 2

Enhance claims notification system with configurable Slack/email routing at
system level.

Add global notification configuration:
- Add environment variables: CLAIMS_ALERTS_SLACK_WEBHOOK_URL,
CLAIMS_ALERTS_SLACK_CHANNEL, CLAIMS_ALERTS_EMAIL_RECIPIENTS,
CLAIMS_ALERTS_EMAIL_ENABLED, CLAIMS_ALERTS_SLACK_ENABLED
- Create configuration loading in Claims_Extraction/ that reads variables with
sensible defaults
- Add startup validation to warn if channels enabled but URLs/recipients missing
- Document environment variables in Env_Vars.md under Claims Monitoring section

Enhance notification routing logic:
- Modify claims_notifications.py to check global config as fallback when
per-user config not set
- Implement channel selection logic prioritizing per-user config, falling back
to global config
- Add Slack message formatting with severity indicators, metric values, and
runbook links
- Use existing EmailService from AuthNZ with HTML templates for claims alert
digests
- Add delivery status logging and metrics (claims_notifications_sent_total)

Update scheduler integration:
- Review claims_alerts_scheduler.py for notification service calls
- Update scheduler to pass alert context (severity, metric values, threshold
breached) to notification functions
- Ensure scheduler respects global notification configuration
- Add logging for notification delivery attempts and outcomes
===============================================================================

Task: 3

Create Grafana dashboards and operational documentation for claims monitoring.

Create Grafana dashboards:
- Create claims-monitoring.json in Docs/Monitoring/Grafana_Dashboards/
- Add panels: Provider Performance (table with request counts, error rates,
p50/p95 latencies), Unsupported Ratio Trends (time series with threshold
annotations at 0.3 and 0.5), Rebuild Queue Health (stat panels for queue size,
last success timestamp, rebuild rate, time since last heartbeat), Data Quality
Overview (pie/bar chart by verification status), Hotspot Index (heatmap/table of
highest unsupported ratios by media/tenant)
- Configure dashboard variables for tenant filtering and time range
- Set refresh interval to 30s, default time range to last 6 hours

Create alert runbooks:
- Create or update Claims_Alerts_Runbook.md in Docs/Operations/
- Add runbook sections for: High Unsupported Ratio (metric explanation,
investigation steps, remediation options), Rebuild Heartbeat Stale (health
checks, common causes, restart procedures), Provider Errors/Latency (health
verification, fallback procedures, escalation paths)
- Include dashboard links, log queries, contact information
- Add severity response times (critical: 15 min, warning: 4 hours)

Update documentation references:
- Update Docs/Deployment/Monitoring/README.md with claims-monitoring dashboard
and claims-alerts.yml
- Add claims monitoring section to Metrics_Cheatsheet.md with metrics
descriptions and example queries
- Update Docs/Operations/ index to reference claims alerts runbook
- Add setup instructions with required environment variables to deployment
guides
- Update Claims_Monitoring_PRD.md to mark alerting/dashboard items complete

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For assumption 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!