Alerting rules

[jeffersonBastos]

Summary

Implements Prometheus alerting rules for the CoW Performance Testing Suite (COW-598). Adds 7 core alerts that notify developers when performance degrades, error rates spike, or resource utilization exceeds thresholds during performance testing.

Approach: Option A (Prometheus alerting rules + Grafana visualization) - no Alertmanager required.

Alerts Implemented

Alert	Severity	Trigger
HighSubmissionLatency	Warning	P95 > 5s for 2m
CriticalSubmissionLatency	Critical	P95 > 10s for 1m
HighErrorRate	Critical	Error rate > 5% for 1m
LowThroughput	Warning	Actual < 80% target for 2m
TestStalled	Critical	No orders for 1m during active test
HighCPUUsage	Warning	CPU > 80% for 5m
CriticalMemoryUsage	Critical	Memory > 95% for 2m

Changes

New Files

• configs/prometheus/alerts/performance-testing.yml - Alert rules with documented parameters at top for easy customization (TODO(COW-617) for future configurability)

Modified Files

• configs/prometheus.yml - Enable rule_files: to load alert rules
• docker-compose.yml - Mount alerts directory in Prometheus container
• configs/dashboards/performance.json - Add alert annotations to show firing alerts
• src/cow_performance/prometheus/metrics.py - Add cow_perf_container_memory_percent gauge
• src/cow_performance/prometheus/exporter.py - Export memory percentage metric

Documentation

• thoughts/plans/2026-02-13-cow-598-alerting-rules.md - Implementation plan
• thoughts/tickets/COW-598-alerting-rules.md - Updated with implementation notes
• thoughts/INDEX.md - Updated with plan reference

How to Test

1. Start the monitoring stack:

   docker compose --profile monitoring up -d

2. Verify alert rules loaded in Prometheus:

   curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].name'
   # Expected: "cow_performance_testing"

3. Check the Prometheus alerts page:

   open http://localhost:9090/alerts

4. Run a test to generate metrics and verify Grafana annotations:

   cow-perf run --prometheus-port 9091 --duration 120
   open http://localhost:3000  # Navigate to Performance Overview dashboard

Checklist

• Tests pass (poetry run pytest tests/unit/)
• Linting passes (poetry run ruff check src/cow_performance/prometheus/)
• Type checking passes (poetry run mypy src/cow_performance/prometheus/)
• YAML config files valid
• JSON dashboard files valid
• Docker Compose config valid
• Manual: Prometheus loads alert rules (requires Docker services)
• Manual: Grafana shows alert annotations (requires Docker services)

Scope Decisions

Reduced scope (7 alerts instead of 15+) based on:

• Grant proposal only specifies "alerting rules" without detail
• Core alerts cover the most critical conditions
• Alertmanager not required (no Slack/email notifications)
• Thresholds hardcoded but organized for easy modification

Deferred to COW-617: Configurable thresholds via TOML/env variables

Breaking Changes

None

Related Issues

• Implements: COW-598
• Related: COW-617 (threshold configuration - future work)
• Depends on: COW-591 (Prometheus Exporters), COW-593 (Grafana Dashboards)

---

🤖 Generated with Claude Code

[linear]

COW-598 13: Alerting Rules