[Incident c3c60628-5a06-4699-9304-ee39a758f000] test-alert fired on sredemo-web in region: unknown

[TimoSalomaki]

Incident overview

• Incident ID: c3c60628-5a06-4699-9304-ee39a758f000
• First seen (UTC): 2025-11-14T10:24:03.726Z
• Detection source: Azure Monitor metric alert (rule: test-alert)
• Severity: Sev1
• Impacted resource/app: sredemo-web (App Service)
• Customer impact: Unknown from alert payload (please confirm affected endpoints/users)

Environment

• Subscription: 7ef48e25-fddf-4dc3-b10e-8b73bd7d5faf
• Resource group: rg-sredemo
• Resource ID: /subscriptions/7ef48e25-fddf-4dc3-b10e-8b73bd7d5faf/resourceGroups/rg-sredemo/providers/Microsoft.Web/sites/sredemo-web
• Region: Unknown (not present in alert payload)
• Service tier/SKU: Not provided in alert payload

Symptoms / errors

• Alert "test-alert" fired at 2025-11-14T10:24:03.726Z. The alert description field was empty; exact symptom/threshold not included in the notification. No explicit error messages, correlation IDs, or request IDs provided by the alert.

Recent changes and activity (last 24h)

• No resource write/modify operations detected in Activity Logs for the app in the last 24h.
• Informational events only:
  • 2025-11-16T17:51:21Z Microsoft.Advisor/recommendations/available (caller: Microsoft.Advisor, correlationId: 967670c5-98d8-4ec7-a771-994f357d4d2c)
  • 2025-11-16T17:51:20Z Microsoft.Advisor/recommendations/available (caller: Microsoft.Advisor, correlationId: 2d4b16ed-b5e8-4624-9f68-f78d07120791)
• No deployments or configuration changes correlated with the incident window were found in the provided logs.

Metrics snapshot (last ~2 hours from 2025-11-16T17:53Z to 2025-11-16T19:53Z UTC)

• Requests (Microsoft.Web/sites: Requests): Very low traffic with occasional single requests per minute.
• Http5xx (Microsoft.Web/sites: Http5xx): 0 across the window; no server error spikes detected.
• HttpResponseTime (Microsoft.Web/sites: HttpResponseTime): Generally low average response times (~7–10 ms), with minor spikes up to ~29 ms around 18:50 UTC and ~22 ms around 18:25 UTC.
• MemoryWorkingSet (Microsoft.Web/sites: MemoryWorkingSet): Gradual increase over time, with notable rises around 18:45–19:06 UTC (peaking ~845–847 MB), then tapering to ~773–787 MB by ~19:10–19:52 UTC. A brief higher plateau near 18:45–19:06 may indicate transient memory pressure.

Reproduction steps

• Not available. Traffic is minimal in the examined window. Please confirm if any endpoints or specific scenarios can reproduce the alert condition.

Attempted mitigations

• None applied; investigation focused on validation, activity logs, and metrics collection.

Proposed next steps for engineering

1. Clarify the exact alert rule criteria for "test-alert" (metric, dimension, operator, threshold, evaluation window) to align investigation with the intended symptom.
2. Confirm current app plan SKU and instance count; check for recent deployments outside the 24h Activity Logs scope (CD pipelines, slot swaps, config changes).
3. Review application logs and APM traces around 2025-11-14T10:24Z (alert fire time) for errors, exceptions, or timeouts not visible in platform metrics.
4. Investigate the transient MemoryWorkingSet rise (18:45–19:06 UTC on 2025-11-16) for potential leaks or caching growth; verify GC behavior and heap allocations if .NET/Java.
5. Validate health checks, dependencies (DB, cache, downstream APIs), and connection pool utilization during the alert timeframe.
6. Provide reproduction steps if known; otherwise, consider canary or synthetic traffic to exercise critical endpoints.

References / dashboards

• Azure Resource: /subscriptions/7ef48e25-fddf-4dc3-b10e-8b73bd7d5faf/resourceGroups/rg-sredemo/providers/Microsoft.Web/sites/sredemo-web
• Alert ID: /subscriptions/7ef48e25-fddf-4dc3-b10e-8b73bd7d5faf/resourcegroups/rg-sredemo/providers/microsoft.web/sites/sredemo-web/providers/Microsoft.AlertsManagement/alerts/c3c60628-5a06-4699-9304-ee39a758f000

Ownership / requests

• Please acknowledge receipt, confirm owner/assignee, and share ETA for triage. If additional metrics or logs are needed, specify time ranges and data sources. If the wrong repository, please provide the correct repo URL for ownership mapping.

---

This issue was created by sredemo--062602c3
Tracked by the SRE agent here