You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: Troubleshoot log search alerts in Azure Monitor | Microsoft Docs
3
-
description: Common issues, errors, and resolutions for log search alert rules in Azure.
2
+
title: Troubleshoot log alerts in Azure Monitor | Microsoft Docs
3
+
description: Common issues, errors, and resolutions for log alert rules in Azure.
4
4
ms.author: abbyweisberg
5
5
ms.topic: conceptual
6
-
ms.custom: build-2023
7
-
ms.date: 02/13/2024
8
-
ms.reviewer: yalavi
6
+
ms.date: 02/28/2024
7
+
ms.reviewer: nolavime
9
8
---
10
9
11
10
# Troubleshoot log search alerts in Azure Monitor
12
11
13
-
This article describes how to resolve common issues with log search alerts in Azure Monitor. It also provides solutions to common problems with the functionality and configuration of log search alerts.
12
+
This article describes how to resolve common issues with log search alerts in Azure Monitor. It also provides solutions to common problems with the functionality and configuration of log alerts.
14
13
15
-
You can use log search alerts to evaluate resources logs every set frequency by using a [Log Analytics](../logs/log-analytics-tutorial.md) query, and fire an alert that's based on the results. Rules can trigger one or more actions using [Action Groups](./action-groups.md). To learn more about functionality and terminology of log search alerts, see [Log search alerts in Azure Monitor](alerts-types.md#log-alerts).
14
+
You can use log alerts to evaluate resources logs every set frequency by using a [Log Analytics](../logs/log-analytics-tutorial.md) query, and fire an alert that's based on the results. Rules can trigger one or more actions using [Action Groups](./action-groups.md). To learn more about functionality and terminology of log search alerts, see [Log alerts in Azure Monitor](alerts-types.md#log-alerts).
16
15
17
16
> [!NOTE]
18
17
> This article doesn't consider cases where the Azure portal shows that an alert rule was triggered but a notification isn't received. For such cases, see [Action or notification on my alert did not work as expected](./alerts-troubleshoot.md#action-or-notification-on-my-alert-did-not-work-as-expected).
@@ -31,7 +30,6 @@ View the health status of your log search alert rule:
31
30
:::image type="content" source="media/log-search-alert-health/log-search-alert-resource-health.png" alt-text="Screenshot of the Resource health section in a log search alert rule.":::
32
31
33
32
See [Monitor the health of log search alert rules](log-alert-rule-health.md#monitor-the-health-of-log-search-alert-rules) to learn more.
34
-
## Log search alert didn't fire
35
33
36
34
### Data ingestion time for logs
37
35
@@ -43,119 +41,32 @@ To mitigate latency, the system retries the alert evaluation multiple times. Aft
43
41
44
42
### Actions are muted or alert rule is defined to resolve automatically
45
43
46
-
Log search alerts provide an option to mute fired alert actions for a set amount of time using **Mute actions** and to only fire once per condition being met using **Automatically resolve alerts**.
44
+
Log alerts provide an option to mute fired alert actions for a set amount of time using **Mute actions** and to only fire once per condition being met using **Automatically resolve alerts**.
47
45
48
46
A common issue is that you think that the alert didn't fire, but it was actually the rule configuration.
### Alert scope resource has been moved, renamed, or deleted
54
51
55
-
When you author an alert rule, Log Analytics creates a permission snapshot for your user ID. This snapshot is saved in the rule and contains the rule scope resource, Azure Resource Manager ID. If the rule scope resource moves, gets renamed, or is deleted, all log search alert rules that refer to that resource will break. To work correctly, alert rules need to be recreated using the new Azure Resource Manager ID.
56
-
57
-
### The alert rule uses a system-assigned managed identity
58
-
59
-
When you create a log search alert rule with system-assigned managed identity, the identity is created without any permissions. After you create the rule, you need to assign the appropriate roles to the rule’s identity so that it can access the data you want to query. For example, you might need to give it a Reader role for the relevant Log Analytics workspaces, or a Reader role and a Database Viewer role for the relevant ADX cluster. See [managed identities](/azure/azure-monitor/alerts/alerts-create-log-alert-rule#configure-the-alert-rule-details) for more information about using managed identities in log search alerts.
60
-
61
-
### Metric measurement alert rule with splitting using the legacy Log Analytics API
62
-
63
-
[Metric measurement](alerts-types.md#log-alerts) is a type of log search alert that's based on summarized time series results. You can use these rules to group by columns to [split alerts](alerts-types.md#monitor-the-same-condition-on-multiple-resources-using-splitting-by-dimensions-1). If you're using the legacy Log Analytics API, splitting doesn't work as expected because it doesn't support grouping.
64
-
65
-
You can use the current ScheduledQueryRules API to set **Aggregate On** in [Metric measurement](alerts-types.md#log-alerts) rules, which work as expected. To learn more about switching to the current ScheduledQueryRules API, see [Upgrade to the current Log Alerts API from legacy Log Analytics Alert API](./alerts-log-api-switch.md).
66
-
67
-
### Override query time range
68
-
69
-
As a part of the configuration of the alert, in the section of the "Advance Options", there is an option to configure "Override query time range" parameter.
70
-
If you want the alert evaluation period to be different than the query time range, enter a time range here.
71
-
The alert time range is limited to a maximum of two days. Even if the query contains an ago command with a time range of longer than two days, the two-day maximum time range is applied. For example, even if the query text contains ago(7d), the query only scans up to two days of data.
72
-
If the query requires more data than the alert evaluation, you can change the time range manually.
73
-
If there's ago command in the query, it will be changed automatically to be 2 days (48 hours).
74
-
75
-
:::image type="content" source="media/alerts-troubleshoot-log/alerts-rule-preview-advanced-options.png" lightbox="media/alerts-troubleshoot-log/alerts-rule-preview-advanced-options.png" alt-text="Screenshot of advanced settings for log search alerts.":::
76
-
77
-
## Log search alert fired unnecessarily
78
-
79
-
A configured [log search alert rule in Azure Monitor](./alerts-log.md) might be triggered unexpectedly. The following sections describe some common reasons.
52
+
A configured [log alert rule in Azure Monitor](./alerts-log.md) might be triggered unexpectedly. The following sections describe some common reasons.
80
53
81
54
### Alert triggered by partial data
82
55
83
56
Azure Monitor processes terabytes of customers' logs from across the world, which can cause [logs ingestion latency](../logs/data-ingestion-time.md).
84
57
85
58
Logs are semi-structured data and are inherently more latent than metrics. If you're experiencing many misfires in fired alerts, you should consider using [metric alerts](alerts-metric-overview.md). You can send data to the metric store from logs using [metric alerts for logs](alerts-metric-logs.md).
86
59
87
-
Log search alerts work best when you try to detect data in the logs. It works less well when you try to detect lack of data in the logs, like alerting on virtual machine heartbeat.
60
+
Log alerts work best when you try to detect data in the logs. It works less well when you try to detect lack of data in the logs, like alerting on virtual machine heartbeat.
88
61
89
62
There are built-in capabilities to prevent false alerts, but they can still occur on very latent data (over ~30 minutes) and data with latency spikes.
90
63
91
64
## Log search alert rule was disabled
92
65
93
66
If a log search alert rule query fails to evaluate continuously for a week, Azure Monitor disables it automatically.
94
67
The following sections list some reasons why Azure Monitor might disable a log search alert rule. Additionally, there's an example of the [Activity log](../../azure-monitor/essentials/activity-log.md) event that is submitted when a rule is disabled.
95
-
## Log search alert was disabled
96
-
97
-
The following sections list some reasons why Azure Monitor might disable a log search alert rule. After those sections, there's an [example of the activity log that is sent when a rule is disabled](#activity-log-example-when-rule-is-disabled).
98
-
99
-
### Alert scope no longer exists or was moved
100
-
101
-
When the scope resources of an alert rule are no longer valid, rule execution fails, and billing stops.
102
-
103
-
If a log search alert fails continuously for a week, Azure Monitor disables it.
104
-
105
-
### <aname="query-used-in-a-log-alert-isnt-valid"></a>Query used in a log search alert isn't valid
106
-
107
-
When a log search alert rule is created, the query is validated for correct syntax. But sometimes, the query provided in the log search alert rule can start to fail. Some common reasons are:
108
-
109
-
- Rules were created via the API, and validation was skipped by the user.
110
-
- The query [runs on multiple resources](../logs/cross-workspace-query.md), and one or more of the resources was deleted or moved.
111
-
- The [query fails](../logs/api/errors.md) because:
112
-
- The logging solution wasn't [deployed to the workspace](../insights/solutions.md#install-a-monitoring-solution), so tables aren't created.
113
-
- Data stopped flowing to a table in the query for more than 30 days.
114
-
-[Custom logs tables](../agents/data-sources-custom-logs.md) aren't yet created, because the data flow hasn't started.
115
-
- Changes in [query language](/azure/kusto/query/) include a revised format for commands and functions, so the query provided earlier is no longer valid.
116
-
117
-
[Azure Advisor](../../advisor/advisor-overview.md) warns you about this behavior. It adds a recommendation about the affected log search alert rule. The category used is 'High Availability' with medium impact and a description of 'Repair your log search alert rule to ensure monitoring'.
118
-
119
-
## Alert rule quota was reached
120
-
121
-
For details about the number of log search alert rules per subscription and maximum limits of resources, see [Azure Monitor service limits](../service-limits.md).
122
-
123
-
### Recommended Steps
124
-
125
-
If you've reached the quota limit, the following steps might help resolve the issue.
126
-
127
-
1. Delete or disable log search alert rules that aren’t used anymore.
128
-
1. Use [splitting of alerts by dimensions](alerts-types.md#monitor-the-same-condition-on-multiple-resources-using-splitting-by-dimensions-1) to reduce rules count. These rules can monitor many resources and detection cases.
129
-
1. If you need the quota limit to be increased, continue to open a support request, and provide the following information:
130
-
131
-
- The Subscription IDs and Resource IDs for which the quota limit needs to be increased
132
-
- The reason for quota increase
133
-
- The resource type for the quota increase, such as **Log Analytics** or **Application Insights**
134
-
- The requested quota limit
135
-
136
-
### To check the current usage of new log search alert rules
137
-
138
-
#### From the Azure portal
139
-
140
-
1. On the Alerts screen in Azure Monitor, select **Alert rules**.
141
-
1. In the **Subscription** dropdown control, filter to the subscription you want. (Make sure you don't filter to a specific resource group, resource type, or resource.)
142
-
1. In the **Signal type** dropdown control, select **Log Search**.
143
-
1. Verify that the **Status** dropdown control is set to **Enabled**.
144
-
145
-
The total number of log search alert rules is displayed above the rules list.
- REST API - [List by subscription](/rest/api/monitor/scheduledqueryrule-2021-08-01/scheduled-query-rules/list-by-subscription)
153
-
154
-
## Activity log example when rule is disabled
155
-
156
-
If query fails for seven days continuously, Azure Monitor disables the log search alert and stops the billing of the rule. You can see the exact time when Azure Monitor disabled the log search alert in the [Azure activity log](../../azure-monitor/essentials/activity-log.md).
157
-
158
-
See this example:
159
70
160
71
```json
161
72
{
@@ -309,20 +220,8 @@ The total number of log search alert rules is displayed above the rules list.
- REST API - [List by subscription](/rest/api/monitor/scheduledqueryrule-2021-08-01/scheduled-query-rules/list-by-subscription)
311
222
312
-
## Query syntax validation error
313
-
314
-
If you get an error message that says "Couldn't validate the query syntax because the service can't be reached", it could be either:
315
-
- A query syntax error.
316
-
- A problem connecting to the service that validates the query.
317
-
318
-
Try the following steps to resolve the problem:
319
-
1. Try running the query in Azure Monitor Logs, and fix any syntax issues.
320
-
2. If your query syntax is valid, check the connection to the service.
321
-
- Flush the DNS cache on your local machine, by opening a command prompt and running the following command: `ipconfig /flushdns`, and then check again. If you still get the same error message, try the next step.
322
-
- Copy and paste this URL into the browser: [https://api.loganalytics.io/v1/version](https://api.loganalytics.io/v1/version). If you get an error, contact your IT administrator to allow the IP addresses associated with **api.loganalytics.io** listed [here](../ip-addresses.md#application-insights-and-log-analytics-apis).
323
-
324
223
## Next steps
325
224
326
-
- Learn about [log search alerts in Azure](./alerts-types.md#log-alerts).
327
-
- Learn more about [configuring log search alerts](../logs/log-query-overview.md).
225
+
- Learn about [log alerts in Azure](./alerts-unified-log.md).
226
+
- Learn more about [configuring log alerts](../logs/log-query-overview.md).
328
227
- Learn more about [log queries](../logs/log-query-overview.md).
0 commit comments