You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-monitor/alerts/alerts-troubleshoot-log.md
+41-38Lines changed: 41 additions & 38 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,40 +17,42 @@ You can use log alerts to evaluate resources logs every set frequency by using a
17
17
> This article doesn't discuss cases where the alert rule was triggered, you can see it in the Azure portal, but the notification was not sent. See [troubleshooting alerts](alerts-troubleshoot.md) for cases like these.
18
18
19
19
## A log search alert didn't fire when it should have
20
+
If your log search alert didn't fire when it should have, check the following items:
20
21
21
22
1.**Is the alert rule is in a degraded or unavailable health state?**
22
23
23
-
View the health status of your log search alert rule:
24
+
View the health status of your log search alert rule:
25
+
26
+
1. In the [portal](https://portal.azure.com/), select **Monitor**, then **Alerts**.
27
+
1. From the top command bar, select **Alert rules**. The page shows all your alert rules on all subscriptions.
28
+
1. Select the log search alert rule that you want to monitor.
29
+
1. From the left pane, under **Help**, select **Resource health**.
30
+
31
+
:::image type="content" source="media/log-search-alert-health/log-search-alert-resource-health.png" alt-text="Screenshot of the Resource health section in a log search alert rule.":::
24
32
25
-
1. In the [portal](https://portal.azure.com/), select **Monitor**, then **Alerts**.
26
-
1. From the top command bar, select **Alert rules**. The page shows all your alert rules on all subscriptions.
27
-
1. Select the log search alert rule that you want to monitor.
28
-
1. From the left pane, under **Help**, select **Resource health**.
29
-
30
-
:::image type="content" source="media/log-search-alert-health/log-search-alert-resource-health.png" alt-text="Screenshot of the Resource health section in a log search alert rule.":::
31
-
32
-
See [Monitor the health of log search alert rules](log-alert-rule-health.md#monitor-the-health-of-log-search-alert-rules) to learn more.
33
+
See [Monitor the health of log search alert rules](log-alert-rule-health.md#monitor-the-health-of-log-search-alert-rules) to learn more.
33
34
34
35
1.**Check the log ingestion latency.**
35
36
36
-
Azure Monitor processes terabytes of customers' logs from across the world, which can cause [logs ingestion latency](../logs/data-ingestion-time.md).
37
-
38
-
Logs are semi-structured data and are inherently more latent than metrics. If you're experiencing more than a 4-minute delay in fired alerts, you should consider using [metric alerts](alerts-metric-overview.md). You can send data to the metric store from logs using [metric alerts for logs](alerts-metric-logs.md).
37
+
Azure Monitor processes terabytes of customers' logs from across the world, which can cause [logs ingestion latency](../logs/data-ingestion-time.md).
38
+
39
+
Logs are semi-structured data and are inherently more latent than metrics. If you're experiencing more than a 4-minute delay in fired alerts, you should consider using [metric alerts](alerts-metric-overview.md). You can send data to the metric store from logs using [metric alerts for logs](alerts-metric-logs.md).
39
40
40
-
To mitigate latency, the system retries the alert evaluation multiple times. After the data arrives, the alert fires, which in most cases don't equal the log record time.
41
+
To mitigate latency, the system retries the alert evaluation multiple times. After the data arrives, the alert fires, which in most cases don't equal the log record time.
41
42
42
43
1.**Are the actions muted or was the alert rule configured to resolve automatically?**
43
44
44
-
A common issue is that you think that the alert didn't fire, but the rule was configured so that the alert would not fire. See the advanced options of the [log search alert rule](./alerts-create-log-alert-rule.md) to verify that both of the following are not selected:
45
-
* The **Mute actions** checkbox: allows you to mute fired alert actions for a set amount of time.
46
-
***Automatically resolve alerts**: configures the alert to only fire once per condition being met.
A common issue is that you think that the alert didn't fire, but the rule was configured so that the alert would not fire. See the advanced options of the [log search alert rule](./alerts-create-log-alert-rule.md) to verify that both of the following are not selected:
46
+
* The **Mute actions** checkbox: allows you to mute fired alert actions for a set amount of time.
47
+
***Automatically resolve alerts**: configures the alert to only fire once per condition being met.
If a log search alert rule query fails to evaluate continuously for a week, Azure Monitor disables it automatically.
53
-
The following sections list some reasons why Azure Monitor might disable a log search alert rule. Additionally, there's an example of the [Activity log](../../azure-monitor/essentials/activity-log.md) event that is submitted when a rule is disabled.
53
+
If a log search alert rule query fails to evaluate continuously for a week, Azure Monitor disables it automatically.
54
+
55
+
The following sections list some reasons why Azure Monitor might disable a log search alert rule. Additionally, there's an example of the [Activity log](../../azure-monitor/essentials/activity-log.md) event that is submitted when a rule is disabled.
54
56
55
57
### Activity log example when rule is disabled
56
58
@@ -117,25 +119,25 @@ The following sections list some reasons why Azure Monitor might disable a log s
117
119
118
120
1.**Was the alert rule resource moved or deleted?**
119
121
120
-
If an alert rule resource moves, gets renamed, or is deleted, all log alert rules referring to that resource will break. To fix this issue, alert rules need to be recreated using a valid target resource for the scope.
122
+
If an alert rule resource moves, gets renamed, or is deleted, all log alert rules referring to that resource will break. To fix this issue, alert rules need to be recreated using a valid target resource for the scope.
121
123
122
124
1.**Does the alert rule uses a system-assigned managed identity?**
123
125
124
-
When you create a log alert rule with system-assigned managed identity, the identity is created without any permissions. After you create the rule, you need to assign the appropriate roles to the rule’s identity so that it can access the data you want to query. For example, you might need to give it a Reader role for the relevant Log Analytics workspaces, or a Reader role and a Database Viewer role for the relevant ADX cluster. See [managed identities](/azure/azure-monitor/alerts/alerts-create-log-alert-rule#configure-the-alert-rule-details) for more information about using managed identities in log alerts.
126
+
When you create a log alert rule with system-assigned managed identity, the identity is created without any permissions. After you create the rule, you need to assign the appropriate roles to the rule’s identity so that it can access the data you want to query. For example, you might need to give it a Reader role for the relevant Log Analytics workspaces, or a Reader role and a Database Viewer role for the relevant ADX cluster. See [managed identities](/azure/azure-monitor/alerts/alerts-create-log-alert-rule#configure-the-alert-rule-details) for more information about using managed identities in log alerts.
125
127
126
128
1.**Is the query used in the log search alert rule valid?**
127
129
128
-
When a log alert rule is created, the query is validated for correct syntax. But sometimes the query provided in the log alert rule can start to fail. Some common reasons are:
130
+
When a log alert rule is created, the query is validated for correct syntax. But sometimes the query provided in the log alert rule can start to fail. Some common reasons are:
129
131
130
-
- Rules were created via the API, and the user skipped validation.
131
-
- The query [runs on multiple resources](../logs/cross-workspace-query.md), and one or more of the resources was deleted or moved.
132
-
- The [query fails](../logs/api/errors.md) because:
133
-
- The logging solution wasn't [deployed to the workspace](../insights/solutions.md#install-a-monitoring-solution), so tables aren't created.
134
-
- Data stopped flowing to a table in the query for more than 30 days.
135
-
-[Custom logs tables](../agents/data-sources-custom-logs.md) haven't been created because the data flow hasn't started.
136
-
- Changes in the [query language](/azure/kusto/query/) include a revised format for commands and functions, so the query provided earlier is no longer valid.
137
-
138
-
[Azure Advisor](../../advisor/advisor-overview.md) warns you about this behavior. It adds a recommendation about the affected log search alert rule. The category used is 'High Availability' with medium impact and a description of 'Repair your log alert rule to ensure monitoring'.
132
+
- Rules were created via the API, and the user skipped validation.
133
+
- The query [runs on multiple resources](../logs/cross-workspace-query.md), and one or more of the resources was deleted or moved.
134
+
- The [query fails](../logs/api/errors.md) because:
135
+
- The logging solution wasn't [deployed to the workspace](../insights/solutions.md#install-a-monitoring-solution), so tables aren't created.
136
+
- Data stopped flowing to a table in the query for more than 30 days.
137
+
-[Custom logs tables](../agents/data-sources-custom-logs.md) haven't been created because the data flow hasn't started.
138
+
- Changes in the [query language](/azure/kusto/query/) include a revised format for commands and functions, so the query provided earlier is no longer valid.
139
+
140
+
[Azure Advisor](../../advisor/advisor-overview.md) warns you about this behavior. It adds a recommendation about the affected log search alert rule. The category used is 'High Availability' with medium impact and a description of 'Repair your log alert rule to ensure monitoring'.
139
141
140
142
141
143
## A log search alert fired when it shouldn't have
@@ -144,14 +146,15 @@ A configured [log alert rule in Azure Monitor](./alerts-log.md) might be trigger
144
146
145
147
1.**Was the alert triggered due to latency issues?**
146
148
147
-
Azure Monitor processes terabytes of customer logs globally, which can cause [logs ingestion latency](../logs/data-ingestion-time.md). There are built-in capabilities to prevent false alerts, but they can still occur on very latent data (over ~30 minutes) and data with latency spikes.
148
-
149
-
Logs are semi-structured data and are inherently more latent than metrics. If you're experiencing many misfires in fired alerts, consider using [metric alerts](alerts-types.md#metric-alerts). You can send data to the metric store from logs using [metric alerts for logs](alerts-metric-logs.md).
150
-
151
-
Log search alerts work best when you are try to detect specific data in the logs. They are less effective when you are trying to detect lack of data in the logs, like alerting on virtual machine heartbeat.
152
-
149
+
Azure Monitor processes terabytes of customer logs globally, which can cause [logs ingestion latency](../logs/data-ingestion-time.md). There are built-in capabilities to prevent false alerts, but they can still occur on very latent data (over ~30 minutes) and data with latency spikes.
150
+
151
+
Logs are semi-structured data and are inherently more latent than metrics. If you're experiencing many misfires in fired alerts, consider using [metric alerts](alerts-types.md#metric-alerts). You can send data to the metric store from logs using [metric alerts for logs](alerts-metric-logs.md).
152
+
153
+
Log search alerts work best when you are try to detect specific data in the logs. They are less effective when you are trying to detect lack of data in the logs, like alerting on virtual machine heartbeat.
154
+
153
155
154
156
## Error messages when configuring log search alert rules
157
+
See the following sections for specific error messages and their resolutions.
155
158
156
159
### The query couldn't be validated since you need permission for the logs
0 commit comments