Skip to content

Commit 38f4e40

Browse files
committed
formatting log alert troubleshooting
1 parent 526c042 commit 38f4e40

File tree

1 file changed

+41
-38
lines changed

1 file changed

+41
-38
lines changed

articles/azure-monitor/alerts/alerts-troubleshoot-log.md

Lines changed: 41 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -17,40 +17,42 @@ You can use log alerts to evaluate resources logs every set frequency by using a
1717
> This article doesn't discuss cases where the alert rule was triggered, you can see it in the Azure portal, but the notification was not sent. See [troubleshooting alerts](alerts-troubleshoot.md) for cases like these.
1818
1919
## A log search alert didn't fire when it should have
20+
If your log search alert didn't fire when it should have, check the following items:
2021

2122
1. **Is the alert rule is in a degraded or unavailable health state?**
2223

23-
View the health status of your log search alert rule:
24+
View the health status of your log search alert rule:
25+
26+
1. In the [portal](https://portal.azure.com/), select **Monitor**, then **Alerts**.
27+
1. From the top command bar, select **Alert rules**. The page shows all your alert rules on all subscriptions.
28+
1. Select the log search alert rule that you want to monitor.
29+
1. From the left pane, under **Help**, select **Resource health**.
30+
31+
:::image type="content" source="media/log-search-alert-health/log-search-alert-resource-health.png" alt-text="Screenshot of the Resource health section in a log search alert rule.":::
2432

25-
1. In the [portal](https://portal.azure.com/), select **Monitor**, then **Alerts**.
26-
1. From the top command bar, select **Alert rules**. The page shows all your alert rules on all subscriptions.
27-
1. Select the log search alert rule that you want to monitor.
28-
1. From the left pane, under **Help**, select **Resource health**.
29-
30-
:::image type="content" source="media/log-search-alert-health/log-search-alert-resource-health.png" alt-text="Screenshot of the Resource health section in a log search alert rule.":::
31-
32-
See [Monitor the health of log search alert rules](log-alert-rule-health.md#monitor-the-health-of-log-search-alert-rules) to learn more.
33+
See [Monitor the health of log search alert rules](log-alert-rule-health.md#monitor-the-health-of-log-search-alert-rules) to learn more.
3334

3435
1. **Check the log ingestion latency.**
3536

36-
Azure Monitor processes terabytes of customers' logs from across the world, which can cause [logs ingestion latency](../logs/data-ingestion-time.md).
37-
38-
Logs are semi-structured data and are inherently more latent than metrics. If you're experiencing more than a 4-minute delay in fired alerts, you should consider using [metric alerts](alerts-metric-overview.md). You can send data to the metric store from logs using [metric alerts for logs](alerts-metric-logs.md).
37+
Azure Monitor processes terabytes of customers' logs from across the world, which can cause [logs ingestion latency](../logs/data-ingestion-time.md).
38+
39+
Logs are semi-structured data and are inherently more latent than metrics. If you're experiencing more than a 4-minute delay in fired alerts, you should consider using [metric alerts](alerts-metric-overview.md). You can send data to the metric store from logs using [metric alerts for logs](alerts-metric-logs.md).
3940

40-
To mitigate latency, the system retries the alert evaluation multiple times. After the data arrives, the alert fires, which in most cases don't equal the log record time.
41+
To mitigate latency, the system retries the alert evaluation multiple times. After the data arrives, the alert fires, which in most cases don't equal the log record time.
4142

4243
1. **Are the actions muted or was the alert rule configured to resolve automatically?**
4344

44-
A common issue is that you think that the alert didn't fire, but the rule was configured so that the alert would not fire. See the advanced options of the [log search alert rule](./alerts-create-log-alert-rule.md) to verify that both of the following are not selected:
45-
* The **Mute actions** checkbox: allows you to mute fired alert actions for a set amount of time.
46-
* **Automatically resolve alerts**: configures the alert to only fire once per condition being met.
47-
48-
:::image type="content" source="media/alerts-troubleshoot-log/LogAlertSuppress.png" lightbox="media/alerts-troubleshoot-log/LogAlertSuppress.png" alt-text="Suppress alerts":::
45+
A common issue is that you think that the alert didn't fire, but the rule was configured so that the alert would not fire. See the advanced options of the [log search alert rule](./alerts-create-log-alert-rule.md) to verify that both of the following are not selected:
46+
* The **Mute actions** checkbox: allows you to mute fired alert actions for a set amount of time.
47+
* **Automatically resolve alerts**: configures the alert to only fire once per condition being met.
48+
49+
:::image type="content" source="media/alerts-troubleshoot-log/LogAlertSuppress.png" lightbox="media/alerts-troubleshoot-log/LogAlertSuppress.png" alt-text="Suppress alerts":::
4950

5051
1. **Was the the log search alert rule disabled?**
5152

52-
If a log search alert rule query fails to evaluate continuously for a week, Azure Monitor disables it automatically.
53-
The following sections list some reasons why Azure Monitor might disable a log search alert rule. Additionally, there's an example of the [Activity log](../../azure-monitor/essentials/activity-log.md) event that is submitted when a rule is disabled.
53+
If a log search alert rule query fails to evaluate continuously for a week, Azure Monitor disables it automatically.
54+
55+
The following sections list some reasons why Azure Monitor might disable a log search alert rule. Additionally, there's an example of the [Activity log](../../azure-monitor/essentials/activity-log.md) event that is submitted when a rule is disabled.
5456

5557
### Activity log example when rule is disabled
5658

@@ -117,25 +119,25 @@ The following sections list some reasons why Azure Monitor might disable a log s
117119

118120
1. **Was the alert rule resource moved or deleted?**
119121

120-
If an alert rule resource moves, gets renamed, or is deleted, all log alert rules referring to that resource will break. To fix this issue, alert rules need to be recreated using a valid target resource for the scope.
122+
If an alert rule resource moves, gets renamed, or is deleted, all log alert rules referring to that resource will break. To fix this issue, alert rules need to be recreated using a valid target resource for the scope.
121123

122124
1. **Does the alert rule uses a system-assigned managed identity?**
123125

124-
When you create a log alert rule with system-assigned managed identity, the identity is created without any permissions. After you create the rule, you need to assign the appropriate roles to the rule’s identity so that it can access the data you want to query. For example, you might need to give it a Reader role for the relevant Log Analytics workspaces, or a Reader role and a Database Viewer role for the relevant ADX cluster. See [managed identities](/azure/azure-monitor/alerts/alerts-create-log-alert-rule#configure-the-alert-rule-details) for more information about using managed identities in log alerts.
126+
When you create a log alert rule with system-assigned managed identity, the identity is created without any permissions. After you create the rule, you need to assign the appropriate roles to the rule’s identity so that it can access the data you want to query. For example, you might need to give it a Reader role for the relevant Log Analytics workspaces, or a Reader role and a Database Viewer role for the relevant ADX cluster. See [managed identities](/azure/azure-monitor/alerts/alerts-create-log-alert-rule#configure-the-alert-rule-details) for more information about using managed identities in log alerts.
125127

126128
1. **Is the query used in the log search alert rule valid?**
127129

128-
When a log alert rule is created, the query is validated for correct syntax. But sometimes the query provided in the log alert rule can start to fail. Some common reasons are:
130+
When a log alert rule is created, the query is validated for correct syntax. But sometimes the query provided in the log alert rule can start to fail. Some common reasons are:
129131

130-
- Rules were created via the API, and the user skipped validation.
131-
- The query [runs on multiple resources](../logs/cross-workspace-query.md), and one or more of the resources was deleted or moved.
132-
- The [query fails](../logs/api/errors.md) because:
133-
- The logging solution wasn't [deployed to the workspace](../insights/solutions.md#install-a-monitoring-solution), so tables aren't created.
134-
- Data stopped flowing to a table in the query for more than 30 days.
135-
- [Custom logs tables](../agents/data-sources-custom-logs.md) haven't been created because the data flow hasn't started.
136-
- Changes in the [query language](/azure/kusto/query/) include a revised format for commands and functions, so the query provided earlier is no longer valid.
137-
138-
[Azure Advisor](../../advisor/advisor-overview.md) warns you about this behavior. It adds a recommendation about the affected log search alert rule. The category used is 'High Availability' with medium impact and a description of 'Repair your log alert rule to ensure monitoring'.
132+
- Rules were created via the API, and the user skipped validation.
133+
- The query [runs on multiple resources](../logs/cross-workspace-query.md), and one or more of the resources was deleted or moved.
134+
- The [query fails](../logs/api/errors.md) because:
135+
- The logging solution wasn't [deployed to the workspace](../insights/solutions.md#install-a-monitoring-solution), so tables aren't created.
136+
- Data stopped flowing to a table in the query for more than 30 days.
137+
- [Custom logs tables](../agents/data-sources-custom-logs.md) haven't been created because the data flow hasn't started.
138+
- Changes in the [query language](/azure/kusto/query/) include a revised format for commands and functions, so the query provided earlier is no longer valid.
139+
140+
[Azure Advisor](../../advisor/advisor-overview.md) warns you about this behavior. It adds a recommendation about the affected log search alert rule. The category used is 'High Availability' with medium impact and a description of 'Repair your log alert rule to ensure monitoring'.
139141

140142

141143
## A log search alert fired when it shouldn't have
@@ -144,14 +146,15 @@ A configured [log alert rule in Azure Monitor](./alerts-log.md) might be trigger
144146

145147
1. **Was the alert triggered due to latency issues?**
146148

147-
Azure Monitor processes terabytes of customer logs globally, which can cause [logs ingestion latency](../logs/data-ingestion-time.md). There are built-in capabilities to prevent false alerts, but they can still occur on very latent data (over ~30 minutes) and data with latency spikes.
148-
149-
Logs are semi-structured data and are inherently more latent than metrics. If you're experiencing many misfires in fired alerts, consider using [metric alerts](alerts-types.md#metric-alerts). You can send data to the metric store from logs using [metric alerts for logs](alerts-metric-logs.md).
150-
151-
Log search alerts work best when you are try to detect specific data in the logs. They are less effective when you are trying to detect lack of data in the logs, like alerting on virtual machine heartbeat.
152-
149+
Azure Monitor processes terabytes of customer logs globally, which can cause [logs ingestion latency](../logs/data-ingestion-time.md). There are built-in capabilities to prevent false alerts, but they can still occur on very latent data (over ~30 minutes) and data with latency spikes.
150+
151+
Logs are semi-structured data and are inherently more latent than metrics. If you're experiencing many misfires in fired alerts, consider using [metric alerts](alerts-types.md#metric-alerts). You can send data to the metric store from logs using [metric alerts for logs](alerts-metric-logs.md).
152+
153+
Log search alerts work best when you are try to detect specific data in the logs. They are less effective when you are trying to detect lack of data in the logs, like alerting on virtual machine heartbeat.
154+
153155

154156
## Error messages when configuring log search alert rules
157+
See the following sections for specific error messages and their resolutions.
155158

156159
### The query couldn't be validated since you need permission for the logs
157160

0 commit comments

Comments
 (0)