You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/sentinel/monitor-data-connector-health.md
+182-9Lines changed: 182 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,18 +1,32 @@
1
1
---
2
-
title: Monitor the health of your data connectors with this Microsoft Sentinel workbook | Microsoft Docs
3
-
description: Use the Health Monitoring workbook to keep track of your data connectors' connectivity and performance.
4
-
author: yelevin
2
+
title: Monitor the health of your Microsoft Sentinel data connectors | Microsoft Docs
3
+
description: Use the SentinelHealth data table and the Health Monitoring workbook to keep track of your data connectors' connectivity and performance.
4
+
author: bagol
5
5
ms.topic: how-to
6
6
ms.custom: mvc, ignite-fall-2021
7
-
ms.date: 11/09/2021
7
+
ms.date: 12/30/2021
8
8
ms.author: yelevin
9
9
---
10
10
11
-
# Monitor the health of your data connectors with this Microsoft Sentinel workbook
11
+
---
12
+
# Monitor the health of your data connectors
12
13
13
14
[!INCLUDE [Banner for top of topics](./includes/banner.md)]
14
15
15
-
The **Data connectors health monitoring workbook** allows you to keep track of your data connectors' health, connectivity, and performance, from within Microsoft Sentinel. The workbook provides additional monitors, detects anomalies, and gives insight regarding the workspace’s data ingestion status. You can use the workbook’s logic to monitor the general health of the ingested data, and to build custom views and rule-based alerts.
16
+
After you've configured and connected your Microsoft Sentinel workspace to your data connectors, you'll want to monitor your connector health, viewing any service or data source issues, such as authentication, throttling, and more.
17
+
18
+
You also might like to configure notifications for health drifts for relevant stakeholders who can take action. For example, configure email messages, Microsoft Teams messages, new tickets in your ticketing system, and so on.
19
+
20
+
This article describes how to use the following features, which allow you to keep track of your data connectors' health, connectivity, and performance from within Microsoft Sentinel:
21
+
22
+
-**Data connectors health monitoring workbook**. This workbook provides additional monitors, detects anomalies, and gives insight regarding the workspace’s data ingestion status. You can use the workbook’s logic to monitor the general health of the ingested data, and to build custom views and rule-based alerts.
23
+
24
+
-***SentinelHealth* data table**. (Public preview) Provides insights on health drifts, such as latest failure events per connector, or connectors with changes from success to failure states, which you can use to create alerts and other automated actions.
25
+
26
+
> [!NOTE]
27
+
> The *SentinelHealth* data table is currently supported only for [selected data connectors](#supported-data-connectors).
28
+
>
29
+
16
30
17
31
## Use the health monitoring workbook
18
32
@@ -28,9 +42,9 @@ The **Data connectors health monitoring workbook** allows you to keep track of y
28
42
29
43
There are three tabbed sections in this workbook:
30
44
31
-
1. The **Overview** tab shows the general status of data ingestion in the selected workspace: volume measures, EPS rates, and time last log received.
45
+
- The **Overview** tab shows the general status of data ingestion in the selected workspace: volume measures, EPS rates, and time last log received.
32
46
33
-
1. The **Data collection anomalies** tab will help you to detect anomalies in the data collection process, by table and data source. Each tab presents anomalies for a particular table (the **General** tab includes a collection of tables). The anomalies are calculated using the **series_decompose_anomalies()** function that returns an **anomaly score**. [Learn more about this function](/azure/data-explorer/kusto/query/series-decompose-anomaliesfunction?WT.mc_id=Portal-fx). Set the following parameters for the function to evaluate:
47
+
- The **Data collection anomalies** tab will help you to detect anomalies in the data collection process, by table and data source. Each tab presents anomalies for a particular table (the **General** tab includes a collection of tables). The anomalies are calculated using the **series_decompose_anomalies()** function that returns an **anomaly score**. [Learn more about this function](/azure/data-explorer/kusto/query/series-decompose-anomaliesfunction?WT.mc_id=Portal-fx). Set the following parameters for the function to evaluate:
34
48
35
49
-**AnomaliesTimeRange**: This time picker applies only to the data collection anomalies view.
36
50
-**SampleInterval**: The time interval in which data is sampled in the given time range. The anomaly score is calculated only on the last interval's data.
@@ -39,7 +53,7 @@ There are three tabbed sections in this workbook:
1. The **Agent info** tab shows you information about the health of the Log Analytics agents installed on your various machines, whether Azure VM, other cloud VM, on-premises VM, or physical. You can monitor the following:
56
+
- The **Agent info** tab shows you information about the health of the Log Analytics agents installed on your various machines, whether Azure VM, other cloud VM, on-premises VM, or physical. You can monitor the following:
43
57
44
58
- System location
45
59
@@ -53,5 +67,164 @@ There are three tabbed sections in this workbook:
53
67
54
68
:::image type="content" source="media/monitor-data-connector-health/data-health-workbook-3.png" alt-text="data connector health monitoring workbook agent info page" lightbox="media/monitor-data-connector-health/data-health-workbook-3.png":::
55
69
70
+
## Use the SentinelHealth data table (Public preview)
71
+
72
+
To get data connector health data from the *SentinelHealth* data table, you must first [turn on the Microsoft Sentinel health feature](#turn-on-microsoft-sentinel-health-for-your-workspace) for your workspace.
73
+
74
+
Once the health feature is turned on, the *SentinelHealth* data table is created at the first success or failure event generated for your data connectors.
75
+
76
+
> [!TIP]
77
+
> To configure the retention time for your health events, see the [Log Analytics retention configuration documentation](/azure/azure-monitor/logs/manage-cost-storage).
78
+
>
79
+
80
+
> [!IMPORTANT]
81
+
>
82
+
> The SentinelHealth data table is currently in **PREVIEW**. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for additional legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
83
+
>
84
+
85
+
### Supported data connectors
86
+
87
+
The *SentinelHealth* data table is currently supported only for the following data connectors:
88
+
89
+
-[Amazon Web Services (CloudTrail)](connect-aws.md)
### Turn on Microsoft Sentinel health for your workspace
97
+
98
+
1. In Microsoft Sentinel, under the **Configuration** menu on the left, select **Settings** and expand the **Health** section.
99
+
100
+
1. Select **Configure Diagnostic Settings** and create a new diagnostic setting.
101
+
102
+
- In the **Diagnostic setting name** field, enter a meaningful name for your setting.
103
+
104
+
- In the **Category details** column, select **DataConnectors**.
105
+
106
+
- Under **Destination details**, select **Send to Log Analytics workspace**, and select your subscription and workspace from the dropdown menus.
107
+
108
+
1. Select **Save** to save your new setting.
109
+
110
+
The *SentinelHealth* data table is created at the first success or failure event generated for your data connectors.
111
+
112
+
113
+
### Access the *SentinelHealth* table
114
+
115
+
In the Microsoft Sentinel **Logs** page, run a query on the *SentinelHealth* table. For example:
116
+
117
+
```kusto
118
+
SentinelHealth
119
+
| take 20
120
+
```
121
+
122
+
### Understanding SentinelHealth table events
123
+
124
+
The following types of health events are logged in the *SentinelHealth* table:
125
+
126
+
-**Data fetch status change**. Logged once an hour as long as a data connector status remains stable, with either continuous success or failure events. For as long as a data connector's status does not change, monitoring only hourly works to prevent redundant auditing and reduce table size. If the data connector's status has continuous failures, additional details about the failures are included in the *ExtendedProperties* column.
127
+
128
+
If the data connector's status changes, either from a success to failure, from failure to success, or has changes in failure reasons, the event is logged immediately to allow your team to take proactive and immediate action.
129
+
130
+
Potentially transient errors, such as source service throttling, are logged only after they've continued for more than 60 minutes. These 60 minutes allow Microsoft Sentinel to overcome a transient issue in the backend and catch up with the data, without requiring any user action. Errors that are definitely not transient are logged immediately.
131
+
132
+
-**Failure summary**. Logged once an hour, per connector, per workspace, with an aggregated failure summary. Failure summary events are created only when the connector has experienced polling errors during the given hour. They contain any extra details provided in the *ExtendedProperties* column, such as the time period for which the connector's source platform was queried, and a distinct list of failures encountered during the time period.
133
+
134
+
For more information, see [SentinelHealth table columns schema](#sentinelhealth-table-columns-schema).
135
+
136
+
### Run queries to detect health drifts
137
+
138
+
Create queries on the *SentinelHealth* table to help you detect health drifts in your data connectors. For example:
139
+
140
+
**Detect latest failure events per connector**:
141
+
142
+
```kusto
143
+
SentinelHealth
144
+
| where TimeGenerated > ago(3d)
145
+
| where OperationName == 'Data fetch status change'
146
+
| where Status in ('Success', 'Failure')
147
+
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId
148
+
| where Status == 'Failure'
149
+
```
150
+
151
+
**Detect connectors with changes from fail to success state**:
152
+
153
+
```kusto
154
+
let lastestStatus = SentinelHealth
155
+
| where TimeGenerated > ago(12h)
156
+
| where OperationName == 'Data fetch status change'
157
+
| where Status in ('Success', 'Failure')
158
+
| project TimeGenerated, SentinelResourceName, SentinelResourceId, LastStatus = Status
159
+
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId;
160
+
let nextToLastestStatus = SentinelHealth
161
+
| where TimeGenerated > ago(12h)
162
+
| where OperationName == 'Data fetch status change'
163
+
| where Status in ('Success', 'Failure')
164
+
| join kind = leftanti (lastestStatus) on SentinelResourceName, SentinelResourceId, TimeGenerated
165
+
| project TimeGenerated, SentinelResourceName, SentinelResourceId, NextToLastStatus = Status
166
+
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId;
167
+
lastestStatus
168
+
| join kind=inner (nextToLastestStatus) on SentinelResourceName, SentinelResourceId
169
+
| where NextToLastStatus == 'Failure' and LastStatus == 'Success'
170
+
```
171
+
172
+
**Detect connectors with changes from success to fail state**:
173
+
174
+
```kusto
175
+
let lastestStatus = SentinelHealth
176
+
| where TimeGenerated > ago(12h)
177
+
| where OperationName == 'Data fetch status change'
178
+
| where Status in ('Success', 'Failure')
179
+
| project TimeGenerated, SentinelResourceName, SentinelResourceId, LastStatus = Status
180
+
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId;
181
+
let nextToLastestStatus = SentinelHealth
182
+
| where TimeGenerated > ago(12h)
183
+
| where OperationName == 'Data fetch status change'
184
+
| where Status in ('Success', 'Failure')
185
+
| join kind = leftanti (lastestStatus) on SentinelResourceName, SentinelResourceId, TimeGenerated
186
+
| project TimeGenerated, SentinelResourceName, SentinelResourceId, NextToLastStatus = Status
187
+
| summarize TimeGenerated = arg_max(TimeGenerated,*) by SentinelResourceName, SentinelResourceId;
188
+
lastestStatus
189
+
| join kind=inner (nextToLastestStatus) on SentinelResourceName, SentinelResourceId
190
+
| where NextToLastStatus == 'Success' and LastStatus == 'Failure'
191
+
```
192
+
193
+
### Configure alerts and automated actions for health issues
194
+
195
+
While you can use the Microsoft Sentinel [analytics rules](automate-incident-handling-with-automation-rules.md) to configure automation in Microsoft Sentinel logs, if you want to be notified and take immediate action for health drifts in your data connectors, we recommend that you use [Azure Monitor alert rules](/azure/azure-monitor/alerts/alerts-overview).
196
+
197
+
For example:
198
+
199
+
1. In an Azure Monitor alert rule, select your Microsoft Sentinel workspace as the rule scope, and **Custom log search** as the first condition.
200
+
201
+
1. Customize the alert logic as needed, such as frequency or lookback duration, and then use [queries](#run-queries-to-detect-health-drifts) to search for health drifts.
202
+
203
+
1. For the rule actions, select an existing action group or create a new one as needed to configure push notifications or other automated actions such as triggering a Logic App, Webhook, or Azure Function in your system.
204
+
205
+
For more information, see [Azure Monitor alerts overview](/azure/azure-monitor/alerts/alerts-overview) and [Azure Monitor alerts log](/azure/azure-monitor/alerts/alerts-log).
206
+
207
+
### SentinelHealth table columns schema
208
+
209
+
The following table describes the columns and data generated in the *SentinelHealth* data table:
|**TenantId**| String | The tenant ID for your Microsoft Sentinel workspace. |
214
+
|**TimeGenerated**| Datetime | The time at which the health event occurred. |
215
+
| <aname="operationname"></a>**OperationName**| String | The health operation. One of the following values: <br><br>-`Data fetch status change` for health or success indications <br>- `Failure summary` for aggregated health summaries. <br><br>For more information, see [Understanding SentinelHealth table events](#understanding-sentinelhealth-table-events). |
216
+
| <aname="sentinelresourceid"></a>**SentinelResourceId**| String | The unique identifier of the Microsoft Sentinel workspace and the associated connector on which the health event occurred. |
217
+
|**SentinelResourceName**| String | The data connector name. |
218
+
| <aname="status"></a>**Status**| String | Indicates `Success` or `Failure` for the `Data fetch status change`[OperationName](#operationname), and `Informational` for the `Failure summary`[OperationName](#operationname). |
219
+
|**Description**| String | Describes the operation, including extended data as needed. For example, for failures, this column might indicate the failure reason. |
220
+
|**WorkspaceId**| String | The workspace GUID on which the health issue occurred. The full Azure Resource Identifier is available in the [SentinelResourceID](#sentinelresourceid) column. |
221
+
|**SentinelResourceType**| String |The Microsoft Sentinel resource type being monitored: `Data connector`|
222
+
|**SentinelResourceKind**| String | The type of data connector being monitored, such as `Office365`. |
223
+
|**RecordId**| String | A unique identifier for the record that can be shared with the support team for better correlation as needed. |
224
+
|**ExtendedProperties**| Dynamic (json) | A JSON bag that varies by the [OperationName](#operationname) value and the [Status](#status) of the event: <br><br>- For `Data fetch status change` events with a success indicator, the bag contains a ‘DestinationTable’ property to indicate where data from this connector is expected to land. For failures, the contents vary depending on the failure type. |
225
+
|**Type**| String |`SentinelHealth`|
226
+
||||
227
+
56
228
## Next steps
229
+
57
230
Learn how to [onboard your data to Microsoft Sentinel](quickstart-onboard.md), [connect data sources](connect-data-sources.md), and [get visibility into your data, and potential threats](get-visibility.md).
Copy file name to clipboardExpand all lines: articles/sentinel/whats-new.md
+25-1Lines changed: 25 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: This article describes new features in Microsoft Sentinel from the
4
4
author: batamig
5
5
ms.author: bagol
6
6
ms.topic: conceptual
7
-
ms.date: 12/01/2021
7
+
ms.date: 01/13/2022
8
8
ms.custom: ignite-fall-2021
9
9
---
10
10
@@ -29,8 +29,18 @@ If you're looking for items older than six months, you'll find them in the [Arch
29
29
30
30
## January 2021
31
31
32
+
-[SentinelHealth data table (Public preview)](#sentinelhealth-data-table-public-preview)
32
33
-[More workspaces supported for Multiple Workspace View](#more-workspaces-supported-for-multiple-workspace-view)
33
34
-[Kusto Query Language workbook and tutorial](#kusto-query-language-workbook-and-tutorial)
35
+
36
+
### SentinelHealth data table (Public preview)
37
+
38
+
Microsoft Sentinel now provides the **SentinelHealth** data table to help you monitor your connector health, providing insights on health drifts, such as latest failure events per connector, or connectors with changes from success to failure states. Use this data to create alerts and other automated actions, such as Microsoft Teams messages, new tickets in a ticketing system, and so on.
39
+
40
+
Turn on the Microsoft Sentinel health feature for your workspace in order to have the **SentinelHealth** data table created at the next success or failure event generated for supported data connectors.
41
+
42
+
For more information, see [Use the SentinelHealth data table (Public preview)](monitor-data-connector-health.md#use-the-sentinelhealth-data-table-public-preview).
43
+
34
44
### More workspaces supported for Multiple Workspace View
35
45
36
46
Now, instead of being limited to 10 workspaces in Microsoft Sentinel's [Multiple Workspace View](multiple-workspace-view.md), you can view data from up to 30 workspaces simultaneously.
@@ -42,6 +52,7 @@ For more information, see:
42
52
-[The need to use multiple Microsoft Sentinel workspaces](extend-sentinel-across-workspaces-tenants.md#the-need-to-use-multiple-microsoft-sentinel-workspaces)
43
53
-[Work with incidents in many workspaces at once](multiple-workspace-view.md)
44
54
-[Manage multiple tenants in Microsoft Sentinel as an MSSP](multiple-tenants-service-providers.md)
55
+
45
56
### Kusto Query Language workbook and tutorial
46
57
47
58
Kusto Query Language is used in Microsoft Sentinel to search, analyze, and visualize data, as the basis for detection rules, workbooks, hunting, and more.
@@ -372,9 +383,22 @@ For more information, see:
372
383
373
384
## September 2021
374
385
386
+
-[Data connector health enhancements (Public preview)](#data-connector-health-enhancements-public-preview)
387
+
375
388
-[New in docs: scaling data connector documentation](#new-in-docs-scaling-data-connector-documentation)
### Data connector health enhancements (Public preview)
392
+
393
+
Azure Sentinel now provides the ability to enhance your data connector health monitoring with a new *SentinelHealth* table. The *SentinelHealth* table is created after you've [turned on the Azure Sentinel health feature](monitor-data-connector-health.md#turn-on-microsoft-sentinel-health-for-your-workspace) in your Azure Sentinel workspace, at the first success or failure health event that's generated.
394
+
395
+
For more information, see [Monitor the health of your data connectors with this Azure Sentinel workbook](monitor-data-connector-health.md).
396
+
397
+
> [!NOTE]
398
+
> The *SentinelHealth* data table is currently supported only for selected data connectors. For more information, see [Supported data connectors](monitor-data-connector-health.md#supported-data-connectors).
399
+
>
400
+
401
+
378
402
### New in docs: scaling data connector documentation
379
403
380
404
As we continue to add more and more built-in data connectors for Azure Sentinel, we've reorganized our data connector documentation to reflect this scaling.
0 commit comments