You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-monitor/essentials/data-collection-transformations-structure.md
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -41,17 +41,18 @@ Transformations in a [data collection rule (DCR)](data-collection-rule-overview.
41
41
42
42
43
43
## Inline reference table
44
-
The [datatable](/azure/data-explorer/kusto/query/datatableoperator?pivots=azuremonitor) operator isn't supported in the subset of KQL available to use in transformations. This would normally be used in KQL to define an inline query-time table. Use dynamic literals instead to work around this limitation.
44
+
The [datatable](/azure/data-explorer/kusto/query/datatableoperator?pivots=azuremonitor) operator isn't supported in the subset of KQL available to use in transformations. This operator would normally be used in KQL to define an inline query-time table. Use dynamic literals instead to work around this limitation.
45
45
46
-
For example, the following isn't supported in a transformation:
46
+
For example, the following statement isn't supported in a transformation:
47
47
48
48
```kusto
49
49
let galaxy = datatable (country:string,entity:string)['ES','Spain','US','United States'];
50
50
source
51
51
| join kind=inner (galaxy) on $left.Location == $right.country
52
52
| extend Galaxy_CF = ['entity']
53
53
```
54
-
You can instead use the following statement which is supported and performs the same functionality:
54
+
55
+
You can instead use the following statement, which is supported and performs the same functionality:
55
56
56
57
```kusto
57
58
let galaxyDictionary = parsejson('{"ES": "Spain","US": "United States"}');
Copy file name to clipboardExpand all lines: articles/azure-monitor/essentials/data-collection.md
+17-17Lines changed: 17 additions & 17 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ ms.date: 07/10/2022
6
6
---
7
7
8
8
# Data collection in Azure Monitor
9
-
Azure Monitor has a [common data platform](data-platform.md) that consolidates data from a variety of sources. Currently, different sources of data for Azure Monitor use different methods to deliver their data, and each typically require different types of configuration. Get a description of the most common data sources at [Sources of monitoring data for Azure Monitor](data-sources.md).
9
+
Azure Monitor has a [common data platform](../data-platform.md) that consolidates data from a variety of sources. Currently, different sources of data for Azure Monitor use different methods to deliver their data, and each typically require different types of configuration. Get a description of the most common data sources at [Sources of monitoring data for Azure Monitor](../data-sources.md).
10
10
11
11
Azure Monitor is implementing a new [ETL](/azure/architecture/data-guide/relational-data/etl)-like data collection pipeline that improves on legacy data collection methods. This process uses a common data ingestion pipeline for all data sources and provides a standard method of configuration that's more manageable and scalable than current methods. Specific advantages of the new data collection include the following:
12
12
@@ -15,44 +15,44 @@ Azure Monitor is implementing a new [ETL](/azure/architecture/data-guide/relatio
15
15
- Consistent method for configuration of different data sources.
16
16
- Scalable configuration options supporting infrastructure as code and DevOps processes.
17
17
18
-
When implementation is complete, all data collected by Azure Monitor will use the new data collection process and be managed by data collection rules. Currently, only certain data collection methods support the ingestion pipeline, and they may have limited configuration options. There's no difference between data collected with the new ingestion pipeline and data collected using other methods. The data is all stored together as [Logs](logs/data-platform-logs.md) and [Metrics](essentials/data-platform-metrics.md), supporting Azure Monitor features such as log queries, alerts, and workbooks. The only difference is in the method of collection.
18
+
When implementation is complete, all data collected by Azure Monitor will use the new data collection process and be managed by data collection rules. Currently, only certain data collection methods support the ingestion pipeline, and they may have limited configuration options. There's no difference between data collected with the new ingestion pipeline and data collected using other methods. The data is all stored together as [Logs](../logs/data-platform-logs.md) and [Metrics](data-platform-metrics.md), supporting Azure Monitor features such as log queries, alerts, and workbooks. The only difference is in the method of collection.
19
19
## Data collection rules
20
-
Azure Monitor data collection is configured using a [data collection rule (DCR)](essentials/data-collection-rule-overview.md). A DCR defines the details of a particular data collection scenario including what data should be collected, how to potentially transform that data, and where to send that data. A single DCR can be used with multiple monitored resources, giving you a consistent method to configure a variety of monitoring scenarios. In some cases, Azure Monitor will create and configure a DCR for you using options in the Azure portal. You may also directly edit DCRs to configure particular scenarios.
20
+
Azure Monitor data collection is configured using a [data collection rule (DCR)](data-collection-rule-overview.md). A DCR defines the details of a particular data collection scenario including what data should be collected, how to potentially transform that data, and where to send that data. A single DCR can be used with multiple monitored resources, giving you a consistent method to configure a variety of monitoring scenarios. In some cases, Azure Monitor will create and configure a DCR for you using options in the Azure portal. You may also directly edit DCRs to configure particular scenarios.
21
21
22
-
See [Data collection rules in Azure Monitor](essentials/data-collection-rule-overview.md) for details on data collection rules including how to view and create them.
22
+
See [Data collection rules in Azure Monitor](data-collection-rule-overview.md) for details on data collection rules including how to view and create them.
23
23
24
24
## Transformations
25
-
One of the most valuable features of the new data collection process is [data transformations](essentials/data-collection-transformations.md), which allow you to apply a KQL query to incoming data to modify it before sending it to its destination. You might filter out unwanted data or modify existing data to improve your query or reporting capabilities.
25
+
One of the most valuable features of the new data collection process is [data transformations](data-collection-transformations.md), which allow you to apply a KQL query to incoming data to modify it before sending it to its destination. You might filter out unwanted data or modify existing data to improve your query or reporting capabilities.
26
26
27
-
See [Data collection transformations in Azure Monitor (preview)](essentials/data-collection-transformations.md) For complete details on transformations including how to write transformation queries.
27
+
See [Data collection transformations in Azure Monitor (preview)](data-collection-transformations.md) For complete details on transformations including how to write transformation queries.
28
28
29
29
30
30
## Data collection scenarios
31
31
The following sections describe the data collection scenarios that are currently supported using DCR and the new data ingestion pipeline.
32
32
33
33
### Azure Monitor agent
34
-
The diagram below shows data collection for the [Azure Monitor agent](agents/azure-monitor-agent-overview.md) running on a virtual machine. In this scenario, the DCR specifies events and performance data to collect from the agent machine, a transformation to filter and modify the data after its collected, and a Log Analytics workspace to send the transformed data. To implement this scenario, you create an association between the DCR and the agent. One agent can be associated with multiple DCRs, and one DCR can be associated with multiple agents.
34
+
The diagram below shows data collection for the [Azure Monitor agent](../agents/azure-monitor-agent-overview.md) running on a virtual machine. In this scenario, the DCR specifies events and performance data to collect from the agent machine, a transformation to filter and modify the data after its collected, and a Log Analytics workspace to send the transformed data. To implement this scenario, you create an association between the DCR and the agent. One agent can be associated with multiple DCRs, and one DCR can be associated with multiple agents.
35
35
36
-
:::image type="content" source="essentials/media/data-collection-transformations/transformation-azure-monitor-agent.png" lightbox="essentials/media/data-collection-transformations/transformation-azure-monitor-agent.png" alt-text="Diagram showing data collection for Azure Monitor agent." border="false":::
36
+
:::image type="content" source="media/data-collection-transformations/transformation-azure-monitor-agent.png" lightbox="media/data-collection-transformations/transformation-azure-monitor-agent.png" alt-text="Diagram showing data collection for Azure Monitor agent." border="false":::
37
37
38
-
See [Collect data from virtual machines with the Azure Monitor agent](agents/data-collection-rule-azure-monitor-agent.md) for details on creating a DCR for the Azure Monitor agent.
38
+
See [Collect data from virtual machines with the Azure Monitor agent](../agents/data-collection-rule-azure-monitor-agent.md) for details on creating a DCR for the Azure Monitor agent.
39
39
40
40
### Log ingestion API
41
-
The diagram below shows data collection for the [Logs ingestion API](logs/logs-ingestion-api-overview.md), which allows you to send data to a Log Analytics workspace from any REST client. In this scenario, the API call connects to a [data collection endpoint (DCE)](essentials/data-collection-endpoint-overview.md) and specifies a DCR to accept its incoming data. The DCR understands the structure of the incoming data, includes a transformation that ensures that the data is in the format of the target table, and specifies a workspace and table to send the transformed data.
41
+
The diagram below shows data collection for the [Logs ingestion API](../logs/logs-ingestion-api-overview.md), which allows you to send data to a Log Analytics workspace from any REST client. In this scenario, the API call connects to a [data collection endpoint (DCE)](data-collection-endpoint-overview.md) and specifies a DCR to accept its incoming data. The DCR understands the structure of the incoming data, includes a transformation that ensures that the data is in the format of the target table, and specifies a workspace and table to send the transformed data.
42
42
43
-
:::image type="content" source="essentials/media/data-collection-transformations/transformation-data-ingestion-api.png" lightbox="essentials/media/data-collection-transformations/transformation-data-ingestion-api.png" alt-text="Diagram showing data collection for custom application using logs ingestion API." border="false":::
43
+
:::image type="content" source="media/data-collection-transformations/transformation-data-ingestion-api.png" lightbox="media/data-collection-transformations/transformation-data-ingestion-api.png" alt-text="Diagram showing data collection for custom application using logs ingestion API." border="false":::
44
44
45
-
See [Logs ingestion API in Azure Monitor (Preview)](logs/logs-ingestion-api-overview.md) for details on the Logs ingestion API.
45
+
See [Logs ingestion API in Azure Monitor (Preview)](../logs/logs-ingestion-api-overview.md) for details on the Logs ingestion API.
46
46
47
47
### Workspace transformation DCR
48
-
The diagram below shows data collection for [resource logs](essentials/resource-logs.md) using a [workspace transformation DCR](essentials/data-collection-transformations.md#workspace-transformation-dcr). This is a special DCR that's associated with a workspace and provides a default transformation for [supported tables](logs/tables-feature-support.md). This transformation is applied to any data sent to the table that doesn't use another DCR. The example here shows resource logs using a diagnostic setting, but this same transformation could be applied to other data collection methods such as Log Analytics agent or Container insights.
48
+
The diagram below shows data collection for [resource logs](resource-logs.md) using a [workspace transformation DCR](data-collection-transformations.md#workspace-transformation-dcr). This is a special DCR that's associated with a workspace and provides a default transformation for [supported tables](../logs/tables-feature-support.md). This transformation is applied to any data sent to the table that doesn't use another DCR. The example here shows resource logs using a diagnostic setting, but this same transformation could be applied to other data collection methods such as Log Analytics agent or Container insights.
49
49
50
-
:::image type="content" source="essentials/media/data-collection-transformations/transformation-diagnostic-settings.png" lightbox="essentials/media/data-collection-transformations/transformation-diagnostic-settings.png" alt-text="Diagram showing data collection for resource logs using a transformation in the workspace transformation DCR." border="false":::
50
+
:::image type="content" source="media/data-collection-transformations/transformation-diagnostic-settings.png" lightbox="media/data-collection-transformations/transformation-diagnostic-settings.png" alt-text="Diagram showing data collection for resource logs using a transformation in the workspace transformation DCR." border="false":::
51
51
52
-
See [Workspace transformation DCR](essentials/data-collection-transformations.md#workspace-transformation-dcr) for details about workspace transformation DCRs and links to walkthroughs for creating them.
52
+
See [Workspace transformation DCR](data-collection-transformations.md#workspace-transformation-dcr) for details about workspace transformation DCRs and links to walkthroughs for creating them.
53
53
54
54
## Next steps
55
55
56
-
- Read more about [data collection rules](essentials/data-collection-rule-overview.md).
57
-
- Read more about [transformations](essentials/data-collection-transformations.md).
56
+
- Read more about [data collection rules](data-collection-rule-overview.md).
57
+
- Read more about [transformations](data-collection-transformations.md).
Copy file name to clipboardExpand all lines: articles/search/monitor-azure-cognitive-search.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,7 +13,7 @@ ms.custom: subject-monitoring
13
13
14
14
# Monitoring Azure Cognitive Search
15
15
16
-
[Azure Monitor](../azure-monitor/overview.md) is enabled with every subscription to provide monitoring capabilities over all Azure resources, including Cognitive Search. When you sign up for search, Azure Monitor collects [**activity logs**](/azure/azure-monitor/agents/data-sources#azure-activity-log) and [**metrics**](/azure/azure-monitor/essentials/data-platform-metrics) as soon as you start using the service.
16
+
[Azure Monitor](../azure-monitor/overview.md) is enabled with every subscription to provide monitoring capabilities over all Azure resources, including Cognitive Search. When you sign up for search, Azure Monitor collects [**activity logs**](/azure/azure-monitor/data-sources#azure-activity-log) and [**metrics**](/azure/azure-monitor/essentials/data-platform-metrics) as soon as you start using the service.
17
17
18
18
Optionally, you can enable diagnostic settings to collect [**resource logs**](../azure-monitor/essentials/resource-logs.md). Resource logs contain detailed information about search service operations that's useful for deeper analysis and investigation.
19
19
@@ -50,7 +50,7 @@ For REST calls, use an [admin API key](search-security-api-keys.md) and [Postman
50
50
51
51
## Monitor activity logs
52
52
53
-
In Azure Cognitive Search, [**activity logs**](/azure/azure-monitor/agents/data-sources#azure-activity-log) reflect control plane activity, such as service and capacity updates, or API key usage or management. Activity logs are collected [free of charge](/azure/azure-monitor/usage-estimated-costs#pricing-model), with no configuration required. Data retention is 90 days, but you can configure durable storage for longer retention.
53
+
In Azure Cognitive Search, [**activity logs**](/azure/azure-monitor/data-sources#azure-activity-log) reflect control plane activity, such as service and capacity updates, or API key usage or management. Activity logs are collected [free of charge](/azure/azure-monitor/usage-estimated-costs#pricing-model), with no configuration required. Data retention is 90 days, but you can configure durable storage for longer retention.
54
54
55
55
1. In the Azure portal, find your search service. From the menu on the left, select **Activity logs** to view the logs for your search service.
0 commit comments