Skip to content

Commit 6bfdf9e

Browse files
committed
Restructure of Azure Monitor pipeline content
1 parent fbf631c commit 6bfdf9e

File tree

3 files changed

+37
-34
lines changed

3 files changed

+37
-34
lines changed

articles/azure-monitor/essentials/data-collection-rule-overview.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,23 @@ ms.custom: references_regions
1010
---
1111

1212
# Data collection rules in Azure Monitor
13+
*Data collection rules (DCR)* are part of an [ETL](/azure/architecture/data-guide/relational-data/etl)-like data collection process that improves on legacy data collection methods for Azure Monitor. This process uses a common data ingestion pipeline, the Azure Monitor pipeline, for all data sources and a standard method of configuration that's more manageable and scalable than other methods. Specific advantages of DCR-based data collection include the following:
14+
15+
- Consistent method for configuration of different data sources.
16+
- Ability to apply a transformation to filter or modify incoming data before it's stored.
17+
- Scalable configuration options supporting infrastructure as code and DevOps processes.
18+
- Option of edge pipeline in your own environment to provide high-end scalability, layered network configurations, and periodic connectivity.
19+
20+
21+
## Azure Monitor pipeline
22+
Data collection using the Azure Monitor pipeline is shown in the diagram below. All data is processed through the *cloud pipeline*, which is automatically available in your Azure subscription and requires no configuration. Each collection scenario is defined in a [data collection rule (DCR)](#data-collection-rules). Some environments may choose to implement a local edge pipeline to manage data collection before it's sent to the cloud. See [edge pipeline](#edge-pipeline) for details on this option.
23+
24+
:::image type="content" source="media/pipeline-overview/pipeline-overview.png" lightbox="media/pipeline-overview/pipeline-overview.png" alt-text="Diagram that shows the data flow for Azure Monitor pipeline." border="false":::
25+
26+
27+
28+
## Data collection rules
29+
1330
Data collection rules (DCRs) are sets of instructions supporting data collection using the [Azure Monitor pipeline](./pipeline-overview.md). They provide a consistent and centralized way to define and customize different data collection scenarios. Depending on the scenario, DCRs specify such details as what data should be collected, how to transform that data, and where to send it.
1431

1532
DCRs are stored in Azure so that you can centrally manage them. Different components of a data collection workflow will access the DCR for particular information that it requires. In some cases, you can use the Azure portal to configure data collection, and Azure Monitor will create and manage the DCR for you. Other scenarios will require you to create your own DCR. You may also choose to customize an existing DCR to meet your required functionality.
@@ -25,6 +42,24 @@ Some data collection scenarios will use data collection rule associations (DCRAs
2542

2643
For example, the diagram above illustrates data collection for the Azure Monitor agent. When the agent is installed, it connects to Azure Monitor to retrieve any DCRs that are associated with it. You can create an association with to the same DCRs for multiple VMs.
2744

45+
## Transformations
46+
*Transformations* allow you to modify incoming data before it's stored in Azure Monitor. They are [KQL queries](../logs/log-query-overview.md) defined in the DCR that run in the cloud pipeline. See [Data collection transformations in Azure Monitor](./data-collection-transformations.md) for details on how to create and use transformations.
47+
48+
The specific use case for Azure Monitor pipeline are:
49+
50+
- **Reduce costs**. Remove unneeded records or columns to save on ingestion costs.
51+
- **Remove sensitive data**. Filter or obfuscate private data.
52+
- **Enrich data**. Add a calculated column to simplify log queries.
53+
- **Format data**. Change the format of incoming data to match the schema of the destination table.
54+
55+
## Edge pipeline
56+
The edge pipeline extends the Azure Monitor pipeline to your own data center. It enables at-scale collection and routing of telemetry data before it's delivered to Azure Monitor in the Azure cloud. See [Configure an edge pipeline in Azure Monitor](./edge-pipeline-configure.md) for details on how to set up an edge pipeline.
57+
58+
The specific use case for Azure Monitor edge pipeline are:
59+
60+
- **Scalability**. The edge pipeline can handle large volumes of data from monitored resources that may be limited by other collection methods such as Azure Monitor agent.
61+
- **Periodic connectivity**. Some environments may have unreliable connectivity to the cloud, or may have long unexpected periods without connection. The edge pipeline can cache data locally and sync with the cloud when connectivity is restored.
62+
- **Layered network**. In some environments, the network is segmented and data cannot be sent directly to the cloud. The edge pipeline can be used to collect data from monitored resources without cloud access and manage the connection to Azure Monitor in the cloud.
2863

2964
## Supported regions
3065
Data collection rules are available in all public regions where Log Analytics workspaces and the Azure Government and China clouds are supported. Air-gapped clouds aren't yet supported.

articles/azure-monitor/essentials/pipeline-overview.md

Lines changed: 1 addition & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -8,45 +8,13 @@ author: bwren
88
---
99

1010
# Overview of Azure Monitor pipeline
11-
*Azure Monitor pipeline* is part of an [ETL](/azure/architecture/data-guide/relational-data/etl)-like data collection process that improves on legacy data collection methods for Azure Monitor. This process uses a common data ingestion pipeline for all data sources and a standard method of configuration that's more manageable and scalable than other methods. Specific advantages of the data collection using the pipeline include the following:
1211

13-
- Common set of destinations for different data sources.
14-
- Ability to apply a transformation to filter or modify incoming data before it's stored.
15-
- Consistent method for configuration of different data sources.
16-
- Scalable configuration options supporting infrastructure as code and DevOps processes.
17-
- Option of edge pipeline in your own environment to provide high-end scalability, layered network configurations, and periodic connectivity.
1812

19-
> [!NOTE]
20-
> When implementation is complete, all data collected by Azure Monitor will use the pipeline. Currently, only [certain data collection methods](#data-collection-scenarios) are supported, and they may have limited configuration options. There's no difference between data collected with the Azure Monitor pipeline and data collected using other methods. The data is all stored together as [Logs](../logs/data-platform-logs.md) and [Metrics](data-platform-metrics.md), supporting Azure Monitor features such as log queries, alerts, and workbooks. The only difference is in the method of collection.
2113

22-
## Components of pipeline data collection
23-
Data collection using the Azure Monitor pipeline is shown in the diagram below. All data is processed through the *cloud pipeline*, which is automatically available in your subscription and needs no configuration. Each collection scenario is configured in a [data collection rule (DCR)](./data-collection-rule-overview.md), which is a set of instructions describing details such as the schema of the incoming data, a transformation to optionally modify the data, and the destination where the data should be sent.
2414

25-
Some environments may choose to implement a local edge pipeline to manage data collection before it's sent to the cloud. See [edge pipeline](#edge-pipeline) for details on this option.
15+
2616

27-
:::image type="content" source="media/pipeline-overview/pipeline-overview.png" lightbox="media/pipeline-overview/pipeline-overview.png" alt-text="Diagram that shows the data flow for Azure Monitor pipeline." border="false":::
2817

29-
## Data collection rules
30-
*Data collection rules (DCRs)* are sets of instructions supporting data collection using the Azure Monitor pipeline. Depending on the scenario, DCRs specify such details as what data should be collected, how to transform that data, and where to send it. In some scenarios, you can use the Azure portal to configure data collection, while other scenarios may require you to create and manage your own DCR. See [Data collection rules in Azure Monitor](./data-collection-rule-overview.md) for details on how to create and work with DCRs.
31-
32-
## Transformations
33-
*Transformations* allow you to modify incoming data before it's stored in Azure Monitor. They are [KQL queries](../logs/log-query-overview.md) defined in the DCR that run in the cloud pipeline. See [Data collection transformations in Azure Monitor](./data-collection-transformations.md) for details on how to create and use transformations.
34-
35-
The specific use case for Azure Monitor pipeline are:
36-
37-
- **Reduce costs**. Remove unneeded records or columns to save on ingestion costs.
38-
- **Remove sensitive data**. Filter or obfuscate private data.
39-
- **Enrich data**. Add a calculated column to simplify log queries.
40-
- **Format data**. Change the format of incoming data to match the schema of the destination table.
41-
42-
## Edge pipeline
43-
The edge pipeline extends the Azure Monitor pipeline to your own data center. It enables at-scale collection and routing of telemetry data before it's delivered to Azure Monitor in the Azure cloud. See [Configure an edge pipeline in Azure Monitor](./edge-pipeline-configure.md) for details on how to set up an edge pipeline.
44-
45-
The specific use case for Azure Monitor edge pipeline are:
46-
47-
- **Scalability**. The edge pipeline can handle large volumes of data from monitored resources that may be limited by other collection methods such as Azure Monitor agent.
48-
- **Periodic connectivity**. Some environments may have unreliable connectivity to the cloud, or may have long unexpected periods without connection. The edge pipeline can cache data locally and sync with the cloud when connectivity is restored.
49-
- **Layered network**. In some environments, the network is segmented and data cannot be sent directly to the cloud. The edge pipeline can be used to collect data from monitored resources without cloud access and manage the connection to Azure Monitor in the cloud.
5018

5119
## Data collection scenarios
5220
The following table describes the data collection scenarios that are currently supported using the Azure Monitor pipeline. See the links in each entry for details.

articles/azure-monitor/toc.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -621,7 +621,7 @@ items:
621621
href: data-sources.md
622622
- name: Data collection methods
623623
items:
624-
- name: Azure Monitor pipeline
624+
- name: Data collection rules
625625
items:
626626
- name: Overview
627627
displayName: Pipeline

0 commit comments

Comments
 (0)