Merge pull request #208258 from MicrosoftDocs/main

Release branch publish release-preview-er-traffic-collector

Author: Saisang

articles/applied-ai-services/metrics-advisor/data-feeds-from-different-sources.md

@@ -46,7 +46,6 @@ Use this article to find the settings and requirements for connecting different
 |[MongoDB](#mongodb) | Basic |
 |[MySQL](#mysql) | Basic |
 |[PostgreSQL](#pgsql) | Basic|
-|[Local files (CSV)](#csv) | Basic|
 
 The following sections specify the parameters required for all authentication types within different data source scenarios. 
 
@@ -116,7 +115,7 @@ The following sections specify the parameters required for all authentication ty
 
 * **JSON format version**: Defines the data schema in the JSON files. Metrics Advisor supports the following versions. You can choose one to fill in the field:
   
-   * **v1** (default value)
+   * **v1** 
 
       Only the metrics *Name* and *Value* are accepted. For example:
     
@@ -533,17 +532,7 @@ For more information, refer to the [tutorial on writing a valid query](tutorials
     ```
     For more information, refer to the [tutorial on writing a valid query](tutorials/write-a-valid-query.md).
 
-## <span id="csv">Local files (CSV)</span>
-
-> [!NOTE]
-> This feature is only used for quick system evaluation focusing on anomaly detection. It only accepts static data from a local CSV, and performs anomaly detection on single time series data. For analyzing multi-dimensional metrics, including real-time data ingestion, anomaly notification, root cause analysis, and cross-metric incident analysis, use other supported data sources.
-
-**Requirements on data in CSV:**
-- Have at least one column, which represents measurements to be analyzed. For a better and quicker user experience, try a CSV file that contains two columns: a timestamp column and a metric column. The timestamp format should be as follows: `2021-03-30T00:00:00Z`, and the `seconds` part is best to be `:00Z`. The time granularity between every record should be the same.
-- The timestamp column is optional. If there's no timestamp, Metrics Advisor will use timestamp starting from today (`00:00:00` Coordinated Universal Time). The service maps each measure in the row at a one-hour interval.
-- There is no re-ordering or gap-filling happening during data ingestion. Make sure that your data in the CSV file is ordered by the timestamp ordering **ascending (ASC)**.
- 
 ## Next steps
 
 * While you're waiting for your metric data to be ingested into the system, read about [how to manage data feed configurations](how-tos/manage-data-feeds.md).
-* When your metric data is ingested, you can [configure metrics and fine tune detection configuration](how-tos/configure-metrics.md).
+* When your metric data is ingested, you can [configure metrics and fine tune detection configuration](how-tos/configure-metrics.md).

articles/expressroute/TOC.yml

@@ -95,6 +95,8 @@
     href: expressroute-global-reach.md
   - name: About ExpressRoute encryption
     href: expressroute-about-encryption.md
+  - name: ExpressRoute Traffic Collector
+    href: traffic-collector.md
   - name: Connect Azure to public cloud
     href: expressroute-connect-azure-to-public-cloud.md
   - name: Connectivity between virtual networks
@@ -159,6 +161,8 @@
       href: expressroute-network-insights.md
     - name: Configure Connection Monitoring for ExpressRoute
       href: how-to-configure-connection-monitor.md
+    - name: Configure ExpressRoute Traffic Collector
+      href: how-to-configure-traffic-collector.md
     - name: ExpressRoute monitoring, metrics, and alerts
       href: expressroute-monitoring-metrics-alerts.md
     - name: View and configure maintenance alerts

articles/expressroute/expressroute-faqs.md

@@ -87,6 +87,14 @@ Metrics explorer supports SUM, MAX, MIN, AVG and COUNT as [aggregation types](..
 | [RxLightLevel](#rxlight) | Physical Connectivity | Count | Average | Rx Light level in dBm | Link, Lane | No | 
 | [TxLightLevel](#txlight) | Physical Connectivity | Count | Average | Tx light level in dBm | Link, Lane | No |
 
+### ExpressRoute Traffic Collector
+
+| Metric | Category | Unit | Aggregation Type | Description | Dimensions | Exportable via Diagnostic Settings? |
+| --- | --- | --- | --- | --- | --- | --- |
+| CPU utilization | Performance | Count | Average | CPU Utilization of the ExpressRoute Traffic Collector | roleInstance | Yes |
+| Memory Utilization | Performance | CountPerSecond | Average | Memory Utilization of the ExpressRoute Traffic Collector | roleInstance | Yes |
+| Count of flow records processed | Availability | Count | Maximum | Count of number of flow records processed or ingested | roleInstance, ExpressRoute Circuit | Yes |
+
 ## Circuits metrics
 
 ### <a name = "circuitbandwidth"></a>Bits In and Out - Metrics across all peerings
@@ -231,23 +239,23 @@ This metric shows the number of routes the ExpressRoute gateway is advertising t
 
 :::image type="content" source="./media/expressroute-monitoring-metrics-alerts/count-of-routes-advertised-to-peer.png" alt-text="Screenshot of count of routes advertised to peer.":::
 
-### <a name = "learnedroutes"></a>Count of Routes Learned from Peer - Split by instance
+### <a name = "learnedroutes"></a>Count of routes learned from peer - Split by instance
 
 Aggregation type: *Max*
 
 This metric shows the number of routes the ExpressRoute gateway is learning from peers connected to the ExpressRoute circuit. These routes can be either from another virtual network connected to the same circuit or learned from on-premises. Set an alert for when the number of learned routes drop below a certain threshold. This could indicate either the gateway is seeing a performance problem or remote peers are no longer advertising routes to the ExpressRoute circuit. 
 
 :::image type="content" source="./media/expressroute-monitoring-metrics-alerts/count-of-routes-learned-from-peer.png" alt-text="Screenshot of count of routes learned from peer.":::
 
-### <a name = "frequency"></a>Frequency of Routes change - Split by instance
+### <a name = "frequency"></a>Frequency of routes change - Split by instance
 
 Aggregation type: *Sum*
 
 This metric shows the frequency of routes being learned from or advertised to remote peers. You should first investigate your on-premises devices to understand why the network is changing so frequently. A high frequency in routes change could indicate a performance problem on the ExpressRoute gateway where scaling the gateway SKU up may resolve the problem. Set an alert for a frequency threshold to be aware of when your ExpressRoute gateway is seeing abnormal route changes.
 
 :::image type="content" source="./media/expressroute-monitoring-metrics-alerts/frequency-of-routes-changed.png" alt-text="Screenshot of frequency of routes changed metric.":::
 
-### <a name = "vm"></a>Number of VMs in the Virtual Network
+### <a name = "vm"></a>Number of VMs in the virtual network
 
 Aggregation type: *Max*
 
@@ -267,6 +275,44 @@ This metric shows the bits per second for ingress and egress to Azure through th
 
 :::image type="content" source="./media/expressroute-monitoring-metrics-alerts/erconnections.jpg" alt-text="Screenshot of gateway connection bandwidth usage metric.":::
 
+## ExpressRoute Traffic Collector metrics
+
+### CPU Utilization - Split by instance
+
+Aggregation type: *Avg* (of percentage of total utilized CPU)  
+
+*Granularity: 5 min*  
+
+You can view the CPU utilization of each ExpressRoute Traffic Collector instance. The CPU utilization may spike briefly during routine host maintenance, but prolonged high CPU utilization could indicate your ExpressRoute Traffic Collector is reaching a performance bottleneck.  
+
+**Guidance:** Set an alert for when avg CPU utilization exceeds a certain threshold.
+
+:::image type="content" source="./media/expressroute-monitoring-metrics-alerts/cpu-usage.png" alt-text="Screenshot of CPU usage for ExpressRoute Traffic Collector." lightbox="./media/expressroute-monitoring-metrics-alerts/cpu-usage.png":::
+
+### Memory Utilization - Split by instance 
+
+Aggregation type: *Avg* (of percentage of total utilized Memory) 
+
+*Granularity: 5 min*
+
+You can view the memory utilization of each ExpressRoute Traffic Collector instance. Memory utilization may spike briefly during routine host maintenance, but prolonged high memory utilization could indicate your Azure Traffic Collector is reaching a performance bottleneck.  
+
+**Guidance:** Set an alert for when avg memory utilization exceeds a certain threshold.  
+
+:::image type="content" source="./media/expressroute-monitoring-metrics-alerts/memory-usage.png" alt-text="Screenshot of memory usage for ExpressRoute Traffic Collector." lightbox="./media/expressroute-monitoring-metrics-alerts/memory-usage.png":::
+
+### Count of flow records processed - Split by instances or ExpressRoute circuit
+
+Aggregation type: *Count*  
+
+*Granularity: 5 min*  
+
+You can view the count of number of flow records processed by ExpressRoute Traffic Collector, aggregated across ExpressRoute Circuits. Customer can split the metrics across each ExpressRoute Traffic Collector instance or ExpressRoute circuit when multiple circuits are associated to the ExpressRoute Traffic Collector. Monitoring this metric will help you understand if you need to deploy more ExpressRoute Traffic Collector instances or migrate ExpressRoute circuit association from one ExpressRoute Traffic Collector deployment to another.  
+
+**Guidance:** Splitting by circuits is recommended when multiple ExpressRoute circuits are associated with an ExpressRoute Traffic Collector deployment. This will help determine the flow count of each ExpressRoute circuit and ExpressRoute Traffic Collector utilization by each ExpressRoute circuit. 
+
+:::image type="content" source="./media/expressroute-monitoring-metrics-alerts/flow-records.png" alt-text="Screenshot of average flow records for an ExpressRoute circuit." lightbox="./media/expressroute-monitoring-metrics-alerts/flow-records.png":::
+
 ## Alerts for ExpressRoute gateway connections
 
 1. To set up alerts, go to **Azure Monitor**, then select **Alerts**.

articles/expressroute/expressroute-monitoring-metrics-alerts.md

@@ -0,0 +1,96 @@
+---
+title: 'Configure ExpressRoute Traffic Collector for ExpressRoute Direct using the Azure portal (Preview)'
+description: Learn how to create an ExpressRoute Traffic Collector resource to import logs into a Log Analytics workspace.
+services: expressroute
+author: duongau
+ms.service: expressroute
+ms.topic: tutorial
+ms.date: 07/15/2022
+ms.author: duau
+---
+
+# Configure ExpressRoute Traffic Collector for ExpressRoute Direct using the Azure portal (Preview)
+
+This article will help you deploy an ExpressRoute Traffic Collector using the Azure portal. You'll learn how to add and remove an ExpressRoute Traffic Collector, associate it to an ExpressRoute Direct circuit and Log Analytics workspace. Once the ExpressRoute Traffic Collector is deployed, sampled flow logs will get imported into a Log Analytics workspace. For more information, see [About ExpressRoute Traffic Collector](traffic-collector.md).
+
+> [!IMPORTANT]
+> ExpressRoute Traffic Collector is currently in PREVIEW.
+> See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
+
+## Limitations
+
+- ExpressRoute Traffic Collector supports a maximum ExpressRoute Direct circuit size of 100 Gbps.
+- You can associate up to 20 ExpressRoute Direct circuits with ExpressRoute Traffic Collector as long as the total circuit bandwidth doesn't exceed 100 Gbps.
+
+## Prerequisites
+
+- ExpressRoute Direct circuit with Private or Microsoft peering configured.
+- A Log Analytics workspace (Create new or use existing).
+
+> [!NOTE]
+> - The ExpressRoute Direct circuit, ExpressRoute Traffic Collector and the Log Analytics workspace must be in the same geo-political region. Cross geo-political resource association is not supported.  
+> - The ExpressRoute Direct circuit and ExpressRoute Traffic Collector must be deployed in the same subscription. Cross subscription deployment is currently not available. 
+> - Log Analytics and ExpressRoute Traffic Collector can be deployed in cross subscription.
+> - When ExpressRoute Traffic Collector gets deployed in an Azure region that supports availability zones, it will have availability zone enabled by default.
+
+## Permissions
+
+- Minimum contributor access is required to deploy ExpressRoute Traffic Collector.
+- Minimum contributor access is required to associate ExpressRoute Direct circuit with ExpressRoute Traffic Collector.
+- Monitor contributor role is required to associate Log Analytics workspace with ExpressRoute Traffic Collector.
+
+For more information, see [Identity and access management](../active-directory/fundamentals/active-directory-ops-guide-iam.md).
+
+## Deploy ExpressRoute Traffic Collector
+
+1. Sign in to the [Azure portal](https://portal.azure.com/)
+
+1. In the portal, go to the list of ExpressRoute circuits and select **ExpressRoute Traffic Collectors**. Then select **+ Create new**.
+
+    :::image type="content" source="./media/how-to-configure-traffic-collector/circuit-list.png" alt-text="Screenshot of the create new ExpressRoute Traffic Collector button from the ExpressRoute circuit list page.":::
+
+1. On the **Create an ExpressRoute Traffic Collector** page, enter or select the following information then select **Next**.
+
+    :::image type="content" source="./media/how-to-configure-traffic-collector/basics.png" alt-text="Screenshot of the basics page for create an ExpressRoute Traffic Collector.":::
+
+    | Setting | Description |
+    | --- | --- |
+    | Subscription | Select the subscription to create the ExpressRoute Traffic Collector resource. This resource needs to be in the same subscription as the ExpressRoute Direct circuit. |
+    | Resource group | Select the resource group to deploy this resource into. |
+    | Name | Enter a name to identify this ExpressRoute Traffic Collector resource. |
+    | Region | Select a region to deploy this resource into. This resource needs to be in the same geo-political region as the Log Analytics workspace and the ExpressRoute Direct circuits. |
+    | Collector Policy | This value is automatically filled in as **Default**. |
+
+1. On the **Select ExpressRoute circuit** tab, select **+ Add ExpressRoute Circuits**. Select the checkbox next to the circuit you would like to add to the Traffic Collector and then select **Add**. Once you're satisfied with the circuits added, select **Next**.
+
+    :::image type="content" source="./media/how-to-configure-traffic-collector/select-circuits.png" alt-text="Screenshot of the select ExpressRoute circuits tab and add circuits page.":::
+
+1. On the **Forward Logs** tab, select the checkbox for **Send to Log Analytics workspace**. You can create a new Log Analytics workspace or choose an existing. The workspace can be in a different Azure subscription but has to be in the same geo-political region. Select **Next** once a workspace has been chosen.
+
+    :::image type="content" source="./media/how-to-configure-traffic-collector/forward-logs.png" alt-text="Screenshot of the forward logs tab to Logs Analytics workspace.":::
+
+1. On the **Tags** tab, you can add optional tags for tracking purpose. Select **Next** to review your configuration.
+
+1. Select **Create** once validation has passed to deploy your ExpressRoute Traffic Collector.
+
+    :::image type="content" source="./media/how-to-configure-traffic-collector/validation.png" alt-text="Screenshot of the create validation page.":::
+
+1. Once deployed you should start seeing sampled flow logs within the configure Log Analytics workspace.
+
+    :::image type="content" source="./media/how-to-configure-traffic-collector/log-analytics.png" alt-text="Screenshot of logs in Log Analytics workspace." lightbox="./media/how-to-configure-traffic-collector/log-analytics.png":::
+
+## Clean up resources
+
+To delete the ExpressRoute Traffic Collector resource, you first need to remove all ExpressRoute circuit associations. 
+
+> [!IMPORTANT]
+> If you delete the ExpressRoute Traffic Collector resource before removing all circuit associations, you'll need to wait about 40 mins for the deletion to timeout before you can try again.
+>
+
+Once all circuits have been removed from the ExpressRoute Traffic Collector, select **Delete** from the overview page to remove the resource from your subscription.
+
+:::image type="content" source="./media/how-to-configure-traffic-collector/overview.png" alt-text="Screenshot of delete button on overview page." lightbox="./media/how-to-configure-traffic-collector/overview.png":::
+
+## Next steps
+
+- [ExpressRoute Traffic Collector Metrics](expressroute-monitoring-metrics-alerts.md#expressroute-traffic-collector-metrics)