Merge pull request #266259 from rcdun/aoi_unify_ingestion_agent

Document unified ingestion agent for Azure Operator Insights

Author: ttorble

articles/operator-insights/TOC.yml

@@ -15,55 +15,75 @@
 - name: Concepts
   expanded: false
   items:
+  - name: Quality of Experience - Affirmed MCC Data Products
+    href: concept-mcc-data-product.md
+  - name: Monitoring - Affirmed MCC Data Product
+    href: concept-monitoring-mcc-data-product.md
   - name: Business continuity disaster recovery
     href: business-continuity-disaster-recovery.md
   - name: Data quality and data monitoring
     href: concept-data-quality-monitoring.md
   - name: Data types
     href: concept-data-types.md
   - name: Data visualization
-    href: concept-data-visualization.md 
-  - name: Quality of Experience - Affirmed MCC Data Products
-    href: concept-mcc-data-product.md
-  - name: Monitoring - Affirmed MCC Data Product
-    href: concept-monitoring-mcc-data-product.md
+    href: concept-data-visualization.md
+  - name: Ingestion agents
+    href: ingestion-agent-overview.md
   - name: Managed identity
     href: managed-identity.md
 - name: How-to guides
   expanded: false
   items:
-  - name: Use Microsoft Purview with an Azure Operator Insights Data Product
-    href: purview-setup.md
+  - name: Set up an ingestion agent for uploading data
+    href: set-up-ingestion-agent.md
   - name: Use Azure Operator Insights Data Product dashboards
     href: dashboards-use.md
   - name: Query data in the Azure Operator Insights Data Product
     href: data-query.md
+  - name: Use Microsoft Purview with an Azure Operator Insights Data Product
+    href: purview-setup.md
   - name: Manage permissions to the consumption URL
     href: consumption-plane-configure-permissions.md
-  - name: Deploy Ingestion Agents for MCC protobuf EDRs
+  - name: Manage Azure Operator Insights ingestion agents
     items:
-    - name: Create and configure MCC EDR Ingestion Agents
-      href: how-to-install-mcc-edr-agent.md
-    - name: Manage MCC EDR Ingestion Agents
-      href: how-to-manage-mcc-edr-agent.md
-    - name: Monitor and troubleshoot MCC EDR Ingestion Agents
-      href: troubleshoot-mcc-edr-agent.md
-    - name: MCC EDR Ingestion Agent configuration reference
-      href: mcc-edr-agent-configuration.md
-  - name: Deploy Ingestion Agents for SFTP files
-    items:
-    - name: Overview
-      href: sftp-agent-overview.md
-    - name: Create and configure SFTP Ingestion Agents
-      href: how-to-install-sftp-agent.md
-    - name: Manage SFTP Ingestion Agents
-      href: how-to-manage-sftp-agent.md
-    - name: Monitor and troubleshoot SFTP Ingestion Agents
-      href: troubleshoot-sftp-agent.md
-    - name: SFTP Ingestion Agent configuration reference
-      href: sftp-agent-configuration.md
+    - name: Change configuration
+      href: change-ingestion-agent-configuration.md
+    - name: Monitor and troubleshoot
+      href: monitor-troubleshoot-ingestion-agent.md
+    - name: Upgrade
+      href: upgrade-ingestion-agent.md
+    - name: Rotate secrets
+      href: rotate-secrets-for-ingestion-agent.md
+    - name: Map Microsoft hostnames to IP addresses
+      href: map-hostnames-ip-addresses.md
+    - name: Configuration reference
+      href: ingestion-agent-configuration-reference.md
   - name: Monitor Azure Operator Insights
     href: monitor-operator-insights.md
+  - name: Manage legacy ingestion agents
+    items:
+    - name: MCC EDR Ingestion Agents
+      items:
+      - name: Create and configure MCC EDR Ingestion Agents
+        href: how-to-install-mcc-edr-agent.md
+      - name: Manage MCC EDR Ingestion Agents
+        href: how-to-manage-mcc-edr-agent.md
+      - name: Monitor and troubleshoot MCC EDR Ingestion Agents
+        href: troubleshoot-mcc-edr-agent.md
+      - name: MCC EDR Ingestion Agent configuration reference
+        href: mcc-edr-agent-configuration.md
+    - name: SFTP ingestion agents
+      items:
+      - name: Overview
+        href: sftp-agent-overview.md
+      - name: Create and configure SFTP Ingestion Agents
+        href: how-to-install-sftp-agent.md
+      - name: Manage SFTP Ingestion Agents
+        href: how-to-manage-sftp-agent.md
+      - name: Monitor and troubleshoot SFTP Ingestion Agents
+        href: troubleshoot-sftp-agent.md
+      - name: SFTP Ingestion Agent configuration reference
+        href: sftp-agent-configuration.md
 - name: Reference
   items:
     - name: Monitoring data

articles/operator-insights/change-ingestion-agent-configuration.md

@@ -0,0 +1,58 @@
+---
+title: Change configuration for ingestion agents for Azure Operator Insights
+description: Learn how to make and roll back configuration changes for Azure Operator Insights ingestion agents.
+author: rcdun
+ms.author: rdunstan
+ms.reviewer: sergeyche
+ms.service: operator-insights
+ms.topic: how-to
+ms.date: 02/29/2024
+
+#CustomerIntent: As a someone managing an agent that has already been set up, I want to update its configuration so that data products in Azure Operator Insights receive the correct data.
+---
+
+# Change configuration for Azure Operator Insights ingestion agents
+
+The ingestion agent is a software package that is installed onto a Linux Virtual Machine (VM) owned and managed by you. You might need to change the agent configuration.
+
+In this article, you'll change your ingestion agent configuration and roll back a configuration change.
+
+## Prerequisites
+
+- Using the documentation for your data product, check for required or recommended configuration for the ingestion agent.
+- See [Configuration reference for Azure Operator Insights ingestion agent](ingestion-agent-configuration-reference.md) for full details of the configuration options.
+
+## Update agent configuration
+
+> [!WARNING]
+> Changing the configuration requires restarting the agent. For the MCC EDR source, a small number of EDRs being handled might be dropped.  It is not possible to gracefully restart without dropping any data. For safety, update agents one at a time, only updating the next when you are sure the previous was successful.
+
+> [!WARNING]
+> If you change the pipeline ID for an SFTP pull source, the agent treats it as a new source and might upload duplicate files with the new pipeline ID. To avoid this, add the `exclude_before_time` parameter to the file source configuration. For example, if you configure `exclude_before_time: "2024-01-01T00:00:00-00:00"` then any files last modified before midnight on January 1, 2024 UTC will be ignored by the agent.
+
+If you need to change the agent's configuration, carry out the following steps.
+
+1. Save a copy of the existing */etc/az-aoi-ingestion/config.yaml* configuration file.
+1. Edit the configuration file to change the config values.
+1. Restart the agent.
+    ```
+    sudo systemctl restart az-aoi-ingestion.service
+    ```
+
+## Roll back configuration changes
+
+If a configuration change fails:
+
+1. Copy the backed-up configuration file from before the change to the */etc/az-aoi-ingestion/config.yaml* file.
+1. Restart the agent.
+    ```
+    sudo systemctl restart az-aoi-ingestion.service
+    ```
+
+## Related content
+
+Learn how to:
+
+- [Monitor and troubleshoot ingestion agents](monitor-troubleshoot-ingestion-agent.md).
+- [Upgrade ingestion agents](upgrade-ingestion-agent.md).
+- [Rotate secrets for ingestion agents](rotate-secrets-for-ingestion-agent.md).

articles/operator-insights/concept-mcc-data-product.md

@@ -35,8 +35,8 @@ The data produced by the MCC varies according to the functionality. This variati
 
 The following data types are provided for all Quality of Experience - Affirmed MCC Data Products.
 
-- `edr`: This data type handles EDRs from the MCC.
-- `edr-sanitized`: This data type contains the same information as `edr` but with personal data suppressed to support operators' compliance with privacy legislation.
+- `edr` contains data from the Event Data Records (EDRs) written by the MCC network elements. EDRs record each significant event arising during calls or sessions handled by the MCC. They provide a comprehensive record of what happened, allowing operators to explore both individual problems and more general patterns.
+- `edr-sanitized` contains data from the `edr` data type but with personal data suppressed. Sanitized data types can be used to support data analysis while also enforcing subscriber privacy.
 - `edr-validation`: This data type contains a subset of performance management statistics and provides you with the ability to optionally ingest a minimum number of PMstats tables for a data quality check.
 - `device`: This optional data type contains device data (for example, device model, make and capabilities) that the Data Product can use to enrich the MCC Event Data Records. To use this data type, you must upload the device reference data in a CSV file. The CSV must conform to the [Device reference schema for the Quality of Experience Affirmed MCC Data Product](device-reference-schema.md).
 - `enrichment`: This data type holds the enriched Event Data Records and covers multiple sub data types for precomputed aggregations targeted to accelerate specific dashboards, granularities, and queries. These multiple sub data types include:
@@ -52,7 +52,74 @@ The following data types are provided for all Quality of Experience - Affirmed M
 To use the Quality of Experience - Affirmed MCC Data Product:
 
 1. Deploy the Data Product by following [Create an Azure Operator Insights Data Product](data-product-create.md).
-1. Configure your network to provide data by setting up an MCC EDR Ingestion Agent. The MCC EDR Ingestion Agent uploads EDRs from your network to Azure Operator Insights. See [Create and configure MCC EDR Ingestion Agents for Azure Operator Insights](how-to-install-mcc-edr-agent.md). Alternatively, you can provide your own ingestion agent.
+1. Configure your network to provide data by setting up an Azure Operator Insights ingestion agent on a virtual machine (VM).
+
+    1. Read [Requirements for the Azure Operator Insights ingestion agent](#requirements-for-the-azure-operator-insights-ingestion-agent).
+    1. [Install the Azure Operator Insights ingestion agent and configure it to upload data](set-up-ingestion-agent.md).
+
+    Alternatively, you can provide your own ingestion agent.
+
+## Requirements for the Azure Operator Insights ingestion agent
+
+Use the VM requirements to set up a suitable VM for the ingestion agent. Use the example configuration to configure the ingestion agent to upload data to the Data Product, as part of following [Install the Azure Operator Insights ingestion agent and configure it to upload data](set-up-ingestion-agent.md).
+
+### VM requirements
+
+Each agent instance must run on its own Linux VM. The number of VMs needed depends on the scale and redundancy characteristics of your deployment. This recommended specification can achieve 1.5-Gbps throughput on a standard D4s_v3 Azure VM. For any other VM spec, we recommend that you measure throughput at the network design stage.
+
+Latency on the MCC to agent connection can negatively affect throughput. Latency should usually be low if the MCC and agent are colocated or the agent runs in an Azure region close to the MCC.
+
+Talk to the Affirmed Support Team to determine your requirements.
+
+Each VM running the agent must meet the following minimum specifications.
+
+| Resource | Requirements                                                        |
+|----------|---------------------------------------------------------------------|
+| OS       | Red Hat Enterprise Linux 8.6 or later, or Oracle Linux 8.8 or later |
+| vCPUs    | 4                                                                   |
+| Memory   | 32 GB                                                               |
+| Disk     | 64 GB                                                               |
+| Network  | Connectivity from MCCs and to Azure                                 |
+| Software | systemd, logrotate, and zip installed                               |
+| Other    | SSH or alternative access to run shell commands                     |
+| DNS      | (Preferable) Ability to resolve Microsoft hostnames. If not, you need to perform extra configuration when you set up the agent (described in [Map Microsoft hostnames to IP addresses for ingestion agents that can't resolve public hostnames](map-hostnames-ip-addresses.md).) |
+
+#### Deploying multiple VMs for fault tolerance
+
+The ingestion agent is designed to be highly reliable and resilient to low levels of network disruption. If an unexpected error occurs, the agent restarts and provides service again as soon as it's running.
+
+The agent doesn't buffer data, so if a persistent error or extended connectivity problems occur, EDRs are dropped.
+
+For extra fault tolerance, you can deploy multiple instances of the ingestion agent and configure the MCC to switch to a different instance if the original instance becomes unresponsive, or to share EDR traffic across a pool of agents. For more information, see the [Affirmed Networks Active Intelligent vProbe System Administration Guide](https://manuals.metaswitch.com/vProbe/latest/vProbe_System_Admin/Content/02%20AI-vProbe%20Configuration/Generating_SESSION__BEARER__FLOW__and_HTTP_Transac.htm) (only available to customers with Affirmed support) or speak to the Affirmed Networks Support Team.
+
+### Required agent configuration
+
+Use the information in this section when [setting up the agent and configuring the agent software](set-up-ingestion-agent.md#configure-the-agent-software).
+
+The ingestion agent must use MCC EDRs as a data source.
+
+|Information | Configuration setting for Azure Operator Ingestion agent  | Value  |
+|---------|---------|---------|
+|Container in the Data Product input storage account |`sink.container_name` | `edr` |
+
+> [!IMPORTANT]
+> `sink.container_name` must be set exactly as specified here. You can change other configuration to meet your requirements.
+
+For more information about all the configuration options, see [Configuration reference for Azure Operator Insights ingestion agent](ingestion-agent-configuration-reference.md).
+
+### Configure Affirmed MCCs
+
+Once the agents are installed and running, configure the MCCs to send EDRs to them.
+
+1. Follow the steps under "Generating SESSION, BEARER, FLOW, and HTTP Transaction EDRs" in the [Affirmed Networks Active Intelligent vProbe System Administration Guide](https://manuals.metaswitch.com/vProbe/latest/vProbe_System_Admin/Content/02%20AI-vProbe%20Configuration/Generating_SESSION__BEARER__FLOW__and_HTTP_Transac.htm) (only available to customers with Affirmed support), making the following changes:
+
+    - Replace the IP addresses of the MSFs in MCC configuration with the IP addresses of the VMs running the ingestion agents.
+
+    - Confirm that the following EDR server parameters are set.
+
+        - port: 36001
+        - encoding: protobuf
+        - keep-alive: 2 seconds
 
 ## Related content