Merge pull request #270078 from rcdun/ko/ingestion-agent-v2

Update ingestion agent docs for v2.0.0

Author: v-shils

articles/operator-insights/ingestion-agent-configuration-reference.md

@@ -20,27 +20,30 @@ Configuration comprises three parts:
 
 This reference shows two pipelines: one with an MCC EDR source and one with an SFTP pull source.
 
-```
+```yaml
 # A unique identifier for this agent instance. Reserved URL characters must be percent-encoded. It's included in the upload path to the Data Product's input storage account.
 agent_id: agent01 
 # Config for secrets providers. We support reading secrets from Azure Key Vault and from the VM's local filesystem.
 # Multiple secret providers can be defined and each must be given a unique name, which is referenced later in the config.
 # A secret provider of type `key_vault` which contains details required to connect to the Azure Key Vault and allow connection to the Data Product's input storage account. This is always required.
 # A secret provider of type `file_system`, which specifies a directory on the VM where secrets are stored. For example for an SFTP pull source, for storing credentials for connecting to an SFTP server.
 secret_providers: 
-  - name: data_product_keyvault
-    provider:
-      type: key_vault
+  - name: data_product_keyvault_mi
+    key_vault:
+      vault_name: contoso-dp-kv
+      managed_identity:
+        object_id: 22330f5b-4d7e-496d-bbdd-84749eeb009b
+  - name: data_product_keyvault_sp
+    key_vault:
       vault_name: contoso-dp-kv
-      auth:
+      service_principal:
         tenant_id: ad5421f5-99e4-44a9-8a46-cc30f34e8dc7
-        identity_name: 98f3263d-218e-4adf-b939-eacce6a590d2
-        cert_path: /path/to/local/certkey.pkcs
+        client_id: 98f3263d-218e-4adf-b939-eacce6a590d2
+        cert_path: /path/to/local/certficate.p12
   - name: local_file_system
-    provider:
-      # The file system provider specifies a folder in which secrets are stored.
-      # Each secret must be an individual file without a file extension, where the secret name is the file name, and the file contains the secret only.
-      type: file_system
+    # The file system provider specifies a folder in which secrets are stored.
+    # Each secret must be an individual file without a file extension, where the secret name is the file name, and the file contains the secret only.
+    file_system:
       # The absolute path to the secrets directory
       secrets_directory: /path/to/secrets/directory
 pipelines:
@@ -63,22 +66,21 @@ pipelines:
 
 All pipelines require sink config, which covers upload of files to the Data Product's input storage account.
 
-```
+```yaml
 sink:
   # The container within the Data Product's input storage account. This *must* be exactly the name of the container that Azure Operator Insights expects. See the Data Product documentation for what value is required.
   container_name: example-container
   # Optional A string giving an optional base path to use in the container in the Data Product's input storage account. Reserved URL characters must be percent-encoded. See the Data Product for what value, if any, is required.
   base_path: base-path
-  # Optional. How often the sink should refresh its SAS token for the Data Product's input storage account. Defaults to 1h.  Examples: 30s, 10m, 1h, 1d.
-  sas_token_cache_period: 1h
-  auth:
-    type: sas_token
+  sas_token:
     # This must reference a secret provider configured above. 
-    secret_provider: data_product_keyvault 
+    secret_provider: data_product_keyvault_mi
     # The name of a secret in the corresponding provider.
     # This will be the name of a secret in the Key Vault.
     # This is created by the Data Product and should not be changed.
     secret_name: input-storage-sas
+    # Optional. How often the sink should refresh its SAS token for the Data Product's input storage account. Defaults to 1h.  Examples: 30s, 10m, 1h, 1d.
+    cache_period: 1h
   # Optional. The maximum number of blobs that can be uploaded to the Data Product's input storage account in parallel. Further blobs will be queued in memory until an upload completes. Defaults to 10.
   # Note: This value is also the maximum number of concurrent SFTP reads for the SFTP pull source.  Ensure your SFTP server can handle this many concurrent connections.  If you set this to a value greater than 10 and are using an OpenSSH server, you may need to increase `MaxSessions` and/or `MaxStartups` in `sshd_config`.
   maximum_parallel_uploads: 10
@@ -95,7 +97,7 @@ Combining different types of source in one agent instance isn't recommended in p
 
 ### MCC EDR source configuration
 
-```
+```yaml
 source:
   mcc_edrs:
     # The maximum amount of data to buffer in memory before uploading. Units are B, KiB, MiB, GiB, etc.
@@ -128,7 +130,7 @@ This configuration specifies which files are ingested from the SFTP server.
 
 Multiple SFTP pull sources can be defined for one agent instance, where they can reference either different SFTP servers, or different folders on the same SFTP server.
 
-```
+```yaml
 source:
   sftp_pull:
     server: Information relating to the SFTP session.
@@ -140,16 +142,15 @@ source:
       known_hosts_file: /path/to/known_hosts
       # The name of the user on the SFTP server which the agent will use to connect.
       user: sftp-user
-      auth:
+      # The form of authentication to the SFTP server. This can take the values 'password' or 'private_key'. The  appropriate field(s) must be configured below depending on which type is specified.
+      password:
         # The name of the secret provider configured above which contains the secret for the SFTP user.
         secret_provider: local_file_system
-        # The form of authentication to the SFTP server. This can take the values 'password' or 'ssh_key'. The  appropriate field(s) must be configured below depending on which type is specified.
-        type: password
-        # Only for use with 'type: password'. The name of the file containing the password in the secrets_directory folder
+        # Only for use with password authentication. The name of the file containing the password in the secrets_directory folder
         secret_name: sftp-user-password
-        # Only for use with 'type: ssh_key'. The name of the file containing the SSH key in the secrets_directory folder
+        # Only for use with private key authentication. The name of the file containing the SSH key in the secrets_directory folder
         key_secret: sftp-user-ssh-key
-        # Optional. Only for use with 'type: ssh_key'. The passphrase for the SSH key. This can be omitted if the key is not protected by a passphrase.
+        # Optional. Only for use with private key authentication. The passphrase for the SSH key. This can be omitted if the key is not protected by a passphrase.
         passphrase_secret_name: sftp-user-ssh-key-passphrase
     filtering:
       # The path to a folder on the SFTP server that files will be uploaded to Azure Operator Insights from.

articles/operator-insights/ingestion-agent-overview.md

@@ -61,7 +61,7 @@ The ingestion agent is designed to be highly reliable and resilient to low level
 
 The ingestion agent authenticates to two separate systems, with separate credentials.
 
-- To authenticate to the ingestion endpoint of an Azure Operator Insights Data Product, the agent obtains a connection string from an Azure Key Vault. The agent authenticates to this Key Vault with a Microsoft Entra ID service principal and certificate that you setup when you created the agent.
+- To authenticate to the ingestion endpoint of an Azure Operator Insights Data Product, the agent obtains a SAS token from an Azure Key Vault. The agent authenticates to this Key Vault with either a Microsoft Entra ID managed identity or service principal and certificate that you setup when you created the agent.
 - To authenticate to your SFTP server, the agent can use password authentication or SSH key authentication.
 
 For configuration instructions, see [Set up authentication to Azure](set-up-ingestion-agent.md#set-up-authentication-to-azure), [Prepare the VMs](set-up-ingestion-agent.md#prepare-the-vms) and [Configure the agent software](set-up-ingestion-agent.md#configure-the-agent-software).

articles/operator-insights/ingestion-agent-release-notes.md

@@ -15,6 +15,23 @@ The Azure Operator Insights ingestion agent receives improvements on an ongoing
 
 This page is updated for each new release of the ingestion agent, so revisit it regularly. If you're looking for items older than six months, you can find them in [archive for What's new with Azure Operator Insights ingestion agent](ingestion-agent-release-notes-archive.md).
 
+## Version 2.0.0 - March 2024
+
+Download for [RHEL8](https://download.microsoft.com/download/8/2/7/82777410-04a8-4219-a8c8-2f2ea1d239c4/az-aoi-ingestion-2.0.0-1.el8.x86_64.rpm).
+
+### Known issues
+
+None
+
+### New features
+
+- Simplified configuration schema. This is a significant breaking change and requires manual updates to the configuration file in order to upgrade existing agents. See the [configuration reference](./ingestion-agent-configuration-reference.md) for the new schema.
+- Added support for authenticating to the Data Product Key Vault with managed identities.
+
+### Fixed
+
+None
+
 ## Version 1.0.0 - February 2024
 
 Download for [RHEL8](https://download.microsoft.com/download/c/6/c/c6c49e4b-dbb8-4d00-be7f-f6916183b6ac/az-aoi-ingestion-1.0.0-1.el8.x86_64.rpm).

articles/operator-insights/managed-identity.md

@@ -27,18 +27,21 @@ For more general information about managed identities, see [What are managed ide
 
 ## User-assigned managed identities in Azure Operator Insights
 
-Azure Operator Insights Data Products use a user-assigned managed identity for:
+Azure Operator Insights use a user-assigned managed identity for:
 
 - Encryption with customer-managed keys, also called CMK-based encryption.
 - Integration with Microsoft Purview. The managed identity allows the Data Product to manage the collection and the data catalog within the collection.
+- Authentication to Azure with an [ingestion agent](ingestion-agent-overview.md). See [use a managed identity for authentication](set-up-ingestion-agent.md#use-a-managed-identity-for-authentication).
 
 When you [create a Data Product](data-product-create.md), you set up the managed identity and associate it with the Data Product. To use the managed identity with Microsoft Purview, you must also [grant the managed identity the appropriate permissions in Microsoft Purview](purview-setup.md#access-and-set-up-your-microsoft-purview-account).
 
 You use Microsoft Entra ID to manage user-assigned managed identities. For more information, see [Create, list, delete, or assign a role to a user-assigned managed identity using the Azure portal](/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities).
 
 ## System-assigned managed identities in Azure Operator Insights
 
-Azure Operator Insights doesn't support system-assigned managed identities.
+Azure Operator Insights Data Products don't support system-assigned managed identities.
+
+Azure Operator Insights ingestion agents on Azure VMs support system-assigned managed identities. See [Use a managed identity for authentication](set-up-ingestion-agent.md#use-a-managed-identity-for-authentication).
 
 ## Related content
 

articles/operator-insights/rotate-secrets-for-ingestion-agent.md

@@ -14,9 +14,9 @@ ms.date: 02/29/2024
 
 The ingestion agent is a software package that is installed onto a Linux Virtual Machine (VM) owned and managed by you.
 
-It uses a service principal to obtain, from the Data Product's Azure Key Vault, the credentials needed to upload data to the Data Product's input storage account.
+It uses a managed identity or service principal to obtain, from the Data Product's Azure Key Vault, the credentials needed to upload data to the Data Product's input storage account.
 
-You must refresh your service principal credentials before they expire. In this article, you'll rotate the service principal certificates on the ingestion agent.
+If you use a service principal, you must refresh its credentials before they expire. In this article, you'll rotate the service principal certificates on the ingestion agent.
 
 ## Prerequisites
 
@@ -25,7 +25,7 @@ None.
 ## Rotate certificates
 
 1. Create a new certificate, and add it to the service principal. For instructions, refer to [Upload a trusted certificate issued by a certificate authority](/entra/identity-platform/howto-create-service-principal-portal).
-1. Obtain the new certificate and private key in the base64-encoded PKCS12 format, as described in [Set up Ingestion Agents for Azure Operator Insights](set-up-ingestion-agent.md).
+1. Obtain the new certificate and private key in the base64-encoded P12 format, as described in [Set up Ingestion Agents for Azure Operator Insights](set-up-ingestion-agent.md#prepare-certificates-for-the-service-principal).
 1. Copy the certificate to the ingestion agent VM.
 1. Save the existing certificate file and replace with the new certificate file.
 1. Restart the agent.