Merge pull request #286498 from PatAltimore/patricka-dataflow-mq-release-aio-m2

AIO dataflow and MQTT changes M2

Author: Jill Grant

articles/iot-operations/.openpublishing.redirection.iot-operations.json

@@ -182,7 +182,7 @@
         },
         {
             "source_path_from_root": "/articles/iot-operations/manage-mqtt-connectivity/howto-manage-secrets.md",
-            "redirect_url": "/azure/iot-operations/manage-mqtt-broker/overview-iot-mq",
+            "redirect_url": "/azure/iot-operations/deploy-iot-ops/howto-manage-secrets",
             "redirect_document_id": false
         },
         {

articles/iot-operations/connect-to-cloud/concept-dataflow-mapping.md

@@ -5,7 +5,7 @@ author: PatAltimore
 ms.author: patricka
 ms.subservice: azure-data-flows
 ms.topic: concept-article
-ms.date: 08/03/2024
+ms.date: 09/24/2024
 ai-usage: ai-assisted
 
 #CustomerIntent: As an operator, I want to understand how to use the dataflow mapping language to transform data.
@@ -58,7 +58,7 @@ The transformations are achieved through *mapping*, which typically involves:
 
 * **Input definition**: Identifying the fields in the input records that are used.
 * **Output definition**: Specifying where and how the input fields are organized in the output records.
-* **Conversion (optional)**: Modifying the input fields to fit into the output fields. Conversion is required when multiple input fields are combined into a single output field.
+* **Conversion (optional)**: Modifying the input fields to fit into the output fields. `expression` is required when multiple input fields are combined into a single output field.
 
 The following mapping is an example:
 
@@ -187,10 +187,10 @@ In the previous example, the path consists of three segments: `Payload`, `Tag.10
 
     ```yaml
     - inputs:
-      - 'Payload.He said: "No. It's done"'
+      - 'Payload.He said: "No. It is done"'
     ```
 
-    In this case, the path is split into the segments `Payload`, `He said: "No`, and `It's done"` (starting with a space).
+    In this case, the path is split into the segments `Payload`, `He said: "No`, and `It is done"` (starting with a space).
     
 ### Segmentation algorithm
 
@@ -205,8 +205,8 @@ Let's consider a basic scenario to understand the use of asterisks in mappings:
 
 ```yaml
 - inputs:
-  - *
-  output: *
+  - '*'
+  output: '*'
 ```
 
 Here's how the asterisk (`*`) operates in this context:
@@ -243,12 +243,12 @@ Mapping configuration that uses wildcards:
 
 ```yaml
 - inputs:
-  - ColorProperties.*
-  output: *
+  - 'ColorProperties.*'
+  output: '*'
 
 - inputs:
-  - TextureProperties.*
-  output: *
+  - 'TextureProperties.*'
+  output: '*'
 ```
 
 Resulting JSON:
@@ -276,6 +276,7 @@ When you place a wildcard, you must follow these rules:
   * **At the beginning:** `*.path2.path3` - Here, the asterisk matches any segment that leads up to `path2.path3`.
   * **In the middle:** `path1.*.path3` - In this configuration, the asterisk matches any segment between `path1` and `path3`.
   * **At the end:** `path1.path2.*` - The asterisk at the end matches any segment that follows after `path1.path2`.
+* The path containing the asterisk must be enclosed in single quotation marks (`'`).
 
 ### Multi-input wildcards
 
@@ -302,10 +303,10 @@ Mapping configuration that uses wildcards:
 
 ```yaml
 - inputs:
-  - *.Max   # - $1
-  - *.Min   # - $2
-  output: ColorProperties.*
-  conversion: ($1 + $2) / 2
+  - '*.Max'   # - $1
+  - '*.Min'   # - $2
+  output: 'ColorProperties.*'
+  expression: ($1 + $2) / 2
 ```
 
 Resulting JSON:
@@ -359,11 +360,11 @@ Initial mapping configuration that uses wildcards:
 
 ```yaml
 - inputs:
-  - *.Max    # - $1
-  - *.Min    # - $2
-  - *.Avg    # - $3
-  - *.Mean   # - $4
-  output: ColorProperties.*
+  - '*.Max'    # - $1
+  - '*.Min'    # - $2
+  - '*.Avg'    # - $3
+  - '*.Mean'   # - $4
+  output: 'ColorProperties.*'
   expression: ($1, $2, $3, $4)
 ```
 
@@ -382,11 +383,11 @@ Corrected mapping configuration:
 
 ```yaml
 - inputs:
-  - *.Max        # - $1
-  - *.Min        # - $2
-  - *.Mid.Avg    # - $3
-  - *.Mid.Mean   # - $4
-  output: ColorProperties.*
+  - '*.Max'        # - $1
+  - '*.Min'        # - $2
+  - '*.Mid.Avg'    # - $3
+  - '*.Mid.Mean'   # - $4
+  output: 'ColorProperties.*'
   expression: ($1, $2, $3, $4)
 ```
 
@@ -398,15 +399,15 @@ When you use the previous example from multi-input wildcards, consider the follo
 
 ```yaml
 - inputs:
-  - *.Max   # - $1
-  - *.Min   # - $2
-  output: ColorProperties.*.Avg
+  - '*.Max'   # - $1
+  - '*.Min'   # - $2
+  output: 'ColorProperties.*.Avg'
   expression: ($1 + $2) / 2
 
 - inputs:
-  - *.Max   # - $1
-  - *.Min   # - $2
-  output: ColorProperties.*.Diff
+  - '*.Max'   # - $1
+  - '*.Min'   # - $2
+  output: 'ColorProperties.*.Diff'
   expression: abs($1 - $2)
 ```
 
@@ -437,9 +438,9 @@ Now, consider a scenario where a specific field needs a different calculation:
 
 ```yaml
 - inputs:
-  - *.Max   # - $1
-  - *.Min   # - $2
-  output: ColorProperties.*
+  - '*.Max'   # - $1
+  - '*.Min'   # - $2
+  output: 'ColorProperties.*'
   expression: ($1 + $2) / 2
 
 - inputs:
@@ -458,9 +459,9 @@ Consider a special case for the same fields to help decide the right action:
 
 ```yaml
 - inputs:
-  - *.Max   # - $1
-  - *.Min   # - $2
-  output: ColorProperties.*
+  - '*.Max'   # - $1
+  - '*.Min'   # - $2
+  output: 'ColorProperties.*'
   expression: ($1 + $2) / 2
 
 - inputs:
@@ -505,8 +506,8 @@ This mapping copies `BaseSalary` from the context dataset directly into the `Emp
 
 ```yaml
 - inputs:
-  - $context(position).*
-  output: Employment.*
+  - '$context(position).*'
+  output: 'Employment.*'
 ```
 
 This configuration allows for a dynamic mapping where every field within the `position` dataset is copied into the `Employment` section of the output record:
@@ -523,13 +524,13 @@ This configuration allows for a dynamic mapping where every field within the `po
 
 ## Last known value
 
-You can track the last known value of a property. Suffix the input field with `?last` to capture the last known value of the field. When a property is missing a value in a subsequent input payload, the last known value is mapped to the output payload.
+You can track the last known value of a property. Suffix the input field with `? $last` to capture the last known value of the field. When a property is missing a value in a subsequent input payload, the last known value is mapped to the output payload.
 
 For example, consider the following mapping:
 
 ```yaml
 - inputs:
-  - Temperature?last
+  - Temperature ? $last
   output: Thermostat.Temperature
 ```
 

articles/iot-operations/connect-to-cloud/howto-configure-adlsv2-endpoint.md

@@ -5,14 +5,16 @@ author: PatAltimore
 ms.author: patricka
 ms.subservice: azure-data-flows
 ms.topic: how-to
-ms.date: 08/27/2024
+ms.date: 10/02/2024
 ai-usage: ai-assisted
 
 #CustomerIntent: As an operator, I want to understand how to configure dataflow endpoints for Azure Data Lake Storage Gen2 in Azure IoT Operations so that I can send data to Azure Data Lake Storage Gen2.
 ---
 
 # Configure dataflow endpoints for Azure Data Lake Storage Gen2
 
+[!INCLUDE [public-preview-note](../includes/public-preview-note.md)]
+
 To send data to Azure Data Lake Storage Gen2 in Azure IoT Operations Preview, you can configure a dataflow endpoint. This configuration allows you to specify the destination endpoint, authentication method, table, and other settings.
 
 ## Prerequisites
@@ -47,7 +49,7 @@ To configure a dataflow endpoint for Azure Data Lake Storage Gen2, we suggest us
           systemAssignedManagedIdentitySettings: {}
     ```
 
-If you need to override the system-assigned managed identity audience, see the [system-assigned managed identity](#system-assigned-managed-identity) section.
+If you need to override the system-assigned managed identity audience, see the [System-assigned managed identity](#system-assigned-managed-identity) section.
 
 ### Use access token authentication
 
@@ -111,7 +113,7 @@ The following authentication methods are available for Azure Data Lake Storage G
 
 Using the system-assigned managed identity is the recommended authentication method for Azure IoT Operations. Azure IoT Operations creates the managed identity automatically and assigns it to the Azure Arc-enabled Kubernetes cluster. It eliminates the need for secret management and allows for seamless authentication with the Azure Data Lake Storage Gen2 account.
 
-Before creating the dataflow endpoint, you need to assign a role to the managed identity that has write permission to the storage account. For example, you can assign the *Storage Blob Data Contributor* role. To learn more about assigning roles to blobs, see [Authorize access to blobs using Microsoft Entra ID](../../storage/blobs/authorize-access-azure-active-directory.md).
+Before creating the dataflow endpoint, assign a role to the managed identity that has write permission to the storage account. For example, you can assign the *Storage Blob Data Contributor* role. To learn more about assigning roles to blobs, see [Authorize access to blobs using Microsoft Entra ID](../../storage/blobs/authorize-access-azure-active-directory.md).
 
 In the *DataflowEndpoint* resource, specify the managed identity authentication method. In most cases, you don't need to specify other settings. Not specifying an audience creates a managed identity with the default audience scoped to your storage account.
 
@@ -136,25 +138,25 @@ datalakeStorageSettings:
 
 Using an access token is an alternative authentication method. This method requires you to create a Kubernetes secret with the SAS token and reference the secret in the *DataflowEndpoint* resource.
 
-1. Get a [SAS token](../../storage/common/storage-sas-overview.md) for an Azure Data Lake Storage Gen2 (ADLSv2) account. For example, use the Azure portal to browse to your storage account. On the left menu, choose **Security + networking** > **Shared access signature**. Use the following table to set the required permissions.
+Get a [SAS token](../../storage/common/storage-sas-overview.md) for an Azure Data Lake Storage Gen2 (ADLSv2) account. For example, use the Azure portal to browse to your storage account. On the left menu, choose **Security + networking** > **Shared access signature**. Use the following table to set the required permissions.
 
-    | Parameter              | Enabled setting             |
-    | ---------------------- | --------------------------- |
-    | Allowed services       | Blob                        |
-    | Allowed resource types | Object, Container           |
-    | Allowed permissions    | Read, Write, Delete, List, Create |
+| Parameter              | Enabled setting             |
+| ---------------------- | --------------------------- |
+| Allowed services       | Blob                        |
+| Allowed resource types | Object, Container           |
+| Allowed permissions    | Read, Write, Delete, List, Create |
 
-1. To enhance security and follow the principle of least privilege, you can generate a SAS token for a specific container. To prevent authentication errors, ensure that the container specified in the SAS token matches the dataflow destination setting in the configuration.
+To enhance security and follow the principle of least privilege, you can generate a SAS token for a specific container. To prevent authentication errors, ensure that the container specified in the SAS token matches the dataflow destination setting in the configuration.
 
-1. Create a Kubernetes secret with the SAS token. Don't include the question mark `?` that might be at the beginning of the token.
+Create a Kubernetes secret with the SAS token. Don't include the question mark `?` that might be at the beginning of the token.
 
 ```bash
 kubectl create secret generic my-sas \
 --from-literal=accessToken='sv=2022-11-02&ss=b&srt=c&sp=rwdlax&se=2023-07-22T05:47:40Z&st=2023-07-21T21:47:40Z&spr=https&sig=<signature>' \
 -n azure-iot-operations
 ```
 
-Finally, create the DataflowEndpoint resource with the secret reference.
+Create the *DataflowEndpoint* resource with the secret reference.
 
 ```yaml
 datalakeStorageSettings:
@@ -168,17 +170,18 @@ datalakeStorageSettings:
 
 To use a user-assigned managed identity, specify the `UserAssignedManagedIdentity` authentication method and provide the `clientId` and `tenantId` of the managed identity.
 
-
 ```yaml
 datalakeStorageSettings:
   authentication:
     method: UserAssignedManagedIdentity
     userAssignedManagedIdentitySettings:
-      clientId: <id>
-      tenantId: <id>
+      clientId: <ID>
+      tenantId: <ID>
 ```
 
-### Batching
+## Advanced settings
+
+You can set advanced settings for the Azure Data Lake Storage Gen2 endpoint, such as the batching latency and message count. 
 
 Use the `batching` settings to configure the maximum number of messages and the maximum latency before the messages are sent to the destination. This setting is useful when you want to optimize for network bandwidth and reduce the number of requests to the destination.
 
@@ -189,9 +192,11 @@ Use the `batching` settings to configure the maximum number of messages and the
 
 For example, to configure the maximum number of messages to 1000 and the maximum latency to 100 seconds, use the following settings:
 
+Set the values in the dataflow endpoint custom resource.
+
 ```yaml
 datalakeStorageSettings:
   batching:
     latencySeconds: 100
     maxMessages: 1000
-```
+```