Merge pull request #283508 from PatAltimore/patricka-dataflows-release-aio-july-updates

Dataflow updates

Author: AnnaMHuff

articles/iot-operations/connect-to-cloud/concept-dataflow-conversions.md

@@ -5,16 +5,18 @@ author: PatAltimore
 ms.author: patricka
 ms.subservice: azure-data-flows
 ms.topic: concept-article
-ms.date: 08/01/2024
+ms.date: 08/03/2024
 
 #CustomerIntent: As an operator, I want to understand how to use dataflow conversions to transform data.
 ---
 
 # Convert data using dataflow conversions
 
+[!INCLUDE [public-preview-note](../includes/public-preview-note.md)]
+
 You can use dataflow conversions to transform data in Azure IoT Operations. The *conversion* element in a dataflow is used to compute values for output fields. You can use input fields, available operations, data types, and type conversions in dataflow conversions.
 
-Dataflow *conversion* element is used to compute values for output fields:
+The dataflow *conversion* element is used to compute values for output fields:
 
 ```yaml
 - inputs:
@@ -167,7 +169,7 @@ This mapping creates an array containing the minimum, maximum, average, and mean
 * Handling missing fields in the input by providing an alternative value.
 * Conditionally removing a field based on its presence.
 
-Example mapping using *missing value*`:
+Example mapping using *missing value*:
 
 ```json
 {
@@ -217,7 +219,7 @@ Functions can be used in the conversion formula to perform various operations.
 
 ## Available operations
 
-Dataflow offers a wide range of out-of-the-box (OOTB) conversion functions that allow users to easily perform unit conversions without the need for complex calculations. These predefined functions cover common conversions such as temperature, pressure, length, weight, and volume. The following is a list of the available conversion functions, along with their corresponding formulas and function names:
+Dataflows offer a wide range of out-of-the-box (OOTB) conversion functions that allow users to easily perform unit conversions without the need for complex calculations. These predefined functions cover common conversions such as temperature, pressure, length, weight, and volume. The following is a list of the available conversion functions, along with their corresponding formulas and function names:
 
 | Conversion | Formula | Function Name |
 | --- | --- | --- |

articles/iot-operations/connect-to-cloud/concept-dataflow-enrich.md

@@ -5,13 +5,15 @@ author: PatAltimore
 ms.author: patricka
 ms.subservice: azure-data-flows
 ms.topic: concept-article
-ms.date: 08/01/2024
+ms.date: 08/03/2024
 
 #CustomerIntent: As an operator, I want to understand how to create a dataflow to enrich data sent to endpoints.
 ---
 
 # Enrich data using dataflows
 
+[!INCLUDE [public-preview-note](../includes/public-preview-note.md)]
+
 You can enrich data using the *contextualization datasets* function. When processing incoming records, these datasets can be queried based on conditions that relate to the fields of the incoming record. This capability allows for dynamic interactions where data from these datasets can be used to supplement information in the output fields and participate in complex calculations during the mapping process.
 
 For example, consider the following dataset with a few records, represented as JSON records:

articles/iot-operations/connect-to-cloud/concept-dataflow-mapping.md

@@ -5,7 +5,7 @@ author: PatAltimore
 ms.author: patricka
 ms.subservice: azure-data-flows
 ms.topic: concept-article
-ms.date: 08/02/2024
+ms.date: 08/03/2024
 
 #CustomerIntent: As an operator, I want to understand how to use the dataflow mapping language to transform data.
 ---
@@ -20,7 +20,7 @@ Mapping allows you to transform data from one format to another. Consider the fo
 
 ```json
 {
-  "Name": "John Doe",
+  "Name": "Grace Owens",
   "Place of birth": "London, TX",
   "Birth Date": "19840202",
   "Start Date": "20180812",
@@ -34,7 +34,7 @@ Compare it with the output record:
 ```json
 {
   "Employee": {
-    "Name": "John Doe",
+    "Name": "Grace Owens",
     "Date of Birth": "19840202"
   },
   "Employment": {
@@ -104,7 +104,7 @@ Dot-notation is widely used in computer science to reference fields, even recurs
   - Person.Address.Street.Number
 ```
 
-However, in a dataflow a path described by dot-notation might include strings and some special characters without needing  escaping:
+However, in a dataflow, a path described by dot-notation might include strings and some special characters without needing  escaping:
 
 ```yaml
 - inputs:
@@ -120,7 +120,7 @@ However, in other cases, escaping is necessary:
 
 The previous example, among other special characters, contains dots within the field name, which, without escaping, would serve as a separator in the dot-notation itself.
 
-While dataflow parses a path, it treats only two characters as special:
+While a dataflow parses a path, it treats only two characters as special:
 
 * Dots ('.') act as field separators.
 * Quotes, when placed at the beginning or the end of a segment, start an escaped section where dots aren't treated as field separators.
@@ -450,12 +450,12 @@ Consider a special case for the same fields to help deciding the right action:
 
 An empty `output` field in the second definition implies not writing the fields in the output record (effectively removing `Opacity`). This setup is more of a `Specialization` than a `Second Rule`.
 
-Resolution of overlapping mappings by dataflow:
+Resolution of overlapping mappings by dataflows:
 
 * The evaluation progresses from the top rule in the mapping definition.
 * If a new mapping resolves to the same fields as a previous rule, the following applies:
   * A `Rank` is calculated for each resolved input based on the number of segments the wildcard captures. For instance, if the `Captured Segments` are `Properties.Opacity`, the `Rank` is 2. If only `Opacity`, the `Rank` is 1. A mapping without wildcards has a `Rank` of 0.
-  * If the `Rank` of the latter rule is equal to or higher than the previous rule, dataflow treats it as a `Second Rule`.
+  * If the `Rank` of the latter rule is equal to or higher than the previous rule, a dataflow treats it as a `Second Rule`.
   * Otherwise, it treats the configuration as a `Specialization`.
 
 For example, the mapping that directs `Opacity.Max` and `Opacity.Min` to an empty output has a `Rank` of zero. Since the second rule has a lower `Rank` than the previous, it's considered a specialization and overrides the previous rule, which would calculate a value for `Opacity`

articles/iot-operations/connect-to-cloud/howto-configure-dataflow-endpoint.md

@@ -1,11 +1,11 @@
 ---
-title: Configure dataflow endpoints
+title: Configure dataflow endpoints in Azure IoT Operations
 description: Configure dataflow endpoints to create connection points for data sources.
 author: PatAltimore
 ms.author: patricka
 ms.subservice: azure-data-flows
-ms.topic: conceptual
-ms.date: 07/26/2024
+ms.topic: how-to
+ms.date: 08/03/2024
 
 #CustomerIntent: As an operator, I want to understand how to configure source and destination endpoints so that I can create a dataflow.
 ---
@@ -23,19 +23,22 @@ kind: DataflowEndpoint
 metadata:
   name: <endpoint-name>
 spec:
-  endpointType: <endpointType> # mqtt, kafka, dataExplorer, dataLakeStorage, fabricOneLake, or localStorage
+  endpointType: <endpointType> # mqtt, kafka, or localStorage
   authentication:
     method: <method> # systemAssignedManagedIdentity, x509Credentials, userAssignedManagedIdentity, or serviceAccountToken
     systemAssignedManagedIdentitySettings: # Required if method is systemAssignedManagedIdentity
       audience: https://eventgrid.azure.net
-    x509CredentialsSettings: # Required if method is x509Credentials
-      certificateSecretName: x509-certificate
-    userAssignedManagedIdentitySettings: # Required if method is userAssignedManagedIdentity
-      clientId: <id>
-      tenantId: <id>
-      audience: https://eventgrid.azure.net
-    serviceAccountTokenSettings: # Required if method is serviceAccountToken
-      audience: my-audience
+    ### OR
+    # x509CredentialsSettings: # Required if method is x509Credentials
+    #  certificateSecretName: x509-certificate
+    ### OR
+    # userAssignedManagedIdentitySettings: # Required if method is userAssignedManagedIdentity
+    #  clientId: <id>
+    #  tenantId: <id>
+    #  audience: https://eventgrid.azure.net
+    ### OR
+    # serviceAccountTokenSettings: # Required if method is serviceAccountToken
+    #  audience: my-audience
   mqttSettings: # Required if endpoint type is mqtt
     host: example.westeurope-1.ts.eventgrid.azure.net:8883
     tls: # Omit for no TLS or MQTT.
@@ -304,85 +307,6 @@ spec:
 
 ## Endpoint types for destinations only
 
-### Azure Data Lake (ADLSv2)
-
-Azure Data Lake endpoints are used for Azure Data Lake destinations. You can configure the endpoint, authentication, table, and other settings.
-
-```yaml
-apiVersion: connectivity.iotoperations.azure.com/v1beta1
-kind: DataflowEndpoint
-metadata:
-  name: adls
-spec:
-  endpointType: dataLakeStorage
-  authentication:
-    method: systemAssignedManagedIdentity
-    systemAssignedManagedIdentitySettings: {}
-  datalakeStorageSettings:
-    host: example.blob.core.windows.net
-```
-
-Other supported authentication method is SAS tokens or user-assigned managed identity.
-
-```yaml
-spec:
-  authentication:
-    method: accessToken
-    accessTokenSecretRef: <your access token secret name>
-    # OR
-    userAssignedManagedIdentitySettings:
-      clientId: <id>
-      tenantId: <id>
-```
-
-You can also configure batching latency, max bytes, and max messages.
-
-```yaml
-spec:
-  endpointType: dataLakeStorage
-  datalakeStorageSettings:
-    batching:
-      latencyMs: 100
-      maxBytes: 1000000
-      maxMessages: 1000
-```
-
-### Azure Data Explorer (ADX)
-
-Azure Data Explorer endpoints are used for Azure Data Explorer destinations. You can configure the endpoint, authentication, and other settings.
-
-```yaml
-apiVersion: connectivity.iotoperations.azure.com/v1beta1
-kind: DataflowEndpoint
-metadata:
-  name: adx
-spec:
-  endpointType: dataExplorer
-  authentication:
-    method: systemAssignedManagedIdentity
-    systemAssignedManagedIdentitySettings: {}
-    # OR
-    method: userAssignedManagedIdentity
-    userAssignedManagedIdentitySettings:
-      clientId: <id>
-      tenantId: <id>
-  dataExplorerSettings:
-    host: example.westeurope.kusto.windows.net
-    database: example-database
-```
-
-Again, you can configure batching latency, max bytes, and max messages.
-
-```yaml
-spec:
-  endpointType: dataExplorer
-  dataExplorerSettings:
-    batching:
-      latencyMs: 100
-      maxBytes: 1000000
-      maxMessages: 1000
-```
-
 ### Local storage and Edge Storage Accelerator
 
 Use the local storage option to send data to a locally available persistent volume, through which you can upload data via Edge Storage Accelerator (ESA) edge volumes. In this case, the format must be parquet.

articles/iot-operations/connect-to-cloud/howto-configure-dataflow-profile.md

@@ -1,13 +1,13 @@
 ---
 title: Configure dataflow profile in Azure IoT Operations
-description: How to configure dataflow profile in Azure IoT Operations to change dataflow behavior.
+description: How to configure a dataflow profile in Azure IoT Operations to change a dataflow behavior.
 author: PatAltimore
 ms.author: patricka
 ms.subservice: azure-data-flows
-ms.topic: conceptual
-ms.date: 07/25/2024
+ms.topic: how-to
+ms.date: 08/03/2024
 
-#CustomerIntent: As an operator, I want to understand how to I can configure a dataflow profile to control dataflow behavior.
+#CustomerIntent: As an operator, I want to understand how to I can configure a a dataflow profile to control a dataflow behavior.
 ---
 
 # Configure dataflow profile
@@ -22,7 +22,7 @@ kind: DataflowProfile
 metadata:
   name: my-dataflow-profile
 spec:
-  maxInstances: 4
+  instanceCount: 1
   tolerations:
     ...
   diagnostics:
@@ -47,7 +47,7 @@ spec:
 
 | Field Name                                      | Description                                                                 |
 |-------------------------------------------------|-----------------------------------------------------------------------------|
-| `maxInstances`                                  | Number of instances to spread the dataflows across. Optional; automatically determined if not set. |
+| `instanceCount`                                  | Number of instances to spread the dataflow across. Optional; automatically determined if not set. Currently in the preview release, set the value to `1`. |
 | `tolerations`                                   | Node tolerations. Optional; see [Kubernetes Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/). |
 | `diagnostics`                                   | Diagnostics settings.                                                       |
 | `diagnostics.logFormat`                         | Format of the logs. For example, `text`.                                           |