Merge pull request #290384 from jlian/release-aio-ga

Docs scrub

Author: denrea

articles/iot-operations/connect-to-cloud/concept-dataflow-conversions.md

@@ -5,7 +5,7 @@ author: PatAltimore
 ms.author: patricka
 ms.subservice: azure-data-flows
 ms.topic: concept-article
-ms.date: 10/30/2024
+ms.date: 11/11/2024
 
 #CustomerIntent: As an operator, I want to understand how to use dataflow conversions to transform data.
 ms.service: azure-iot-operations
@@ -84,7 +84,7 @@ In this example, the conversion results in an array containing the values of `[M
 
 ## Data types
 
-Different serialization formats support various data types. For instance, JSON offers a few primitive types: string, number, Boolean, and null. Also included are arrays of these primitive types. In contrast, other serialization formats like Avro have a more complex type system, including integers with multiple bit field lengths and timestamps with different resolutions. Examples are milliseconds and microseconds.
+Different serialization formats support various data types. For instance, JSON offers a few primitive types: string, number, Boolean, and null. It also includes arrays of these primitive types.
 
 When the mapper reads an input property, it converts it into an internal type. This conversion is necessary for holding the data in memory until it's written out into an output field. The conversion to an internal type happens regardless of whether the input and output serialization formats are the same.
 
@@ -105,11 +105,7 @@ The internal representation utilizes the following data types:
 
 ### Input record fields
 
-When an input record field is read, its underlying type is converted into one of these internal type variants. The internal representation is versatile enough to handle most input types with minimal or no conversion. However, some input types require conversion or are unsupported. Some examples:
-
-* **Avro** `UUID` **type**: It's converted to a `string` because there's no specific `UUID` type in the internal representation.
-* **Avro** `decimal` **type**: It isn't supported by the mapper, so fields of this type can't be included in mappings.
-* **Avro** `duration` **type**: Conversion can vary. If the `months` field is set, it's unsupported. If only `days` and `milliseconds` are set, it's converted to the internal `duration` representation.
+When an input record field is read, its underlying type is converted into one of these internal type variants. The internal representation is versatile enough to handle most input types with minimal or no conversion.
 
 For some formats, surrogate types are used. For example, JSON doesn't have a `datetime` type and instead stores `datetime` values as strings formatted according to ISO8601. When the mapper reads such a field, the internal representation remains a string.
 
@@ -126,10 +122,6 @@ The mapper is designed to be flexible by converting internal types into output t
   * Converted to `0`/`1` if the output field is numerical.
   * Converted to `true`/`false` if the output field is string.
 
-### Explicit type conversions
-
-Although the automatic conversions operate as you might expect based on common implementation practices, there are instances where the right conversion can't be determined automatically and results in an *unsupported* error. To address these situations, several conversion functions are available to explicitly define how data should be transformed. These functions provide more control over how data is converted and help maintain data integrity even when automatic methods fall short.
-
 ### Use a conversion formula with types
 
 In mappings, an optional formula can specify how data from the input is processed before being written to the output field. If no formula is specified, the mapper copies the input field to the output by using the internal type and conversion rules.
@@ -186,29 +178,6 @@ expression: 'min($1)'
 
 This configuration selects the smallest value from the `Measurements` array for the output field.
 
-It's also possible to use functions that result in a new array:
-
-# [Bicep](#tab/bicep)
-
-```bicep
-inputs: [
-  'Measurements' // - $1
-]
-output: 'Measurements'
-expression: 'take($1, 10)'  // taking at max 10 items
-```
-
-# [Kubernetes (preview)](#tab/kubernetes)
-
-```yaml
-- inputs:
-  - Measurements # - $1
-  output: Measurements
-  expression: take($1, 10)  # taking at max 10 items
-```
-
----
-
 Arrays can also be created from multiple single values:
 
 # [Bicep](#tab/bicep)
@@ -302,17 +271,22 @@ The `conversion` uses the `if` function that has three parameters:
 
 ## Available functions
 
-Functions can be used in the conversion formula to perform various operations:
+Dataflows provide a set of built-in functions that can be used in conversion formulas. These functions can be used to perform common operations like arithmetic, comparison, and string manipulation. The available functions are:
 
-* `min` to select a single item from an array
-* `if` to select between values
-* String manipulation (for example, `uppercase()`)
-* Explicit conversion (for example, `ISO8601_datetime`)
-* Aggregation (for example, `avg()`)
+| Function | Description | Examples |
+|----------|-------------|---------|
+| `min`    | Return the minimum value from an array. | `min(2, 3, 1)` returns `1`, `min($1)` returns the minimum value from the array `$1` |
+| `max`    | Return the maximum value from an array. | `max(2, 3, 1)` returns `3`, `max($1)` returns the maximum value from the array `$1` |
+| `if`     | Return between values based on a condition. | `if($1 > 10, 'High', 'Low')` returns `'High'` if `$1` is greater than `10`, otherwise `'Low'` |
+| `len`    | Return the character length of a string or the number of elements in a tuple. | `len("Azure")` returns `5`, `len(1, 2, 3)` returns `3`, `len($1)` returns the number of elements in the array `$1` |
+| `floor`  | Return the largest integer less than or equal to a number. | `floor(2.9)` returns `2` |
+| `round`  | Return the nearest integer to a number, rounding half-way cases away from 0.0. | `round(2.5)` returns `3` |
+| `ceil`   | Return the smallest integer greater than or equal to a number. | `ceil(2.1)` returns `3` |
+| `scale`  | Scale a value from one range to another. | `scale($1, 0, 10, 0, 100)` scales the input value from the range 0 to 10 to the range 0 to 100 |
 
-## Available operations
+### Conversion functions
 
-Dataflows offer a wide range of out-of-the-box conversion functions that allow users to easily perform unit conversions without the need for complex calculations. These predefined functions cover common conversions such as temperature, pressure, length, weight, and volume. The following list shows the available conversion functions, along with their corresponding formulas and function names:
+Dataflows provide several built-in conversion functions for common unit conversions like temperature, pressure, length, weight, and volume. Here are some examples:
 
 | Conversion | Formula | Function name |
 | --- | --- | --- |
@@ -323,7 +297,7 @@ Dataflows offer a wide range of out-of-the-box conversion functions that allow u
 | Lbs to kg | Kg = lbs * 0.453592 | `lbToKg` |
 | Gallons to liters | Liters = gallons * 3.78541 | `galToL` |
 
-In addition to these unidirectional conversions, we also support the reverse calculations:
+Reverse conversions are also supported:
 
 | Conversion | Formula | Function name |
 | --- | --- | --- |
@@ -334,13 +308,7 @@ In addition to these unidirectional conversions, we also support the reverse cal
 | Kg to lbs | Lbs = kg / 0.453592 | `kgToLb` |
 | Liters to gallons | Gallons = liters / 3.78541 | `lToGal` |
 
-These functions are designed to simplify the conversion process. They allow users to input values in one unit and receive the corresponding value in another unit effortlessly.
-
-We also provide a scaling function to scale the range of value to the user-defined range. For the example `scale($1,0,10,0,100)`, the input value is scaled from the range 0 to 10 to the range 0 to 100.
-
-Moreover, users have the flexibility to define their own conversion functions by using simple mathematical formulas. Our system supports basic operators such as addition (`+`), subtraction (`-`), multiplication (`*`), and division (`/`). These operators follow standard rules of precedence. For example, multiplication and division are performed before addition and subtraction. Precedence can be adjusted by using parentheses to ensure the correct order of operations. This capability empowers users to customize their unit conversions to meet specific needs or preferences, enhancing the overall utility and versatility of the system.
-
-For more complex calculations, functions like `sqrt` (which finds the square root of a number) are also available.
+Additionally, you can define your own conversion functions using basic mathematical formulas. The system supports operators like addition (`+`), subtraction (`-`), multiplication (`*`), and division (`/`). These operators follow standard rules of precedence, which can be adjusted using parentheses to ensure the correct order of operations. This allows you to customize unit conversions to meet specific needs.
 
 ## Available operators by precedence
 

articles/iot-operations/connect-to-cloud/concept-dataflow-mapping.md

@@ -5,7 +5,7 @@ author: PatAltimore
 ms.author: patricka
 ms.subservice: azure-data-flows
 ms.topic: concept-article
-ms.date: 10/30/2024
+ms.date: 11/11/2024
 ai-usage: ai-assisted
 
 #CustomerIntent: As an operator, I want to understand how to use the dataflow mapping language to transform data.
@@ -118,11 +118,22 @@ The example maps:
 
 Field references show how to specify paths in the input and output by using dot notation like `Employee.DateOfBirth` or accessing data from a contextual dataset via `$context(position)`.
 
-### MQTT user properties
+### MQTT and Kafka metadata properties
 
-When you use MQTT as a source or destination, you can access MQTT user properties in the mapping language. User properties can be mapped in the input or output. 
+When you use MQTT or Kafka as a source or destination, you can access various metadata properties in the mapping language. These properties can be mapped in the input or output.
 
-In the following example, the MQTT `topic` property is mapped to the `origin_topic` field in the output. 
+#### Metadata properties
+
+* **Topic**: Works for both MQTT and Kafka. It contains the string where the message was published. Example: `$metadata.topic`.
+* **User property**: In MQTT, this refers to the free-form key/value pairs an MQTT message can carry. For example, if the MQTT message was published with a user property with key "priority" and value "high", then the `$metadata.user_property.priority` reference hold the value "high". User property keys can be arbitrary strings and may require escaping: `$metadata.user_property."weird key"` uses the key "weird key" (with a space).
+* **System property**: This term is used for every property that is not a user property. Currently, only a single system property is supported: `$metadata.system_property.content_type`, which reads the content type property of the MQTT message (if set).
+* **Header**: This is the Kafka equivalent of the MQTT user property. Kafka can use any binary value for a key, but dataflow supports only UTF-8 string keys. Example: `$metadata.header.priority`. This functionality is similar to user properties.
+
+#### Mapping metadata properties
+
+##### Input mapping
+
+In the following example, the MQTT `topic` property is mapped to the `origin_topic` field in the output:
 
 # [Bicep](#tab/bicep)
 
@@ -143,7 +154,30 @@ output: origin_topic
 
 ---
 
-You can also map MQTT properties to an output header. In the following example, the MQTT `topic` is mapped to the `origin_topic` field in the output's user property:
+If the user property `priority` is present in the MQTT message, the following example demonstrates how to map it to an output field:
+
+# [Bicep](#tab/bicep)
+
+```bicep
+inputs: [
+  '$metadata.user_property.priority'
+]
+output: 'priority'
+```
+
+# [Kubernetes (preview)](#tab/kubernetes)
+
+```yaml
+inputs:
+  - $metadata.user_property.priority
+output: priority
+```
+
+---
+
+##### Output mapping
+
+You can also map metadata properties to an output header or user property. In the following example, the MQTT `topic` is mapped to the `origin_topic` field in the output's user property:
 
 # [Bicep](#tab/bicep)
 
@@ -164,6 +198,48 @@ output: $metadata.user_property.origin_topic
 
 ---
 
+If the incoming payload contains a `priority` field, the following example demonstrates how to map it to an MQTT user property:
+
+# [Bicep](#tab/bicep)
+
+```bicep
+inputs: [
+  'priority'
+]
+output: '$metadata.user_property.priority'
+```
+
+# [Kubernetes (preview)](#tab/kubernetes)
+
+```yaml
+inputs:
+  - priority
+output: $metadata.user_property.priority
+```
+
+---
+
+The same example for Kafka:
+
+# [Bicep](#tab/bicep)
+
+```bicep
+inputs: [
+  'priority'
+]
+output: '$metadata.header.priority'
+```
+
+# [Kubernetes (preview)](#tab/kubernetes)
+
+```yaml
+inputs:
+  - priority
+output: $metadata.header.priority
+```
+
+---
+
 ## Contextualization dataset selectors
 
 These selectors allow mappings to integrate extra data from external databases, which are referred to as *contextualization datasets*.
@@ -371,6 +447,7 @@ output: '*'
 ```
 
 ---
+This configuration shows a basic mapping where every field in the input is directly mapped to the same field in the output without any changes. The asterisk (`*`) serves as a wildcard that matches any field in the input record.
 
 Here's how the asterisk (`*`) operates in this context:
 
@@ -379,8 +456,6 @@ Here's how the asterisk (`*`) operates in this context:
 * **Captured segment**: The portion of the path that the asterisk matches is referred to as the `captured segment`.
 * **Output mapping**: In the output configuration, the `captured segment` is placed where the asterisk appears. This means that the structure of the input is preserved in the output, with the `captured segment` filling the placeholder provided by the asterisk.
 
-This configuration demonstrates the most generic form of mapping, where every field in the input is directly mapped to a corresponding field in the output without modification.
-
 Another example illustrates how wildcards can be used to match subsections and move them together. This example effectively flattens nested structures within a JSON object.
 
 Original JSON:
@@ -454,9 +529,9 @@ Resulting JSON:
 
 When you place a wildcard, you must follow these rules:
 
-* **Single asterisk per dataDestination:** Only one asterisk (`*`) is allowed within a single path.
+* **Single asterisk per data reference:** Only one asterisk (`*`) is allowed within a single data reference.
 * **Full segment matching:** The asterisk must always match an entire segment of the path. It can't be used to match only a part of a segment, such as `path1.partial*.path3`.
-* **Positioning:** The asterisk can be positioned in various parts of `dataDestination`:
+* **Positioning:** The asterisk can be positioned in various parts of a data reference:
   * **At the beginning:** `*.path2.path3` - Here, the asterisk matches any segment that leads up to `path2.path3`.
   * **In the middle:** `path1.*.path3` - In this configuration, the asterisk matches any segment between `path1` and `path3`.
   * **At the end:** `path1.path2.*` - The asterisk at the end matches any segment that follows after `path1.path2`.
@@ -643,7 +718,7 @@ When you use the previous example from multi-input wildcards, consider the follo
     '*.Min'   // - $2
   ]
   output: 'ColorProperties.*.Diff'
-  expression: 'abs($1 - $2)'
+  expression: '$1 - $2'
 }
 ```
 
@@ -660,7 +735,7 @@ When you use the previous example from multi-input wildcards, consider the follo
   - '*.Max'   # - $1
   - '*.Min'   # - $2
   output: 'ColorProperties.*.Diff'
-  expression: abs($1 - $2)
+  expression: $1 - $2
 ```
 
 ---
@@ -752,6 +827,7 @@ Consider a special case for the same fields to help decide the right action:
     'Opacity.Max'   // - $1
     'Opacity.Min'   // - $2
   ]
+  output: ''
 }
 ```
 

articles/iot-operations/connect-to-cloud/howto-configure-dataflow-profile.md

@@ -6,7 +6,7 @@ ms.author: patricka
 ms.service: azure-iot-operations
 ms.subservice: azure-data-flows
 ms.topic: how-to
-ms.date: 10/30/2024
+ms.date: 11/11/2024
 
 #CustomerIntent: As an operator, I want to understand how to I can configure a a dataflow profile to control a dataflow behavior.
 ---
@@ -23,6 +23,8 @@ The most important setting is the instance count, which determines the number of
 
 By default, a dataflow profile named "default" is created when Azure IoT Operations is deployed. This dataflow profile has a single instance count. You can use this dataflow profile to get started with Azure IoT Operations.
 
+Currently, when using the [operations experience portal](https://iotoperations.azure.com/), the default dataflow profile is used for all dataflows.
+
 # [Bicep](#tab/bicep)
 
 ```bicep
@@ -105,17 +107,13 @@ You can scale the dataflow profile to adjust the number of instances that run th
 
 Scaling can also improve the resiliency of the dataflows by providing redundancy in case of failures.
 
-To manually scale the dataflow profile, specify the maximum number of instances you want to run. For example, to set the instance count to 3:
+To manually scale the dataflow profile, specify the number of instances you want to run. For example, to set the instance count to 3:
 
 # [Bicep](#tab/bicep)
 
 ```bicep
-resource dataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2024-11-01' = {
-  parent: aioInstance
-  name: '<NAME>'
-  properties: {
-    instanceCount: 3
-  }
+properties: {
+  instanceCount: 3
 }
 ```
 

articles/iot-operations/connect-to-cloud/howto-configure-fabric-endpoint.md

@@ -6,7 +6,7 @@ ms.author: patricka
 ms.service: azure-iot-operations
 ms.subservice: azure-data-flows
 ms.topic: how-to
-ms.date: 11/04/2024
+ms.date: 11/11/2024
 ai-usage: ai-assisted
 
 #CustomerIntent: As an operator, I want to understand how to configure dataflow endpoints for Microsoft Fabric OneLake in Azure IoT Operations so that I can send data to Microsoft Fabric OneLake.
@@ -116,7 +116,7 @@ apiVersion: connectivity.iotoperations.azure.com/v1
 kind: DataflowEndpoint
 metadata:
   name: <ENDPOINT_NAME>
-  namespace: azure-iotoperations
+  namespace: azure-iot-operations
 spec:
   endpointType: FabricOneLake
   fabricOneLakeSettings:
@@ -330,4 +330,4 @@ fabricOneLakeSettings:
 
 ## Next steps
 
-To learn more about dataflows, see [Create a dataflow](howto-create-dataflow.md).
+To learn more about dataflows, see [Create dataflow](howto-create-dataflow.md).