You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/iot-operations/connect-to-cloud/howto-create-dataflow.md
+29-8Lines changed: 29 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -60,6 +60,15 @@ To create a dataflow in the operations experience portal, select **Dataflow** >
60
60
61
61
# [Bicep](#tab/bicep)
62
62
63
+
This Bicep template file from [Bicep File to create Dataflow](https://github.com/Azure-Samples/explore-iot-operations/blob/main/samples/quickstarts/dataflow.bicep) deploys the necessary resources for dataflows.
64
+
65
+
1. Download the file to your local, and make sure to replace the values for `customLocationName`, `aioInstanceName`, `schemaRegistryName`, `opcuaSchemaName`, and `persistentVCName`.
66
+
67
+
1. Next, deploy the resources using the [az stack group](/azure/azure-resource-manager/bicep/deployment-stacks?tabs=azure-powershell) command in your terminal:
68
+
69
+
```azurecli
70
+
az stack group create --name MyDeploymentStack --resource-group $RESOURCE_GROUP --template-file /workspaces/explore-iot-operations/<filename>.bicep --action-on-unmanage 'deleteResources' --deny-settings-mode 'none' --yes
71
+
```
63
72
The overall structure of a dataflow configuration for Bicep is as follows:
64
73
65
74
```bicep
@@ -171,10 +180,10 @@ Configuring an asset as a source is only available in the operations experience
171
180
172
181
# [Bicep](#tab/bicep)
173
182
174
-
To configure a source using an MQTT endpoint, use the following configuration:
183
+
The MQTT endpoint is configured as a source in the Bicep template file, using the following configuration
175
184
176
185
```bicep
177
-
{
186
+
{
178
187
operationType: 'Source'
179
188
sourceSettings: {
180
189
endpointRef: defaultDataflowEndpoint.name
@@ -187,11 +196,12 @@ To configure a source using an MQTT endpoint, use the following configuration:
187
196
188
197
Datasources allow you to specify multiple MQTT or Kafka topics without needing to modify the endpoint configuration. This means the same endpoint can be reused across multiple dataflows, even if the topics vary. To learn more, see [Reuse dataflow endpoints](./howto-configure-dataflow-endpoint.md#reuse-endpoints).
189
198
190
-
#### Specify schema to deserialize data
199
+
<!-- TODO: Put the right article link here -->
200
+
For more information about creating an MQTT endpoint as a dataflow source, see [MQTT Endpoint](concept-schema-registry.md).
191
201
192
-
If the source data has optional fields or fields with different types, specify a deserialization schema to ensure consistency. For example, the data might have fields that aren't present in all messages. Without the schema, the transformation can't handle these fields as they would have empty values. With the schema, you can specify default values or ignore the fields.
202
+
#### Specify schema to deserialize data
193
203
194
-
The following configuration demonstrates how to define a schema in your Bicep file. This schema will ensure proper deserialization of asset data. In this example, the schema defines fields such as `asset_id`, `asset_name`, `location`, `temperature`, `manufacturer`, `production_date`, and `serial_number`. Each field is assigned a specific data type (e.g., `string`) and marked as non-nullable. This ensures all incoming messages contain these fields with valid data. Such structure maintains consistency and enables the system to handle structured input more reliably.
204
+
Schemas are documents that describe the format of a message and its contents to enable processing and contextualization. You can upload schemas using the ARM/Bicep templates. The following configuration demonstrates how to define a schema in your Bicep file. In this example, the schema defines fields such as `asset_id`, `asset_name`, `location`, `temperature`, `manufacturer`, `production_date`, and `serial_number`. Each field is assigned a specific data type (e.g., `string`) and marked as non-nullable. This ensures all incoming messages contain these fields with valid data.
195
205
196
206
```bicep
197
207
var assetDeltaSchema = '''
@@ -254,9 +264,6 @@ Once the schema is registered, it can be referenced in transformations to ensure
254
264
}
255
265
```
256
266
257
-
> [!NOTE]
258
-
> The only supported serialization format is Delta or Parquet. The schema is optional.
259
-
260
267
For more information about schema registry, see [Understand message schemas](concept-schema-registry.md).
261
268
262
269
# [Kubernetes](#tab/kubernetes)
@@ -341,6 +348,20 @@ In the operations experience portal, select **Dataflow** > **Add transform (opti
0 commit comments