Skip to content

Commit fea6fc4

Browse files
committed
Add comprehensive WASM development guide and enhance data flow graph documentation
- Add new howto-develop-wasm-modules.md with complete development guide - Cover both WASM module and graph definition development - Include tabbed sections for Rust and Python development paths - Add detailed explanations of timely dataflow architecture - Provide comprehensive SDK reference and build instructions - Explain graph definition structure and registry deployment - Update howto-dataflow-graph-wasm.md with better architectural explanation - Follow Microsoft Writing Style Guide with appropriate bullet point usage
1 parent 4e57711 commit fea6fc4

File tree

2 files changed

+894
-41
lines changed

2 files changed

+894
-41
lines changed

articles/iot-operations/connect-to-cloud/howto-dataflow-graph-wasm.md

Lines changed: 157 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,11 @@ Azure IoT Operations data flow graphs support WebAssembly (WASM) modules for cus
2525
- Deploy an Azure IoT Operations instance on an Arc-enabled Kubernetes cluster. For more information, see [Deploy Azure IoT Operations](../deploy-iot-ops/howto-deploy-iot-operations.md).
2626
- Use Azure Container Registry (ACR) to store WASM modules and graphs.
2727
- Install the ORAS CLI to push WASM modules to the registry.
28+
- To develop custom WASM modules, see [Develop WebAssembly modules for data flow graphs](howto-develop-wasm-modules.md).
2829

2930
## Overview
3031

31-
WebAssembly (WASM) modules in Azure IoT Operations data flow graphs let you process data at the edge with high performance and security. WASM runs in a sandboxed environment and supports programming languages such as Rust, C++, and AssemblyScript.
32+
WebAssembly (WASM) modules in Azure IoT Operations data flow graphs let you process data at the edge with high performance and security. WASM runs in a sandboxed environment and supports Rust and Python.
3233

3334
### How WASM data flow graphs work
3435

@@ -51,7 +52,7 @@ These examples show how to set up and deploy WASM data flow graphs for common sc
5152

5253
Azure IoT Operations needs a container registry to pull WASM modules and graph definitions. You can use Azure Container Registry (ACR) or another OCI-compatible registry.
5354

54-
To create and configure an Azure Container Registry, see [Deploy Azure Container Registry](). <!-- TODO -->
55+
To create and configure an Azure Container Registry, see [Deploy Azure Container Registry](/azure/container-registry/container-registry-get-started-portal).
5556

5657
### Install ORAS CLI
5758

@@ -63,40 +64,37 @@ For this preview, use prebuilt sample modules:
6364

6465
```bash
6566
# Pull sample modules and graphs
66-
oras pull tkmodules.azurecr.io/graph-simple:1.0.0
67-
oras pull tkmodules.azurecr.io/graph-complex:1.0.0
68-
oras pull tkmodules.azurecr.io/temperature:1.0.0
69-
oras pull tkmodules.azurecr.io/window:1.0.0
70-
oras pull tkmodules.azurecr.io/snapshot:1.0.0
71-
oras pull tkmodules.azurecr.io/format:1.0.0
72-
oras pull tkmodules.azurecr.io/humidity:1.0.0
73-
oras pull tkmodules.azurecr.io/collection:1.0.0
74-
oras pull tkmodules.azurecr.io/enrichment:1.0.0
75-
oras pull tkmodules.azurecr.io/filter:1.0.0
67+
oras pull ghcr.io/azure-samples/explore-iot-operations/graph-simple:1.0.0
68+
oras pull ghcr.io/azure-samples/explore-iot-operations/graph-complex:1.0.0
69+
oras pull ghcr.io/azure-samples/explore-iot-operations/temperature:1.0.0
70+
oras pull ghcr.io/azure-samples/explore-iot-operations/window:1.0.0
71+
oras pull ghcr.io/azure-samples/explore-iot-operations/snapshot:1.0.0
72+
oras pull ghcr.io/azure-samples/explore-iot-operations/format:1.0.0
73+
oras pull ghcr.io/azure-samples/explore-iot-operations/humidity:1.0.0
74+
oras pull ghcr.io/azure-samples/explore-iot-operations/collection:1.0.0
75+
oras pull ghcr.io/azure-samples/explore-iot-operations/enrichment:1.0.0
76+
oras pull ghcr.io/azure-samples/explore-iot-operations/filter:1.0.0
7677
```
7778

7879
### Push modules to your registry
7980

8081
```bash
8182
# Log in to your ACR
82-
az acr login --name <your-acr-name>
83+
az acr login --name <YOUR_ACR_NAME>
8384

8485
# Push modules to your registry
85-
oras push <your-acr-name>.azurecr.io/graph-simple:1.0.0 graph-simple.yaml
86-
oras push <your-acr-name>.azurecr.io/graph-complex:1.0.0 graph-complex.yaml
87-
oras push <your-acr-name>.azurecr.io/temperature:1.0.0 temperature-1.0.0.wasm
88-
oras push <your-acr-name>.azurecr.io/window:1.0.0 window-1.0.0.wasm
89-
oras push <your-acr-name>.azurecr.io/snapshot:1.0.0 snapshot-1.0.0.wasm
90-
oras push <your-acr-name>.azurecr.io/format:1.0.0 format-1.0.0.wasm
91-
oras push <your-acr-name>.azurecr.io/humidity:1.0.0 humidity-1.0.0.wasm
92-
oras push <your-acr-name>.azurecr.io/collection:1.0.0 collection-1.0.0.wasm
93-
oras push <your-acr-name>.azurecr.io/enrichment:1.0.0 enrichment-1.0.0.wasm
94-
oras push <your-acr-name>.azurecr.io/filter:1.0.0 filter-1.0.0.wasm
86+
oras push <YOUR_ACR_NAME>.azurecr.io/graph-simple:1.0.0 graph-simple.yaml
87+
oras push <YOUR_ACR_NAME>.azurecr.io/graph-complex:1.0.0 graph-complex.yaml
88+
oras push <YOUR_ACR_NAME>.azurecr.io/temperature:1.0.0 temperature-1.0.0.wasm
89+
oras push <YOUR_ACR_NAME>.azurecr.io/window:1.0.0 window-1.0.0.wasm
90+
oras push <YOUR_ACR_NAME>.azurecr.io/snapshot:1.0.0 snapshot-1.0.0.wasm
91+
oras push <YOUR_ACR_NAME>.azurecr.io/format:1.0.0 format-1.0.0.wasm
92+
oras push <YOUR_ACR_NAME>.azurecr.io/humidity:1.0.0 humidity-1.0.0.wasm
93+
oras push <YOUR_ACR_NAME>.azurecr.io/collection:1.0.0 collection-1.0.0.wasm
94+
oras push <YOUR_ACR_NAME>.azurecr.io/enrichment:1.0.0 enrichment-1.0.0.wasm
95+
oras push <YOUR_ACR_NAME>.azurecr.io/filter:1.0.0 filter-1.0.0.wasm
9596
```
9697

97-
> [!IMPORTANT]
98-
> Update the ACR references in your data flow deployments if you use a different registry from the sample modules.
99-
10098
### Create a registry endpoint
10199

102100
A registry endpoint defines the connection to your container registry. Data flow graphs use registry endpoints to pull WASM modules and graph definitions from container registries. For detailed information about configuring registry endpoints with different authentication methods and registry types, see [Configure registry endpoints](howto-configure-registry-endpoint.md).
@@ -212,12 +210,58 @@ For detailed instructions, see [Assign Azure roles using the Azure portal](/azur
212210

213211
## Example 1: Basic deployment with one WASM module
214212

215-
This scenario shows a simple data flow that uses a WASM module to convert temperature data from Fahrenheit to Celsius. The source code for this module is available [here](). <!--PLACEHOLDER--> Instead of building the module yourself, use the precompiled version that's already pushed to the ACR as `graph-simple:1.0.0` in earlier steps.
213+
This example converts temperature data from Fahrenheit to Celsius using a WASM module. The [temperature module source code](https://github.com/Azure-Samples/explore-iot-operations/tree/wasm/samples/wasm/operators/temperature) is available on GitHub. Use the precompiled version `graph-simple:1.0.0` that you pushed to your container registry.
214+
215+
### How it works
216+
217+
The [graph definition](https://github.com/Azure-Samples/explore-iot-operations/blob/wasm/samples/wasm/graph-simple.yaml) creates a simple three-stage pipeline:
218+
219+
1. **Source**: Receives temperature data from MQTT
220+
2. **Map**: Processes data with the temperature WASM module
221+
3. **Sink**: Sends converted data back to MQTT
216222

217-
<!-- TODO: Add simple graph YAML definition and explanation -->
223+
The graph references the temperature module:
224+
225+
```yaml
226+
operations:
227+
- operationType: "map"
228+
name: "module-temperature/map"
229+
module: "temperature:1.0.0"
230+
```
231+
232+
The [temperature module](https://github.com/Azure-Samples/explore-iot-operations/blob/wasm/samples/wasm/operators/temperature/src/lib.rs) converts Fahrenheit to Celsius using the standard formula `(F - 32) × 5/9 = C`:
233+
234+
```rust
235+
if measurement.unit == MeasurementTemperatureUnit::Fahrenheit {
236+
measurement.value = Some((measurement.value.unwrap() - 32.) * 5. / 9.);
237+
measurement.unit = MeasurementTemperatureUnit::Celsius;
238+
}
239+
```
240+
241+
Input format:
242+
```json
243+
{"temperature": {"value": 100.0, "unit": "F"}}
244+
```
245+
246+
Output format:
247+
```json
248+
{"temperature": {"value": 37.8, "unit": "C"}}
249+
```
250+
251+
The following configuration creates a data flow graph that uses this temperature conversion pipeline. The graph references the `graph-simple:1.0.0` artifact, which contains the YAML definition and pulls the temperature module from your container registry.
218252

219253
### Configure the data flow graph
220254

255+
This configuration defines three nodes that implement the temperature conversion workflow: a source node that subscribes to incoming temperature data, a graph processing node that runs the WASM module, and a destination node that publishes the converted results.
256+
257+
The data flow graph resource "wraps" the graph definition artifact and connects its abstract source/sink operations to concrete endpoints:
258+
259+
- The graph definition's `source` operation connects to the data flow's source node (MQTT topic)
260+
- The graph definition's `sink` operation connects to the data flow's destination node (MQTT topic)
261+
- The graph definition's processing operations run within the graph processing node
262+
263+
This separation allows the same graph definition to be deployed with different endpoints across environments while keeping the processing logic unchanged.
264+
221265
# [Bicep](#tab/bicep)
222266

223267
```bicep
@@ -432,12 +476,85 @@ mosquitto_pub -h aio-broker -p 18883 \
432476

433477
## Example 2: Deploy a complex graph
434478

435-
This scenario shows a more complex data flow graph that processes multiple data sources and includes advanced processing modules.
479+
This example demonstrates a sophisticated data processing workflow that handles multiple data types including temperature, humidity, and image data. The [complex graph definition](https://github.com/Azure-Samples/explore-iot-operations/blob/wasm/samples/wasm/graph-complex.yaml) orchestrates multiple WASM modules to perform advanced analytics and object detection.
480+
481+
### How it works
482+
483+
The complex graph processes three data streams and combines them into enriched sensor analytics:
484+
485+
1. Temperature processing: Converts Fahrenheit to Celsius, filters invalid readings, and calculates statistics
486+
2. Humidity processing: Accumulates humidity measurements over time intervals
487+
3. Image processing: Performs object detection on camera snapshots and formats results
488+
489+
The graph uses specialized modules from the [operators directory](https://github.com/Azure-Samples/explore-iot-operations/tree/wasm/samples/wasm/operators):
490+
491+
- Window module: Delays data for time-based processing
492+
- Temperature modules: Handle conversion, filtering, and statistical analysis
493+
- Humidity module: Processes environmental data
494+
- Snapshot modules: Manage image data routing and object detection
495+
- Format module: Prepares images for processing
496+
- Collection module: Aggregates multi-sensor data
497+
- Enrichment module: Adds metadata and overtemperature alerts
498+
499+
The following diagram shows the data flow through the various processing modules:
436500

437501
<!-- TODO: Add complex graph YAML definition and explanation -->
438502

503+
```mermaid
504+
graph TD
505+
source["source"]
506+
delay["module-window/delay"]
507+
branch_snapshot["module-snapshot/branch"]
508+
branch_temp["module-temperature/branch"]
509+
map_temp["module-temperature/map (F -> C)"]
510+
filter_temp["module-temperature/filter (valid)"]
511+
accumulate_temp["module-temperature/accumulate (max, min, avg, last)"]
512+
accumulate_humidity["module-humidity/accumulate (max, min, avg, last)"]
513+
map_format["module-format/map (image, snapshot)"]
514+
map_snapshot["module-snapshot/map (object detection)"]
515+
accumulate_snapshot["module-snapshot/accumulate (collection)"]
516+
concatenate["concatenate"]
517+
accumulate_collection["module-collection/accumulate"]
518+
map_enrichment["module-enrichment/map (overtemp, topic)"]
519+
sink["sink"]
520+
521+
source --> delay
522+
delay --> branch_snapshot
523+
branch_snapshot -->|sensor data| branch_temp
524+
branch_snapshot -->|snapshot| map_format
525+
map_format --> map_snapshot
526+
map_snapshot --> accumulate_snapshot
527+
accumulate_snapshot --> concatenate
528+
branch_temp -->|temp| map_temp
529+
branch_temp -->|humidity| accumulate_humidity
530+
map_temp --> filter_temp
531+
filter_temp --> accumulate_temp
532+
accumulate_temp --> concatenate
533+
accumulate_humidity --> concatenate
534+
concatenate --> accumulate_collection
535+
accumulate_collection --> map_enrichment
536+
map_enrichment --> sink
537+
```
538+
539+
As shown in the diagram, data flows from a single source through multiple processing stages:
540+
541+
- The window module delays incoming data for time-based processing
542+
- Branch operations route data based on content type (sensor data vs. snapshots)
543+
- Temperature data undergoes conversion, filtering, and statistical accumulation
544+
- Humidity data gets accumulated with statistical analysis
545+
- Image data gets formatted and processed through object detection
546+
- All processed data streams merge at the concatenate operation
547+
- The collection module aggregates the multi-sensor results
548+
- The enrichment module adds final metadata and alerts before output
549+
550+
Branch operations enable parallel processing of different sensor inputs, allowing the graph to handle multiple data types efficiently within a single workflow.
551+
439552
### Configure the complex data flow graph
440553

554+
This configuration implements the multi-sensor processing workflow using the `graph-complex:1.0.0` artifact. Notice how the data flow graph deployment is similar to Example 1 - both use the same three-node pattern (source, graph processor, destination) even though the processing logic is completely different.
555+
556+
This similarity occurs because the data flow graph resource acts as a host environment that loads and executes graph definitions. The actual processing logic resides in the graph definition artifact (`graph-simple:1.0.0` vs `graph-complex:1.0.0`), which contains the YAML specification of operations and connections between WASM modules. The data flow graph resource provides the runtime infrastructure to pull the artifact, instantiate the modules, and route data through the defined workflow.
557+
441558
# [Bicep](#tab/bicep)
442559

443560
```bicep
@@ -489,10 +606,6 @@ resource complexDataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfile
489606
registryEndpointRef: registryEndpointName
490607
artifact: 'graph-complex:1.0.0'
491608
configuration: [
492-
{
493-
key: 'processing-mode'
494-
value: 'enhanced'
495-
}
496609
{
497610
key: 'snapshot_topic'
498611
value: 'sensor/images/raw'
@@ -559,8 +672,6 @@ spec:
559672
registryEndpointRef: <REGISTRY_ENDPOINT_NAME>
560673
artifact: graph-complex:1.0.0
561674
configuration:
562-
- key: processing-mode
563-
value: enhanced
564675
- key: snapshot_topic
565676
value: sensor/images/raw
566677
@@ -649,14 +760,18 @@ while true; do
649760
done
650761
```
651762

652-
## Develop WASM modules
763+
## Develop custom WASM modules
653764

654-
> [!NOTE]
655-
> The VS Code extension for WASM module development is currently in private preview. Contact your Microsoft representative for access.
765+
To create custom data processing logic for your data flow graphs, develop WebAssembly modules in Rust or Python. Custom modules enable you to implement specialized business logic, data transformations, and analytics that aren't available in the built-in operators.
766+
767+
For comprehensive development guidance including:
768+
- Setting up your development environment
769+
- Creating operators in Rust and Python
770+
- Understanding the data model and interfaces
771+
- Building and testing your modules
772+
- Configuring graph definitions with parameters
656773

657-
<!-- TODO: Add WASM module development overview -- ### Use WASM operators -->
658-
<!-- TODO: Add WASM operators documentation content ## Develop graph definition -->
659-
<!-- TODO: Add graph definition development overview -->
774+
See [Develop WebAssembly modules for data flow graphs](howto-develop-wasm-modules.md).
660775

661776
## Configuration reference
662777

@@ -1023,6 +1138,7 @@ For detailed configuration information, see [Configure registry endpoints](howto
10231138

10241139
## Related content
10251140

1141+
- [Develop WebAssembly modules for data flow graphs](howto-develop-wasm-modules.md)
10261142
- [Configure registry endpoints](howto-configure-registry-endpoint.md)
10271143
- [Configure MQTT data flow endpoints](howto-configure-mqtt-endpoint.md)
10281144
- [Configure Azure Event Hubs and Kafka data flow endpoints](howto-configure-kafka-endpoint.md)

0 commit comments

Comments
 (0)