You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/iot-operations/connect-to-cloud/howto-dataflow-graph-wasm.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,8 +24,8 @@ Azure IoT Operations data flow graphs support WebAssembly (WASM) modules for cus
24
24
25
25
- Deploy an Azure IoT Operations instance on an Arc-enabled Kubernetes cluster. For more information, see [Deploy Azure IoT Operations](../deploy-iot-ops/howto-deploy-iot-operations.md).
26
26
- Use Azure Container Registry (ACR) to store WASM modules and graphs.
27
-
- Install the ORAS CLI to push WASM modules to the registry.
28
-
-To develop custom WASM modules, see[Develop WebAssembly modules for data flow graphs](howto-develop-wasm-modules.md).
27
+
- Install the OCI Registry As Storage (ORAS) CLI to push WASM modules to the registry.
28
+
-Develop custom WASM modules by following guidance in[Develop WebAssembly modules for data flow graphs](howto-develop-wasm-modules.md).
29
29
30
30
## Overview
31
31
@@ -551,7 +551,7 @@ Branch operations enable parallel processing of different sensor inputs, allowin
551
551
552
552
### Configure the complex data flow graph
553
553
554
-
This configuration implements the multi-sensor processing workflow using the `graph-complex:1.0.0` artifact. Notice how the data flow graph deployment is similar to Example 1 - both use the same three-node pattern (source, graph processor, destination) even though the processing logic is completely different.
554
+
This configuration implements the multi-sensor processing workflow using the `graph-complex:1.0.0` artifact. Notice how the data flow graph deployment is similar to Example 1 - both use the same three-node pattern (source, graph processor, destination) even though the processing logic is different.
555
555
556
556
This similarity occurs because the data flow graph resource acts as a host environment that loads and executes graph definitions. The actual processing logic resides in the graph definition artifact (`graph-simple:1.0.0` vs `graph-complex:1.0.0`), which contains the YAML specification of operations and connections between WASM modules. The data flow graph resource provides the runtime infrastructure to pull the artifact, instantiate the modules, and route data through the defined workflow.
557
557
@@ -855,7 +855,7 @@ spec:
855
855
856
856
### Disk persistence
857
857
858
-
Disk persistence allows data flow graphs to maintain state across restarts. When enabled, the graph can recover processing state if pods are restarted. This is useful for stateful processing scenarios where losing intermediate data would be problematic.
858
+
Disk persistence allows data flow graphs to maintain state across restarts. When enabled, the graph can recover processing state if pods are restarted. The storage capability is useful for stateful processing scenarios where losing intermediate data would be problematic.
859
859
860
860
The setting accepts `Enabled` or `Disabled` (case-insensitive), with `Disabled` as the default.
861
861
@@ -1047,7 +1047,7 @@ For storage destinations like Azure Data Lake or Fabric OneLake, you can specify
1047
1047
1048
1048
Node connections define the data flow path between nodes. Each connection specifies a source node and destination node, creating the processing pipeline. Connections can optionally include schema validation to ensure data integrity between processing stages.
1049
1049
1050
-
When you specify schema validation, the system validates data format and structure as it flows between nodes. This helps catch data inconsistencies early and ensures WASM modules receive data in the expected format.
1050
+
When you specify schema validation, the system validates data format and structure as it flows between nodes. The validation helps catch data inconsistencies early and ensures WASM modules receive data in the expected format.
1051
1051
1052
1052
# [Bicep](#tab/bicep)
1053
1053
@@ -1131,7 +1131,7 @@ Storage endpoints typically require output schema settings to define data serial
1131
1131
1132
1132
#### Registry endpoints
1133
1133
1134
-
Registry endpoints provide access to container registries for pulling WASM modules and graph definitions. They're not used directly in data flow but are referenced by graph processing nodes.
1134
+
Registry endpoints provide access to container registries for pulling WASM modules and graph definitions. They're not used directly in data flow but graph processing nodes reference them.
1135
1135
1136
1136
For detailed configuration information, see [Configure registry endpoints](howto-configure-registry-endpoint.md).
Copy file name to clipboardExpand all lines: articles/iot-operations/connect-to-cloud/howto-develop-wasm-modules.md
+33-29Lines changed: 33 additions & 29 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,11 +18,7 @@ ai-usage: ai-assisted
18
18
>
19
19
> See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for legal terms that apply to Azure features that are in beta, preview, or not yet released into general availability.
20
20
21
-
This article shows you how to develop custom WebAssembly (WASM) modules and graph definitions for Azure IoT Operations data flow graphs. You can create modules in Rust or Python to implement custom data processing logic, and define graph configurations that specify how your modules connect into complete processing workflows.
22
-
23
-
## Prerequisites
24
-
25
-
Complete the setup in [Use WebAssembly with data flow graphs](howto-dataflow-graph-wasm.md) to understand the basic concepts and deployment process. Choose your development language: [Rust](#tab/rust) or [Python](#tab/python). You also need access to a container registry for storing your compiled modules.
21
+
This article shows you how to develop custom WebAssembly (WASM) modules and graph definitions for Azure IoT Operations data flow graphs. You can create modules in Rust or Python to implement custom processing logic. You can also define graph configurations that specify how your modules connect into complete processing workflows.
26
22
27
23
## Overview
28
24
@@ -40,19 +36,23 @@ Azure IoT Operations data flow graphs process streaming data through configurabl
40
36
41
37
Data flow graphs build on the [Timely dataflow](https://docs.rs/timely/latest/timely/dataflow/operators/index.html) computational model, which originated from Microsoft Research's Naiad project. This approach ensures:
42
38
43
-
- Deterministic processing: Same input always produces the same output
44
-
- Progress tracking: The system knows when computations are complete
Traditional stream processing systems face challenges with out-of-order data where events arrive later than expected, partial results where you don't know when computations are complete, and coordination issues when synchronizing distributed processing.
45
+
Traditional stream processing systems face challenges with several issues. Out-of-order data can cause events to arrive later than expected. Partial results make it difficult to know when computations are complete. Coordination issues arise when synchronizing distributed processing.
50
46
51
-
Timely dataflow solves these problems through:
47
+
Timely dataflow solves problems through:
52
48
53
49
#### Timestamps and progress tracking
54
50
55
-
Every data item carries a timestamp representing its logical time. The system tracks progress through these timestamps, enabling deterministic processing where same input always produces same output, exactly-once semantics where there's no duplicate or missed processing, and watermarks that let you know when no more data will arrive for a given time.
51
+
Every data item carries a timestamp representing its logical time. The system tracks progress through timestamps, enabling several key capabilities:
52
+
53
+
-**Deterministic processing**: Same input always produces same output
54
+
-**Exactly once semantics**: No duplicate or missed processing
55
+
-**Watermarks**: Know when no more data will arrive for a given time
This ensures causal ordering where effects follow causes, progress guarantees where the system knows when processing is complete, and distributed coordination where multiple nodes stay synchronized.
67
+
The hybrid logical clock approach ensures several capabilities:
68
+
69
+
-**Causal ordering**: Effects follow causes
70
+
-**Progress guarantees**: The system knows when processing is complete
Python development uses componentize-py with WebAssembly Interface Types (WIT) for code generation. No additional environment configuration is required beyond installing the prerequisites.
209
+
Python development uses componentize-py with WebAssembly Interface Types (WIT) for code generation. No other environment configuration is required beyond installing the prerequisites.
206
210
207
211
---
208
212
@@ -242,7 +246,7 @@ Create a Python file for your operator. The filename should match your module na
242
246
touch temperature_converter.py
243
247
```
244
248
245
-
Python WASM modules don't require additional project configuration files. The module structure is defined through the Python class that implements the operator interface.
249
+
Python WASM modules don't require other project configuration files. The Python class that implements the operator interface defines the module structure.
246
250
247
251
---
248
252
@@ -473,7 +477,7 @@ Use local builds for fastest iteration during development and when you need full
473
477
474
478
### Docker build
475
479
476
-
Build using a containerized environment with all dependencies pre-installed:
480
+
Build using a containerized environment with all dependencies preinstalled:
477
481
478
482
```bash
479
483
# Build release version (default)
@@ -510,7 +514,7 @@ Use local builds for fastest iteration during development and when you need to d
510
514
511
515
### Docker build
512
516
513
-
Build using a containerized environment with schema paths pre-configured:
517
+
Build using a containerized environment with schema paths preconfigured:
514
518
515
519
```bash
516
520
# Build release version (app-name should match your Python filename without .py extension)
@@ -537,7 +541,7 @@ For comprehensive examples of map, filter, branch, accumulate, and delay operato
537
541
-**Delay operators**: Time-based processing control
538
542
-**Complex workflows**: Multi-operator configurations with state management
539
543
540
-
For a complete implementation example, see the [branch module](https://github.com/Azure-Samples/explore-iot-operations/tree/wasm/samples/wasm/rust/examples/branch) which demonstrates parameter usage for conditional routing logic.
544
+
For a complete implementation example, see the [branch module](https://github.com/Azure-Samples/explore-iot-operations/tree/wasm/samples/wasm/rust/examples/branch), which demonstrates parameter usage for conditional routing logic.
541
545
542
546
# [Python](#tab/python)
543
547
@@ -651,14 +655,14 @@ graph LR
651
655
652
656
### Registry deployment
653
657
654
-
Both graph definitions and WASM modules must be uploaded to a container registry as OCI artifacts before they can be referenced by data flow graphs:
658
+
Both graph definitions and WASM modules must upload to a container registry as Open Container Initiative (OCI) artifacts before data flow graphs can reference them:
655
659
656
660
- **Graph definitions**: Packaged as OCI artifacts with media type `application/vnd.oci.image.config.v1+json`
657
661
- **WASM modules**: Packaged as OCI artifacts containing the compiled WebAssembly binary
658
-
- **Versioning**: Use semantic versioning (e.g., `my-graph:1.0.0`, `temperature-converter:2.1.0`) for proper dependency management
662
+
- **Versioning**: Use semantic versioning (such as `my-graph:1.0.0`, `temperature-converter:2.1.0`) for proper dependency management
659
663
- **Registry support**: Compatible with Azure Container Registry, Docker Hub, and other OCI-compliant registries
660
664
661
-
This separation enables reusable logic where the same graph definition can be deployed with different endpoints, environment independence where development, staging, and production can use different data sources, and modular deployment where you can update endpoint configurations without changing processing logic.
665
+
The separation enables reusable logic where the same graph definition deploys with different endpoints. It provides environment independence where development, staging, and production use different data sources. It also supports modular deployment where you update endpoint configurations without changing processing logic.
662
666
663
667
For detailed instructions on uploading graph definitions and WASM modules to registries, see [Use WebAssembly with data flow graphs](howto-dataflow-graph-wasm.md).
For a complete implementation example, see the [branch module](https://github.com/Azure-Samples/explore-iot-operations/tree/wasm/samples/wasm/rust/examples/branch) which demonstrates parameter usage for conditional routing logic.
729
+
For a complete implementation example, see the [branch module](https://github.com/Azure-Samples/explore-iot-operations/tree/wasm/samples/wasm/rust/examples/branch), which demonstrates parameter usage for conditional routing logic.
726
730
727
731
## Next steps
728
732
729
-
- For complete examples and advanced patterns, see the [Azure IoT Operations WASM samples](https://github.com/Azure-Samples/explore-iot-operations/tree/wasm/samples/wasm) repository.
733
+
- See complete examples and advanced patterns in the [Azure IoT Operations WASM samples](https://github.com/Azure-Samples/explore-iot-operations/tree/wasm/samples/wasm) repository.
730
734
- Learn how to deploy your modules in [Use WebAssembly with data flow graphs](howto-dataflow-graph-wasm.md).
731
735
- Configure your data flow endpoints in [Configure data flow endpoints](howto-configure-dataflow-endpoint.md).
0 commit comments