Data flow graph public preview (#136)

# Azure IoT Operations Data Flow Graphs: WASM Development (Public
Preview)

## Overview

This PR introduces **Azure IoT Operations Data Flow Graphs** - a
powerful new capability for real-time data processing using WebAssembly
(WASM) modules. This public preview enables developers to build custom
data processing pipelines using Rust and Python, with support for
complex workflows including branching, filtering, aggregation, and
machine learning inference.

## 🚀 Key Features

### **Multi-Language WASM Development**
- **Rust SDK**: Full-featured SDK with procedural macros, logging,
metrics, and state store APIs
- **Python Support**: componentize-py integration with generated
bindings
- **Docker Builders**: Streamlined containerized build environments for
both languages
- **Local Development**: Complete toolchain support for native
development workflows

### **Rich Operator Types**
- **Map**: Transform data (unit conversion, format changes)
- **Filter**: Conditional data filtering based on predicates
- **Branch**: Route data to different paths based on conditions
- **Accumulate**: Time-windowed aggregation and statistics
- **Delay**: Control timing and batch processing
- **Source/Sink**: Integration with external systems

### **Production-Ready Architecture**
- **Timely Dataflow**: Built on Microsoft Research's proven
computational model
- **Event-Time Semantics**: Process data based on when events occurred
- **Hybrid Logical Clock**: Ensures causal ordering and progress
guarantees
- **Fault Tolerance**: Built-in support for handling failures and
ensuring data consistency

## 📁 What's Included

### **Sample Applications**
- graph-simple.yaml - Basic temperature conversion pipeline
- `graph-complex.yaml` - Multi-sensor processing with ML inference
- **10+ Rust Examples**: Temperature, humidity, image processing, data
enrichment
- **Python Examples**: Map, filter, branch operators with complete
implementations

### **Development Tools**
- **Docker Builders**: 
  - `ghcr.io/azure-samples/explore-iot-operations/rust-wasm-builder`
  - `ghcr.io/azure-samples/explore-iot-operations/python-wasm-builder`
- **GitHub Actions**: Automated builder image publishing
- **WIT Schema**: Complete WebAssembly Interface Type definitions

### **Sample Data & Assets**
- Temperature and humidity sensor payloads
- Sample images for computer vision workflows
- Pre-trained ONNX models (MobileNet, SqueezeNet)

## 🔧 Technical Highlights

### **Rust Development**
```rust
#[map_operator(init = "temperature_converter_init")]
fn temperature_converter(input: DataModel) -> DataModel {
    // Transform Fahrenheit to Celsius with full type safety
}
```

### **Python Development**  
```python
class Map(exports.Map):
    def process(self, message: types.DataModel) -> types.DataModel:
        # Process data with generated type bindings
```

### **Graph Configuration**
```yaml
operations:
  - operationType: "map"
    name: "temperature/map"
    module: "temperature:1.0.0"
```

## 🛠️ Build & Deploy

**Docker Builds (Recommended)**:
```bash
# Rust
docker run --rm -v "$(pwd):/workspace" ghcr.io/azure-samples/explore-iot-operations/rust-wasm-builder --app-name my-module

# Python  
docker run --rm -v "$(pwd):/workspace" ghcr.io/azure-samples/explore-iot-operations/python-wasm-builder --app-name my_module --app-type map
```

**Local Development**: Full support for native Rust/Python toolchains
with registry configuration

## 📊 Impact

- **81 files changed, 10,835 insertions**: Comprehensive feature
implementation
- **Production-ready**: Based on proven academic research and production
systems
- **Developer-friendly**: Multiple language support with excellent
tooling
- **Scalable**: Distributed processing with automatic coordination

## 🔗 Integration

This feature integrates with:
- **Azure IoT Operations**: Native deployment and management
- **Azure Container Registry**: WASM module and graph storage
- **Kubernetes**: Arc-enabled cluster deployment
- **ORAS**: OCI artifact distribution

## 📚 Documentation

- Comprehensive README with quick start guide
- Complete Rust and Python development guides  
- Docker builder documentation
- Sample workflows and deployment instructions

## ✅ Testing

- All Docker builders tested and validated
- GitHub Actions workflows for automated publishing
- Sample modules verified with both build methods
- End-to-end validation completed

Author: jlian

.github/workflows/wasm-python-build.yml

@@ -0,0 +1,57 @@
+name: Build and Publish WASM Python Builder Image
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [ main, wasm ]
+    paths:
+      - 'samples/wasm/python/Dockerfile'
+      - 'samples/wasm/python/schema/**'
+      - 'samples/wasm/python/inject_pdb.py'
+  pull_request:
+    branches: [ main ]
+    paths:
+      - 'samples/wasm/python/Dockerfile'
+      - 'samples/wasm/python/schema/**'
+      - 'samples/wasm/python/inject_pdb.py'
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+
+    - name: Set up Docker Buildx
+      uses: docker/setup-buildx-action@v3
+
+    - name: Log in to GitHub Container Registry
+      uses: docker/login-action@v3
+      with:
+        registry: ghcr.io
+        username: ${{ github.actor }}
+        password: ${{ secrets.GITHUB_TOKEN }}
+
+    - name: Extract metadata for Docker
+      id: meta
+      uses: docker/metadata-action@v5
+      with:
+        images: ghcr.io/azure-samples/explore-iot-operations/python-wasm-builder
+        tags: |
+          type=ref,event=branch
+          type=ref,event=pr
+          type=raw,value=latest
+
+    - name: Build and push Docker image
+      uses: docker/build-push-action@v5
+      with:
+        context: samples/wasm/python
+        file: samples/wasm/python/Dockerfile
+        push: true
+        tags: ${{ steps.meta.outputs.tags }}
+        labels: ${{ steps.meta.outputs.labels }}
+        platforms: linux/amd64,linux/arm64

.github/workflows/wasm-rust-build.yml

@@ -0,0 +1,52 @@
+name: Build and Push WASM Rust Builder Image
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [ main, wasm ]
+    paths:
+      - 'samples/wasm/rust/Dockerfile'
+  pull_request:
+    branches: [ main ]
+    paths:
+      - 'samples/wasm/rust/Dockerfile'
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+    defaults:
+      run:
+        shell: bash
+    steps:
+      - name: "Checkout GitHub Action"
+        uses: actions/checkout@main
+
+      - name: "Login to GitHub Container Registry"
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{github.actor}}
+          password: ${{secrets.GITHUB_TOKEN}}
+
+      - name: Extract metadata
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ghcr.io/azure-samples/explore-iot-operations/rust-wasm-builder
+          tags: |
+            type=ref,event=branch
+            type=ref,event=pr
+            type=sha
+            type=raw,value=latest
+
+      - name: "Build and Push Rust Builder Image"
+        uses: docker/build-push-action@v5
+        with:
+          context: samples/wasm/rust
+          file: samples/wasm/rust/Dockerfile
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}

docker/wasm-rust-build/Dockerfile

@@ -0,0 +1,13 @@
+ARG RUST_VERSION=1.85
+FROM rust:${RUST_VERSION}-alpine
+
+ARG ARCH="x86_64"
+
+RUN apk add --no-cache clang lld musl-dev git perl make cmake
+RUN rustup target add wasm32-wasip2 ${ARCH}-unknown-linux-gnu
+
+# Set up environment for Cargo registry
+ENV CARGO_REGISTRIES_AZURE_VSCODE_TINYKUBE_INDEX="sparse+https://pkgs.dev.azure.com/azure-iot-sdks/iot-operations/_packaging/preview/Cargo/index/"
+ENV CARGO_NET_GIT_FETCH_WITH_CLI=true
+
+WORKDIR /src

docker/wasm-rust-build/README.md

@@ -0,0 +1,52 @@
+# WASM Rust Build Base Image
+
+This Docker image provides a base environment for building WebAssembly (WASM) applications with Rust targeting the `wasm32-wasip2` platform.
+
+## Features
+
+- Based on Alpine Linux with Rust 1.85
+- Pre-configured for WASM compilation with `wasm32-wasip2` target
+- Required build tools: clang, lld, musl-dev, git, perl, make, cmake
+- Environment variables for Azure IoT Operations Cargo registry
+
+## Usage
+
+This base image is intended to be used as a build stage in multi-stage Docker builds for WASM operators:
+
+```dockerfile
+ARG IMAGE=ghcr.io/azure-samples/explore-iot-operations/wasm-rust-build:latest
+FROM $IMAGE AS operator-build
+
+ARG APP_NAME
+ARG BUILD_MODE="release"
+
+WORKDIR /src
+COPY ./Cargo.toml ./Cargo.toml
+COPY ./src ./src
+
+RUN if [ "${BUILD_MODE}" = "release" ]; then \
+        cargo build --release --target wasm32-wasip2; \
+    else \
+        cargo build --target wasm32-wasip2; \
+    fi
+
+FROM scratch
+ARG BUILD_MODE
+ARG APP_NAME
+COPY --from=operator-build "/src/target/wasm32-wasip2/${BUILD_MODE}/${APP_NAME}.wasm" "${APP_NAME}.wasm"
+ENTRYPOINT [ "${APP_NAME}.wasm" ]
+```
+
+## Build Arguments
+
+- `RUST_VERSION`: Rust version to use (default: 1.85)
+- `ARCH`: Target architecture (default: x86_64)
+
+## Environment Variables
+
+- `CARGO_REGISTRIES_AZURE_VSCODE_TINYKUBE_INDEX`: Azure IoT Operations Cargo registry URL
+- `CARGO_NET_GIT_FETCH_WITH_CLI`: Use Git CLI for fetching dependencies
+
+## Publishing
+
+This image is automatically built and published to GitHub Container Registry via GitHub Actions when changes are made to the `docker/wasm-rust-build/` directory.