Skip to content

Commit 226538f

Browse files
jlianjlian
committed
Request disk persistence
1 parent 7f5c98c commit 226538f

File tree

2 files changed

+98
-15
lines changed

2 files changed

+98
-15
lines changed

‎articles/iot-operations/connect-to-cloud/howto-create-dataflow.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -654,6 +654,79 @@ sourceSettings:
654654

655655
To learn more, see [Understand message schemas](concept-schema-registry.md).
656656

657+
## Request disk persistence (preview)
658+
659+
Request disk persistence allows data flows to maintain state across restarts. When you enable this feature, the graph can recover processing state if connected broker restarts. This feature is useful for stateful processing scenarios where losing intermediate data would be problematic. When you enable request disk persistence, the broker persists the MQTT data, like messages in the subscriber queue, to disk. This approach ensures that your data flow's data source doesn't experience data loss during power outages or broker restarts. The broker maintains optimal performance because persistence is configured per data flow, so only the data flows that need persistence use this feature.
660+
661+
The data flow graph makes this persistence request during subscription using an MQTTv5 user property. This feature only works when:
662+
663+
1. The data flow uses the MQTT broker or asset as source
664+
1. The MQTT broker has persistence enabled with dynamic persistence mode set to `Enabled` for the data type, like subscriber queues
665+
666+
This configuration allows MQTT clients like data flows to request disk persistence for their subscriptions using MQTTv5 user properties. For detailed MQTT broker persistence configuration, see [Configure MQTT broker persistence](../manage-mqtt-broker/howto-broker-persistence.md).
667+
668+
The setting accepts `Enabled` or `Disabled`, with `Disabled` as the default.
669+
670+
# [Operations experience](#tab/portal)
671+
672+
When creating or editing a data flow, select **Edit**, then check **Yes** next to **Request data persistence**.
673+
674+
# [Azure CLI](#tab/cli)
675+
676+
Add the `requestDiskPersistence` property to your data flow configuration file:
677+
678+
```json
679+
{
680+
"mode": "Enabled",
681+
"requestDiskPersistence": "Enabled",
682+
"operations": [
683+
// ... your data flow operations
684+
]
685+
}
686+
```
687+
688+
# [Bicep](#tab/bicep)
689+
690+
Add the `requestDiskPersistence` property to your data flow resource. The API version is `2025-07-01-preview` or later:
691+
692+
```bicep
693+
resource dataflow 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflows@2025-07-01-preview' = {
694+
parent: defaultDataflowProfile
695+
name: dataflowName
696+
extendedLocation: {
697+
name: customLocation.id
698+
type: 'CustomLocation'
699+
}
700+
properties: {
701+
mode: 'Enabled'
702+
requestDiskPersistence: 'Enabled'
703+
operations: [
704+
// ... your data flow operations
705+
]
706+
}
707+
}
708+
```
709+
710+
# [Kubernetes (preview)](#tab/kubernetes)
711+
712+
Add the `requestDiskPersistence` property to your data flow spec. The API version is `connectivity.iotoperations.azure.com/v1beta1` or later:
713+
714+
```yaml
715+
apiVersion: connectivity.iotoperations.azure.com/v1beta1
716+
kind: Dataflow
717+
metadata:
718+
name: <DATAFLOW_NAME>
719+
namespace: azure-iot-operations
720+
spec:
721+
profileRef: default
722+
mode: Enabled
723+
requestDiskPersistence: Enabled
724+
operations:
725+
# ... your data flow operations
726+
```
727+
728+
---
729+
657730
## Transformation
658731

659732
The transformation operation is where you can transform the data from the source before you send it to the destination. Transformations are optional. If you don't need to make changes to the data, don't include the transformation operation in the data flow configuration. Multiple transformations are chained together in stages regardless of the order in which they're specified in the configuration. The order of the stages is always:

‎articles/iot-operations/connect-to-cloud/howto-dataflow-graph-wasm.md

Lines changed: 25 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -109,15 +109,15 @@ param customLocationName string = '<CUSTOM_LOCATION_NAME>'
109109
param registryEndpointName string = '<REGISTRY_ENDPOINT_NAME>'
110110
param acrName string = '<YOUR_ACR_NAME>'
111111
112-
resource aioInstance 'Microsoft.IoTOperations/instances@2024-11-01' existing = {
112+
resource aioInstance 'Microsoft.IoTOperations/instances@2025-07-01-preview'' existing = {
113113
name: aioInstanceName
114114
}
115115
116116
resource customLocation 'Microsoft.ExtendedLocation/customLocations@2021-08-31-preview' existing = {
117117
name: customLocationName
118118
}
119119
120-
resource registryEndpoint 'Microsoft.IoTOperations/instances/registryEndpoints@2024-11-01' = {
120+
resource registryEndpoint 'Microsoft.IoTOperations/instances/registryEndpoints@2025-07-01-preview'' = {
121121
parent: aioInstance
122122
name: registryEndpointName
123123
extendedLocation: {
@@ -270,20 +270,20 @@ param customLocationName string = '<CUSTOM_LOCATION_NAME>'
270270
param dataflowGraphName string = '<GRAPH_NAME>'
271271
param registryEndpointName string = '<REGISTRY_ENDPOINT_NAME>'
272272
273-
resource aioInstance 'Microsoft.IoTOperations/instances@2024-11-01' existing = {
273+
resource aioInstance 'Microsoft.IoTOperations/instances@2025-07-01-preview'' existing = {
274274
name: aioInstanceName
275275
}
276276
277277
resource customLocation 'Microsoft.ExtendedLocation/customLocations@2021-08-31-preview' existing = {
278278
name: customLocationName
279279
}
280280
281-
resource defaultDataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2024-11-01' existing = {
281+
resource defaultDataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2025-07-01-preview'' existing = {
282282
parent: aioInstance
283283
name: 'default'
284284
}
285285
286-
resource dataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2024-11-01' = {
286+
resource dataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2025-07-01-preview'' = {
287287
parent: defaultDataflowProfile
288288
name: dataflowGraphName
289289
extendedLocation: {
@@ -563,20 +563,20 @@ param customLocationName string = '<CUSTOM_LOCATION_NAME>'
563563
param dataflowGraphName string = '<COMPLEX_GRAPH_NAME>'
564564
param registryEndpointName string = '<REGISTRY_ENDPOINT_NAME>'
565565
566-
resource aioInstance 'Microsoft.IoTOperations/instances@2024-11-01' existing = {
566+
resource aioInstance 'Microsoft.IoTOperations/instances@2025-07-01-preview'' existing = {
567567
name: aioInstanceName
568568
}
569569
570570
resource customLocation 'Microsoft.ExtendedLocation/customLocations@2021-08-31-preview' existing = {
571571
name: customLocationName
572572
}
573573
574-
resource defaultDataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2024-11-01' existing = {
574+
resource defaultDataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2025-07-01-preview'' existing = {
575575
parent: aioInstance
576576
name: 'default'
577577
}
578578
579-
resource complexDataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2024-11-01' = {
579+
resource complexDataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2025-07-01-preview'' = {
580580
parent: defaultDataflowProfile
581581
name: dataflowGraphName
582582
extendedLocation: {
@@ -794,7 +794,7 @@ The mode property determines whether the data flow graph is actively processing
794794
# [Bicep](#tab/bicep)
795795

796796
```bicep
797-
resource dataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2024-11-01' = {
797+
resource dataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2025-07-01-preview'' = {
798798
// ... other properties
799799
properties: {
800800
mode: 'Enabled' // or 'Disabled'
@@ -827,12 +827,12 @@ The profile reference connects your data flow graph to a data flow profile, whic
827827
In Bicep, you specify the profile by creating the data flow graph as a child resource of the profile:
828828

829829
```bicep
830-
resource defaultDataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2024-11-01' existing = {
830+
resource defaultDataflowProfile 'Microsoft.IoTOperations/instances/dataflowProfiles@2025-07-01-preview'' existing = {
831831
parent: aioInstance
832832
name: 'default'
833833
}
834834
835-
resource dataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2024-11-01' = {
835+
resource dataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2025-07-01-preview'' = {
836836
parent: defaultDataflowProfile // This establishes the profile relationship
837837
// ... other properties
838838
}
@@ -853,16 +853,26 @@ spec:
853853

854854
---
855855

856-
### Disk persistence
856+
### Request disk persistence
857857

858-
Disk persistence allows data flow graphs to maintain state across restarts. When enabled, the graph can recover processing state if pods are restarted. The storage capability is useful for stateful processing scenarios where losing intermediate data would be problematic.
858+
> [!IMPORTANT]
859+
> There is a known issue with disk persistence for data flow graphs. This feature is currently not working as expected. For more information, see [Known issues](../troubleshoot/known-issues.md).
860+
861+
Request disk persistence allows data flow graphs to maintain state across restarts. When you enable this feature, the graph can recover processing state if connected broker restarts. This feature is useful for stateful processing scenarios where losing intermediate data would be problematic. When you enable request disk persistence, the broker persists the MQTT data, like messages in the subscriber queue, to disk. This approach ensures that your data flow's data source won't experience data loss during power outages or broker restarts. The broker maintains optimal performance because persistence is configured per data flow, so only the data flows that need persistence use this feature.
862+
863+
The data flow graph makes this persistence request during subscription using an MQTTv5 user property. This feature only works when:
864+
865+
1. The data flow uses the MQTT broker as a source (source node with MQTT endpoint)
866+
1. The MQTT broker has persistence enabled with dynamic persistence mode set to `Enabled` for the data type, like subscriber queues
867+
868+
This configuration allows MQTT clients like data flow graphs to request disk persistence for their subscriptions using MQTTv5 user properties. For detailed MQTT broker persistence configuration, see [Configure MQTT broker persistence](../manage-mqtt-broker/howto-broker-persistence.md).
859869

860-
The setting accepts `Enabled` or `Disabled` (case-insensitive), with `Disabled` as the default.
870+
The setting accepts `Enabled` or `Disabled`, with `Disabled` as the default.
861871

862872
# [Bicep](#tab/bicep)
863873

864874
```bicep
865-
resource dataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2024-11-01' = {
875+
resource dataflowGraph 'Microsoft.IoTOperations/instances/dataflowProfiles/dataflowGraphs@2025-07-01-preview'' = {
866876
// ... other properties
867877
properties: {
868878
requestDiskPersistence: 'Enabled'

0 commit comments

Comments
 (0)