Skip to content

Commit 9d8a6f5

Browse files
JillGrant615JillGrant615
authored
Merge pull request #298459 from hhunter-ms/auto-purge
Auto purge documentation
2 parents 86dacf9 + 564d97b commit 9d8a6f5

File tree

3 files changed

+283
-2
lines changed

3 files changed

+283
-2
lines changed

‎articles/azure-functions/durable/TOC.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@
116116
- name: Monitoring dashboard
117117
href: ./durable-task-scheduler/durable-task-scheduler-dashboard.md
118118
- name: Auto-purge
119-
href: ./durable-task-scheduler/durable-task-scheduler-dashboard.md
119+
href: ./durable-task-scheduler/durable-task-scheduler-auto-purge.md
120120
- name: Durable Functions
121121
items:
122122
- name: Throughput benchmark

‎articles/azure-functions/durable/durable-task-scheduler/durable-task-scheduler-auto-purge.md

Lines changed: 278 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,278 @@
1+
---
2+
title: Set autopurge retention policies for Azure Functions Durable Task Scheduler (preview)
3+
description: Learn about how and why you'd want to configure autopurge retention policies for Durable Task Scheduler.
4+
ms.topic: conceptual
5+
ms.date: 04/29/2025
6+
---
7+
8+
# Set autopurge retention policies for Azure Functions Durable Task Scheduler (preview)
9+
10+
Orchestration history data should be purged periodically to free up storage resources. Otherwise, the app observes performance degradation as history data accumulates overtime. The Durable Task Scheduler offers a lightweight, configurable autopurge feature that helps you manage orchestration data clean-up without manual intervention.
11+
12+
Autopurge operates asynchronously in the background, optimized to minimize system resource usage and prevent interference with other Durable Task operations. Although autopurge doesn't adhere to a strict schedule, its clean-up rate generally aligns with your orchestration scheduling rate.
13+
14+
## How it works
15+
16+
Autopurge is an opt-in feature. You can enable it by defining retention policies that control how long to keep the data of orchestrations in certain statuses. The autopurge feature purges orchestration data associated with terminal statuses. "Terminal" refers to orchestrations that have reached a final state with no further scheduling, event processing, or work item generation. Terminal statuses include:
17+
- `Completed`
18+
- `Failed`
19+
- `Canceled`
20+
- `Terminated`
21+
22+
The orchestration instances eligible for autopurge match those targeted by [the Durable SDK PurgeInstancesAsync API](/dotnet/api/microsoft.durabletask.client.durabletaskclientextensions.purgeinstancesasync?view=durabletask-dotnet-1.x&preserve-view=true).
23+
24+
Autopurge ignores orchestration data associated with non-terminal statuses. "Non-terminal" statuses indicate that the orchestration instance is either actively executing, paused, or in a state where it may resume in the future (waiting for external events or timers). These orchestrations that are continuing as new, where the current *execution* is completed, but a new instance has been started as a continuation.
25+
26+
These statuses include:
27+
- `Pending`
28+
- `Running`
29+
- `Suspended`
30+
- `Continued_As_New`
31+
32+
[Once enabled,](#enable-autopurge) autopurge periodically deletes orchestration data older than the retention period you set. Autopurge only
33+
34+
> [!NOTE]
35+
> Retention policies you define are applied to **all** task hubs in a scheduler.
36+
37+
### Policy value
38+
39+
Retention value can range from 0 (purge as soon as possible) to the maximum integer value, with the unit being **days**.
40+
41+
The retention period refers to the time period since the orchestration entered terminal state. For example, you set a retention value of 1 day. If the orchestration takes 10 days to finish, autopurge won't delete it until the following day. Autopurge isn't triggered until the orchestration finishes.
42+
43+
Although retention periods have no maximum limit, we recommend you avoid retaining large volumes of stale orchestration data for extended periods. This practice ensures efficient use of storage resources and maintains optimal app performance.
44+
45+
### Types of policies
46+
47+
When configuring an autopurge retention policy, you can set either a *specific* or a *default* policy.
48+
49+
- **Default policy** purges orchestration data *regardless* of `orchestrationState`. The following policy purges orchestration data of all statuses covered by the feature after 2 days:
50+
51+
```json
52+
{
53+
"retentionPeriodInDays": 2
54+
}
55+
```
56+
57+
- **Specific policy** defines purging of orchestration data for specific `orchestrationState`. The following policy tells Durable Task Scheduler to keep *completed* orchestration data for 1 day, after which this data is purged.
58+
59+
```json
60+
{
61+
"retentionPeriodInDays": 1,
62+
"orchestrationState": "Completed"
63+
}
64+
```
65+
66+
Add specific policies to override the default policy applied to orchestrations. In the following example, the second and third policies override the default policy (`"retentionPeriodInDays": 1`).
67+
- Data associated with `completed` orchestrations is deleted as soon as possible.
68+
- Data associated with `failed` orchestrations is purged after 60 days.
69+
70+
```json
71+
[
72+
{
73+
"retentionPeriodInDays": 1
74+
},
75+
{
76+
"retentionPeriodInDays": 0,
77+
"orchestrationState": "Completed"
78+
},
79+
{
80+
"retentionPeriodInDays": 60,
81+
"orchestrationState": "Failed"
82+
}
83+
]
84+
```
85+
86+
Since no specific policy is set for `canceled` or `terminated` orchestrations, the default policy still applies to them, purging their data after 1 day.
87+
88+
[For more information, see the API reference spec for Durable Task Scheduler retention policies.](/rest/api/durabletask/retention-policies/create-or-replace?view=rest-durabletask-2025-04-01-preview&preserve-view=true)
89+
90+
## Enable autopurge
91+
92+
You can define retention policies using:
93+
94+
- Durable Task CLI
95+
- Azure Resource Manager (ARM)
96+
- Bicep
97+
98+
# [Durable Task CLI](#tab/cli)
99+
100+
Make sure you have the latest version of the Durable Task CLI extension.
101+
102+
```azurecli
103+
az extension add --name durabletask
104+
az extension update --name durabletask
105+
```
106+
107+
Create or update the retention policy by running the following command.
108+
109+
```azurecli
110+
az durabletask retention-policy create --scheduler-name SCHEDULER_NAME --resource-group RESOURCE_GROUP --default-days 1 --completed-days 0 --failed-days 60
111+
```
112+
113+
The following properties specify the retention duration for orchestration data of different statuses.
114+
115+
| Property | Description |
116+
| -------- | ----------- |
117+
| `--canceled-days` or `-x` | The number of days to retain canceled orchestrations. |
118+
| `--completed-days` or `-c` | The number of days to retain completed orchestrations. |
119+
| `--default-days` or `-d` | The number of days to retain orchestrations. |
120+
| `--failed-days` or `-f` | The number of days to retain failed orchestrations. |
121+
| `--terminated-days` or `-t` | The number of days to retain terminated orchestrations. |
122+
123+
**Example response**
124+
125+
If creation is successful, you receive the following response.
126+
127+
```json
128+
{
129+
"id": "/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.DurableTask/schedulers/SCHEDULER_NAMER/retentionPolicies/default",
130+
"name": "default",
131+
"properties": {
132+
"provisioningState": "Succeeded",
133+
"retentionPolicies": [
134+
{
135+
"retentionPeriodInDays": 1
136+
},
137+
{
138+
"orchestrationState": "Completed",
139+
"retentionPeriodInDays": 0
140+
},
141+
{
142+
"orchestrationState": "Failed",
143+
"retentionPeriodInDays": 60
144+
}
145+
]
146+
},
147+
"resourceGroup": "RESOURCE_GROUP",
148+
"systemData": {
149+
"createdAt": "2025-04-23T23:41:17.3165122Z",
150+
"createdBy": "[email protected]",
151+
"createdByType": "User",
152+
"lastModifiedAt": "2025-04-23T23:41:17.3165122Z",
153+
"lastModifiedBy": "[email protected]",
154+
"lastModifiedByType": "User"
155+
},
156+
"type": "microsoft.durabletask/schedulers/retentionpolicies"
157+
}
158+
```
159+
160+
> [!TIP]
161+
> Learn more about the retention policy command via [the CLI reference](/cli/azure/durabletask/retention-policy?view=azure-cli-latest&preserve-view=true).
162+
163+
# [Azure Resource Manager](#tab/arm)
164+
165+
You can create or update retention policies using the Azure Resource Manager API using the following request. Make sure you're pulling from the latest preview version.
166+
167+
```HTTP
168+
PUT https://management.azure.com/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.DurableTask/schedulers/SCHEDULER_NAME/retentionPolicies/default?api-version=2025-04-01-preview
169+
170+
{
171+
"properties": {
172+
"retentionPolicies": [
173+
{
174+
"retentionPeriodInDays": 1
175+
},
176+
{
177+
"retentionPeriodInDays": 0,
178+
"orchestrationState": "Completed"
179+
},
180+
{
181+
"retentionPeriodInDays": 60,
182+
"orchestrationState": "Failed"
183+
}
184+
]
185+
}
186+
}
187+
```
188+
189+
**Example response**
190+
191+
If creation is successful, you receive the following response.
192+
193+
```json
194+
{
195+
"properties": {
196+
"provisioningState": "Succeeded",
197+
"retentionPolicies": [
198+
{
199+
"retentionPeriodInDays": 1
200+
},
201+
{
202+
"retentionPeriodInDays": 0,
203+
"orchestrationState": "Completed"
204+
},
205+
{
206+
"retentionPeriodInDays": 60,
207+
"orchestrationState": "Failed"
208+
}
209+
]
210+
},
211+
"id": "/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.DurableTask/schedulers/SCHEDULER_NAME/retentionPolicies/default",
212+
"name": "default",
213+
"type": "Microsoft.DurableTask/schedulers/retentionPolicies",
214+
"systemData": {
215+
"createdBy": "[email protected]",
216+
"createdByType": "User",
217+
"createdAt": "2025-03-31T23:34:09.612Z",
218+
"lastModifiedBy": "[email protected]",
219+
"lastModifiedByType": "User",
220+
"lastModifiedAt": "2025-03-31T23:34:09.612Z"
221+
}
222+
}
223+
```
224+
225+
# [Bicep](#tab/bicep)
226+
227+
You can create or update retention policies by adding the `retentionPolicies` configuration to your Bicep file. Make sure you're pulling from the latest preview version.
228+
229+
```bicep
230+
resource exampleResource 'Microsoft.DurableTask/schedulers/retentionPolicies@2025-04-01-preview' = {
231+
parent: parentResource
232+
name: 'default'
233+
properties: {
234+
retentionPolicies: [
235+
{
236+
retentionPeriodInDays: 1
237+
}
238+
{
239+
"retentionPeriodInDays": 0,
240+
"orchestrationState": "Completed"
241+
}
242+
{
243+
retentionPeriodInDays: 60
244+
orchestrationState: 'Failed'
245+
}
246+
]
247+
}
248+
}
249+
```
250+
---
251+
252+
## Disable autopurge
253+
254+
# [Durable Task CLI](#tab/cli)
255+
256+
Delete the retention policies using the following command. The Durable Task Scheduler stops cleaning orchestration data within 5 to 10 minutes.
257+
258+
```azurecli
259+
az durabletask retention-policy delete --scheduler-name SCHEDULER_NAME --resource-group RESOURCE_GROUP
260+
```
261+
262+
# [Azure Resource Manager](#tab/arm)
263+
264+
Delete the retention policy using an API call.
265+
266+
```HTTP
267+
DELETE https://management.azure.com/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.DurableTask/schedulers/SCHEDULER_NAME/retentionPolicies/default?api-version=2025-04-01-preview
268+
```
269+
270+
# [Bicep](#tab/bicep)
271+
272+
Remove `retentionPolicies` from your Bicep file.
273+
274+
---
275+
276+
## Next steps
277+
278+
Monitor and manage your orchestration status and history using [the Durable Task Scheduler dashboard](./durable-task-scheduler-dashboard.md).

‎articles/azure-functions/durable/durable-task-scheduler/durable-task-scheduler.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ The Durable Task Scheduler runs in Azure as a separate resource from your app. T
4646
Using a managed scheduler like Durable Task Scheduler reduces CPU and memory resource consumption caused by the overhead of managing partitions and other complex state store interactions. Using a managed scheduler (instead of a BYO storage provider, for example) allows your app instances to run more efficiently with less resource contention.
4747

4848
- **Fault isolation**
49-
when Durable Task Scheduler experiences stability or availability issues, it won't affect the stability or availability of your connected apps. By separating the scheduler from the app, you can reduce the risk of cascading failures and improve overall reliability.
49+
When Durable Task Scheduler experiences stability or availability issues, it won't affect the stability or availability of your connected apps. By separating the scheduler from the app, you can reduce the risk of cascading failures and improve overall reliability.
5050

5151
- **Independent scaling**
5252
The scheduler resource can be scaled independently of the app, allowing for better infrastructure resource management and cost optimization. For example, multiple apps can share the same scheduler resource, improving overall resource utilization. This capability is especially useful for organizations with multiple teams or projects.
@@ -114,6 +114,9 @@ You can see all of the emulator versions available by running the following comm
114114
curl -s https://mcr.microsoft.com/v2/dts/dts-emulator/tags/list
115115
```
116116

117+
### Autopurge retention policies
118+
119+
Large volumes of completed orchestration instance data can lead to storage bloat, incur higher storage costs, and degrade performance. The autopurge feature for Durable Task Scheduler provides a streamlined, configurable solution to manage orchestration instance clean-up automatically. [Learn more about setting autopurge retention policies for Azure Functions Durable Task Scheduler.](./durable-task-scheduler-auto-purge.md)
117120

118121
## Limitations and considerations
119122

0 commit comments

Comments
 (0)