From a17025dc612cabf499c0e27f6670f2d442a8f338 Mon Sep 17 00:00:00 2001 From: Andy Staples Date: Tue, 19 Aug 2025 11:23:00 -0600 Subject: [PATCH 1/6] Add versioning samples/docs --- docs/features.md | 6 +- docs/supported-patterns.md | 45 ++++++++++++- examples/version_aware_orchestrator.py | 93 ++++++++++++++++++++++++++ 3 files changed, 140 insertions(+), 4 deletions(-) create mode 100644 examples/version_aware_orchestrator.py diff --git a/docs/features.md b/docs/features.md index d5c1b8c..c497a1c 100644 --- a/docs/features.md +++ b/docs/features.md @@ -4,7 +4,11 @@ The following features are currently supported: ### Orchestrations -Orchestrations are implemented using ordinary Python functions that take an `OrchestrationContext` as their first parameter. The `OrchestrationContext` provides APIs for starting child orchestrations, scheduling activities, and waiting for external events, among other things. Orchestrations are fault-tolerant and durable, meaning that they can automatically recover from failures and rebuild their local execution state. Orchestrator functions must be deterministic, meaning that they must always produce the same output given the same input. +Orchestrators are implemented using ordinary Python functions that take an `OrchestrationContext` as their first parameter. The `OrchestrationContext` provides APIs for starting child orchestrations, scheduling activities, and waiting for external events, among other things. Orchestrations are fault-tolerant and durable, meaning that they can automatically recover from failures and rebuild their local execution state. Orchestrator functions must be deterministic, meaning that they must always produce the same output given the same input. + +#### Orchestration versioning + +Orchestrations may be assigned a version when they are first created. If an orchestration is given a version, it will continually be checked during its lifecycle to ensure that it remains compatible with the underlying orchestrator code. If the orchestrator code is updated while an orchestration is running, rules can be set that will define the behavior - whether the orchestration should fail, abandon for reprocessing at a later time, or attempt to run anyway. For more information, see [Orchestration versioning in Durable Functions](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-orchestration-versioning) (Note that concepts specific to Azure Functions, such as host.json settings, do not apply) and [The provided examples](./supported-patterns.md) ### Activities diff --git a/docs/supported-patterns.md b/docs/supported-patterns.md index bbac4a7..86c55e6 100644 --- a/docs/supported-patterns.md +++ b/docs/supported-patterns.md @@ -20,7 +20,7 @@ def sequence(ctx: task.OrchestrationContext, _): return [result1, result2, result3] ``` -You can find the full sample [here](../examples/activity_sequence.py). +Link to the full [function chaining example](../examples/activity_sequence.py). ### Fan-out/fan-in @@ -48,7 +48,7 @@ def orchestrator(ctx: task.OrchestrationContext, _): return {'work_items': work_items, 'results': results, 'total': sum(results)} ``` -You can find the full sample [here](../examples/fanout_fanin.py). +Link to the full [fan-out sample](../examples/fanout_fanin.py). ### Human interaction and durable timers @@ -79,4 +79,43 @@ def purchase_order_workflow(ctx: task.OrchestrationContext, order: Order): As an aside, you'll also notice that the example orchestration above works with custom business objects. Support for custom business objects includes support for custom classes, custom data classes, and named tuples. Serialization and deserialization of these objects is handled automatically by the SDK. -You can find the full sample [here](../examples/human_interaction.py). \ No newline at end of file +Link to the full [human interaction sample](../examples/human_interaction.py). + +### Version-aware orchestrator + +When utilizing orchestration versioning, it is possible for an orchestrator to remain backwards-compatible with orchestrations created using the previously defined version. For instance, consider an orchestration defined with the following signature: + +```python +def my_orchestrator(ctx: task.OrchestrationContext, order: Order): + """Dummy orchestrator function illustrating old logic""" + yield ctx.call_activity(activity_one) + yield ctx.call_activity(activity_two) + return "Success" +``` + +Assume that any orchestrations created using this orchestrator were versioned 1.0.0. If the signature of this method needs to be updated to call activity_three between the calls to activity_one and activity_two, ordinarily this would break any running orchestrations at the time of deployment. However, the following orchestrator will be able to process both orchestraions versioned 1.0.0 and 2.0.0 after the change: + +```python +def my_orchestrator(ctx: task.OrchestrationContext, order: Order): + """Version-aware dummy orchestrator capable of processing both old and new orchestrations""" + yield ctx.call_activity(activity_one) + if ctx.version > '1.0.0': + yield ctx.call_activity(activity_three) + yield ctx.call_activity(activity_two) +``` + +Alternatively, if the orchestrator changes completely, the following syntax might be preferred: + +```python +def my_orchestrator(ctx: task.OrchestrationContext, order: Order): + if ctx.version == '1.0.0': + yield ctx.call_activity(activity_one) + yield ctx.call_activity(activity_two) + return "Success + yield ctx.call_activity(activity_one) + yield ctx.call_activity(activity_three) + yield ctx.call_activity(activity_two) + return "Success" +``` + +Link to the full [version-aware orchestrator sample](../examples/version_aware_orchestrator.py) \ No newline at end of file diff --git a/examples/version_aware_orchestrator.py b/examples/version_aware_orchestrator.py new file mode 100644 index 0000000..f9ee502 --- /dev/null +++ b/examples/version_aware_orchestrator.py @@ -0,0 +1,93 @@ +"""End-to-end sample that demonstrates how to configure an orchestrator +that a dynamic number activity functions in parallel, waits for them all +to complete, and prints an aggregate summary of the outputs.""" +import os +import random + +from azure.identity import DefaultAzureCredential + +from durabletask import client, task, worker +from durabletask.azuremanaged.client import DurableTaskSchedulerClient +from durabletask.azuremanaged.worker import DurableTaskSchedulerWorker + + +def activity_v1(ctx: task.ActivityContext, input: str) -> str: + """Activity function that returns a result for a given work item""" + print("processing input:", input) + return "Success from activity v1" + + +def activity_v2(ctx: task.ActivityContext, input: str) -> str: + """Activity function that returns a result for a given work item""" + print("processing input:", input) + return "Success from activity v2" + + +def orchestrator(ctx: task.OrchestrationContext, _): + """Orchestrator function that checks the orchestration version and has version-aware behavior + Use case: Updating an orchestrator with new logic while maintaining compatibility with previously + started orchestrations""" + if ctx.version == "1.0.0": + # For version 1.0.0, we use the original logic + result: int = yield ctx.call_activity(activity_v1, input="input for v1") + elif ctx.version == "2.0.0": + # For version 2.0.0, we use the updated logic + result: int = yield ctx.call_activity(activity_v2, input="input for v2") + else: + raise ValueError(f"Unsupported version: {ctx.version}") + return { + 'result': result, + } + + +# Use environment variables if provided, otherwise use default emulator values +taskhub_name = os.getenv("TASKHUB", "default") +endpoint = os.getenv("ENDPOINT", "http://localhost:8080") + +print(f"Using taskhub: {taskhub_name}") +print(f"Using endpoint: {endpoint}") + +# Set credential to None for emulator, or DefaultAzureCredential for Azure +credential = None if endpoint == "http://localhost:8080" else DefaultAzureCredential() + +# configure and start the worker - use secure_channel=False for emulator +secure_channel = endpoint != "http://localhost:8080" +with DurableTaskSchedulerWorker(host_address=endpoint, secure_channel=secure_channel, + taskhub=taskhub_name, token_credential=credential) as w: + # This worker is versioned for v2, as the orchestrator code has already been updated + # CURRENT_OR_OLDER allows this worker to process orchestrations versioned below 2.0.0 - e.g. 1.0.0 + w.use_versioning(worker.VersioningOptions( + version="2.0.0", + default_version="2.0.0", + match_strategy=worker.VersionMatchStrategy.CURRENT_OR_OLDER, + failure_strategy=worker.VersionFailureStrategy.FAIL + )) + w.add_orchestrator(orchestrator) + w.add_activity(activity_v1) + w.add_activity(activity_v2) + w.start() + + # create a client, start an orchestration, and wait for it to finish + # The client's version matches the worker's version + c = DurableTaskSchedulerClient(host_address=endpoint, secure_channel=secure_channel, + taskhub=taskhub_name, token_credential=credential, + default_version="2.0.0") + # Schedule a new orchestration manually versioned to 1.0.0 + # Normally, this would have been scheduled before the worker started from a worker also versioned v1.0.0, + # Here we are doing it manually to avoid creating two workers + instance_id_v1 = c.schedule_new_orchestration(orchestrator, version="1.0.0") + state_v1 = c.wait_for_orchestration_completion(instance_id_v1, timeout=30) + if state_v1 and state_v1.runtime_status == client.OrchestrationStatus.COMPLETED: + print(f'Orchestration v1 completed! Result: {state_v1.serialized_output}') + elif state_v1: + print(f'Orchestration v1 failed: {state_v1.failure_details}') + + # Also check that the orchestrator can be run with the current version + instance_id = c.schedule_new_orchestration(orchestrator) + state = c.wait_for_orchestration_completion(instance_id, timeout=30) + if state and state.runtime_status == client.OrchestrationStatus.COMPLETED: + print(f'Orchestration completed! Result: {state.serialized_output}') + elif state: + print(f'Orchestration failed: {state.failure_details}') + + exit() From 9bb8ef764e204c6727d50d0eeb9405f551b7abb3 Mon Sep 17 00:00:00 2001 From: Andy Staples Date: Tue, 19 Aug 2025 11:28:07 -0600 Subject: [PATCH 2/6] Add sub-orchestration documentation --- docs/features.md | 2 +- durabletask/worker.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/features.md b/docs/features.md index c497a1c..6ed3257 100644 --- a/docs/features.md +++ b/docs/features.md @@ -20,7 +20,7 @@ Orchestrations can schedule durable timers using the `create_timer` API. These t ### Sub-orchestrations -Orchestrations can start child orchestrations using the `call_sub_orchestrator` API. Child orchestrations are useful for encapsulating complex logic and for breaking up large orchestrations into smaller, more manageable pieces. +Orchestrations can start child orchestrations using the `call_sub_orchestrator` API. Child orchestrations are useful for encapsulating complex logic and for breaking up large orchestrations into smaller, more manageable pieces. Sub-orchestrations can also be versioned in a similar manner to their parent orchestrations, however, they do not inherit the parent's version. Instead, they will use the default_version defined in the current worker's VersioningOptions unless otherwise specified during `call_sub_orchestrator`. ### External events diff --git a/durabletask/worker.py b/durabletask/worker.py index 2a1626d..bd230c9 100644 --- a/durabletask/worker.py +++ b/durabletask/worker.py @@ -111,7 +111,7 @@ def __init__(self, version: Optional[str] = None, Args: version: The version of orchestrations that the worker can work on. - default_version: The default version that will be used for starting new orchestrations. + default_version: The default version that will be used for starting new sub-orchestrations. match_strategy: The versioning strategy for the Durable Task worker. failure_strategy: The versioning failure strategy for the Durable Task worker. """ From d55098d9122bb7b21e1697fd5988a8d4e993e803 Mon Sep 17 00:00:00 2001 From: Andy Staples Date: Tue, 19 Aug 2025 11:29:16 -0600 Subject: [PATCH 3/6] Lint fix --- examples/version_aware_orchestrator.py | 1 - 1 file changed, 1 deletion(-) diff --git a/examples/version_aware_orchestrator.py b/examples/version_aware_orchestrator.py index f9ee502..15ac961 100644 --- a/examples/version_aware_orchestrator.py +++ b/examples/version_aware_orchestrator.py @@ -2,7 +2,6 @@ that a dynamic number activity functions in parallel, waits for them all to complete, and prints an aggregate summary of the outputs.""" import os -import random from azure.identity import DefaultAzureCredential From 065b9772ce0dec874e14461f9b21bd2c816d10c7 Mon Sep 17 00:00:00 2001 From: Andy Staples Date: Tue, 19 Aug 2025 11:31:54 -0600 Subject: [PATCH 4/6] Phrasing --- docs/supported-patterns.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/supported-patterns.md b/docs/supported-patterns.md index 86c55e6..612678a 100644 --- a/docs/supported-patterns.md +++ b/docs/supported-patterns.md @@ -20,7 +20,7 @@ def sequence(ctx: task.OrchestrationContext, _): return [result1, result2, result3] ``` -Link to the full [function chaining example](../examples/activity_sequence.py). +See the full [function chaining example](../examples/activity_sequence.py). ### Fan-out/fan-in @@ -48,7 +48,7 @@ def orchestrator(ctx: task.OrchestrationContext, _): return {'work_items': work_items, 'results': results, 'total': sum(results)} ``` -Link to the full [fan-out sample](../examples/fanout_fanin.py). +See the full [fan-out sample](../examples/fanout_fanin.py). ### Human interaction and durable timers @@ -79,7 +79,7 @@ def purchase_order_workflow(ctx: task.OrchestrationContext, order: Order): As an aside, you'll also notice that the example orchestration above works with custom business objects. Support for custom business objects includes support for custom classes, custom data classes, and named tuples. Serialization and deserialization of these objects is handled automatically by the SDK. -Link to the full [human interaction sample](../examples/human_interaction.py). +See the full [human interaction sample](../examples/human_interaction.py). ### Version-aware orchestrator @@ -118,4 +118,4 @@ def my_orchestrator(ctx: task.OrchestrationContext, order: Order): return "Success" ``` -Link to the full [version-aware orchestrator sample](../examples/version_aware_orchestrator.py) \ No newline at end of file +See the full [version-aware orchestrator sample](../examples/version_aware_orchestrator.py) \ No newline at end of file From 663b3ace6f319779e55cb6c97b764c118de74027 Mon Sep 17 00:00:00 2001 From: Andy Staples Date: Tue, 19 Aug 2025 11:45:30 -0600 Subject: [PATCH 5/6] Messaging around links to Functions docs --- docs/features.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/features.md b/docs/features.md index 6ed3257..7e861da 100644 --- a/docs/features.md +++ b/docs/features.md @@ -8,7 +8,7 @@ Orchestrators are implemented using ordinary Python functions that take an `Orch #### Orchestration versioning -Orchestrations may be assigned a version when they are first created. If an orchestration is given a version, it will continually be checked during its lifecycle to ensure that it remains compatible with the underlying orchestrator code. If the orchestrator code is updated while an orchestration is running, rules can be set that will define the behavior - whether the orchestration should fail, abandon for reprocessing at a later time, or attempt to run anyway. For more information, see [Orchestration versioning in Durable Functions](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-orchestration-versioning) (Note that concepts specific to Azure Functions, such as host.json settings, do not apply) and [The provided examples](./supported-patterns.md) +Orchestrations may be assigned a version when they are first created. If an orchestration is given a version, it will continually be checked during its lifecycle to ensure that it remains compatible with the underlying orchestrator code. If the orchestrator code is updated while an orchestration is running, rules can be set that will define the behavior - whether the orchestration should fail, abandon for reprocessing at a later time, or attempt to run anyway. For more information, see [The provided examples](./supported-patterns.md). For more information about versioning in the context of Durable Functions, see [Orchestration versioning in Durable Functions](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-orchestration-versioning) (Note that concepts specific to Azure Functions, such as host.json settings, do not apply to this SDK). ### Activities @@ -20,7 +20,7 @@ Orchestrations can schedule durable timers using the `create_timer` API. These t ### Sub-orchestrations -Orchestrations can start child orchestrations using the `call_sub_orchestrator` API. Child orchestrations are useful for encapsulating complex logic and for breaking up large orchestrations into smaller, more manageable pieces. Sub-orchestrations can also be versioned in a similar manner to their parent orchestrations, however, they do not inherit the parent's version. Instead, they will use the default_version defined in the current worker's VersioningOptions unless otherwise specified during `call_sub_orchestrator`. +Orchestrations can start child orchestrations using the `call_sub_orchestrator` API. Child orchestrations are useful for encapsulating complex logic and for breaking up large orchestrations into smaller, more manageable pieces. Sub-orchestrations can also be versioned in a similar manner to their parent orchestrations, however, they do not inherit the parent orchestrator's version. Instead, they will use the default_version defined in the current worker's VersioningOptions unless otherwise specified during `call_sub_orchestrator`. ### External events From e6ac43b21e4dc00389be328d7f7f765f62425ca5 Mon Sep 17 00:00:00 2001 From: Andy Staples Date: Tue, 19 Aug 2025 12:35:55 -0600 Subject: [PATCH 6/6] Add settings guide and examples --- docs/features.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/features.md b/docs/features.md index 7e861da..d3fcc56 100644 --- a/docs/features.md +++ b/docs/features.md @@ -10,6 +10,32 @@ Orchestrators are implemented using ordinary Python functions that take an `Orch Orchestrations may be assigned a version when they are first created. If an orchestration is given a version, it will continually be checked during its lifecycle to ensure that it remains compatible with the underlying orchestrator code. If the orchestrator code is updated while an orchestration is running, rules can be set that will define the behavior - whether the orchestration should fail, abandon for reprocessing at a later time, or attempt to run anyway. For more information, see [The provided examples](./supported-patterns.md). For more information about versioning in the context of Durable Functions, see [Orchestration versioning in Durable Functions](https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-orchestration-versioning) (Note that concepts specific to Azure Functions, such as host.json settings, do not apply to this SDK). +##### Orchestration versioning options + +Both the Durable worker and durable client have versioning configuration available. Because versioning checks are handled by the worker, the only information the client needs is a default_version, taken in its constructor, to use as the version for new orchestrations unless otherwise specified. The worker takes a VersioningOptions object with a `default_version` for new sub-orchestrations, a `version` used by the worker for orchestration version comparisons, and two more options giving control over versioning behavior in case of match failures, a `VersionMatchStrategy` and `VersionFailureStrategy`. + +**VersionMatchStrategy** + +| VersionMatchStrategy.NONE | VersionMatchStrategy.STRICT | VersionMatchStrategy.CURRENT_OR_OLDER | +|-|-|-| +| Do not compare orchestration versions | Only allow orchestrations with the same version as the worker | Allow orchestrations with the same or older version as the worker | + +**VersionFailureStrategy** + +| VersionFailureStrategy.REJECT | VersionFailureStrategy.FAIL | +|-|-| +| Abandon execution of the orchestrator, but allow it to be reprocessed later | Fail the orchestration | + +**Strategy examples** + +Scenario 1: You are implementing versioning for the first time in your worker. You want to have a default version for new orchestrations, but do not care about comparing versions with currently running ones. Choose VersionMatchStrategy.NONE, and VersionFailureStrategy does not matter. + +Scenario 2: You are updating an orchestrator's code, and you do not want old orchestrations to continue to be processed on the new code. Bump the default version and the worker version, set VersionMatchStrategy.STRICT and VersionFailureStrategy.FAIL. + +Scenario 3: You are updating an orchestrator's code, and you have ensured the code is version-aware so that it remains backward-compatible with existing orchestrations. Bump the default version and the worker version, and set VersionMatchStrategy.CURRENT_OR_OLDER and VersionFailureStrategy.FAIL. + +Scenario 4: You are performing a high-availability deployment, and your orchestrator code contains breaking changes making it not backward-compatible. Bump the default version and the worker version, and set VersionFailureStrategy.REJECT and VersionMatchStrategy.STRICT. Ensure that at least a few of the previous version of workers remain available to continue processing the older orchestrations - eventually, all older orchestrations _should_ land on the correct workers for processing. Once all remaining old orchestrations have been processed, shut down the remaining old workers. + ### Activities Activities are implemented using ordinary Python functions that take an `ActivityContext` as their first parameter. Activity functions are scheduled by orchestrations and have at-least-once execution guarantees, meaning that they will be executed at least once but may be executed multiple times in the event of a transient failure. Activity functions are where the real "work" of any orchestration is done.