Skip to content

Commit 131a204

Browse files
authored
Merge pull request #94584 from cgillum/cgillum-edits
Update Durable Functions Samples for 2.0 GA
2 parents 9c67ab7 + c0be511 commit 131a204

34 files changed

+864
-532
lines changed

articles/azure-functions/durable/durable-functions-bindings.md

Lines changed: 135 additions & 36 deletions
Large diffs are not rendered by default.

articles/azure-functions/durable/durable-functions-cloud-backup.md

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -7,23 +7,25 @@ manager: jeconnoc
77
keywords:
88
ms.service: azure-functions
99
ms.topic: conceptual
10-
ms.date: 12/07/2018
10+
ms.date: 11/02/2019
1111
ms.author: azfuncdf
1212
---
1313

1414
# Fan-out/fan-in scenario in Durable Functions - Cloud backup example
1515

1616
*Fan-out/fan-in* refers to the pattern of executing multiple functions concurrently and then performing some aggregation on the results. This article explains a sample that uses [Durable Functions](durable-functions-overview.md) to implement a fan-in/fan-out scenario. The sample is a durable function that backs up all or some of an app's site content into Azure Storage.
1717

18+
[!INCLUDE [v1-note](../../../includes/functions-durable-v1-tutorial-note.md)]
19+
1820
[!INCLUDE [durable-functions-prerequisites](../../../includes/durable-functions-prerequisites.md)]
1921

2022
## Scenario overview
2123

2224
In this sample, the functions upload all files under a specified directory recursively into blob storage. They also count the total number of bytes that were uploaded.
2325

24-
It's possible to write a single function that takes care of everything. The main problem you would run into is **scalability**. A single function execution can only run on a single VM, so the throughput will be limited by the throughput of that single VM. Another problem is **reliability**. If there's a failure midway through, or if the entire process takes more than 5 minutes, the backup could fail in a partially-completed state. It would then need to be restarted.
26+
It's possible to write a single function that takes care of everything. The main problem you would run into is **scalability**. A single function execution can only run on a single VM, so the throughput will be limited by the throughput of that single VM. Another problem is **reliability**. If there's a failure midway through, or if the entire process takes more than 5 minutes, the backup could fail in a partially completed state. It would then need to be restarted.
2527

26-
A more robust approach would be to write two regular functions: one would enumerate the files and add the file names to a queue, and another would read from the queue and upload the files to blob storage. This is better in terms of throughput and reliability, but it requires you to provision and manage a queue. More importantly, significant complexity is introduced in terms of **state management** and **coordination** if you want to do anything more, like report the total number of bytes uploaded.
28+
A more robust approach would be to write two regular functions: one would enumerate the files and add the file names to a queue, and another would read from the queue and upload the files to blob storage. This approach is better in terms of throughput and reliability, but it requires you to provision and manage a queue. More importantly, significant complexity is introduced in terms of **state management** and **coordination** if you want to do anything more, like report the total number of bytes uploaded.
2729

2830
A Durable Functions approach gives you all of the mentioned benefits with very low overhead.
2931

@@ -35,7 +37,7 @@ This article explains the following functions in the sample app:
3537
* `E2_GetFileList`
3638
* `E2_CopyFileToBlob`
3739

38-
The following sections explain the configuration and code that are used for C# scripting. The code for Visual Studio development is shown at the end of the article.
40+
The following sections explain the configuration and code that is used for C# scripting. The code for Visual Studio development is shown at the end of the article.
3941

4042
## The cloud backup orchestration (Visual Studio Code and Azure portal sample code)
4143

@@ -49,7 +51,7 @@ Here is the code that implements the orchestrator function:
4951

5052
[!code-csharp[Main](~/samples-durable-functions/samples/csx/E2_BackupSiteContent/run.csx)]
5153

52-
### JavaScript (Functions 2.x only)
54+
### JavaScript (Functions 2.0 only)
5355

5456
[!code-javascript[Main](~/samples-durable-functions/samples/javascript/E2_BackupSiteContent/index.js)]
5557

@@ -61,7 +63,7 @@ This orchestrator function essentially does the following:
6163
4. Waits for all uploads to complete.
6264
5. Returns the sum total bytes that were uploaded to Azure Blob Storage.
6365

64-
Notice the `await Task.WhenAll(tasks);` (C#) and `yield context.df.Task.all(tasks);` (JavaScript) lines. All the individual calls to the `E2_CopyFileToBlob` function were *not* awaited. This is intentional to allow them to run in parallel. When we pass this array of tasks to `Task.WhenAll` (C#) or `context.df.Task.all` (JavaScript), we get back a task that won't complete *until all the copy operations have completed*. If you're familiar with the Task Parallel Library (TPL) in .NET or [`Promise.all`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all) in JavaScript, then this is not new to you. The difference is that these tasks could be running on multiple VMs concurrently, and the Durable Functions extension ensures that the end-to-end execution is resilient to process recycling.
66+
Notice the `await Task.WhenAll(tasks);` (C#) and `yield context.df.Task.all(tasks);` (JavaScript) lines. All the individual calls to the `E2_CopyFileToBlob` function were *not* awaited, which allows them to run in parallel. When we pass this array of tasks to `Task.WhenAll` (C#) or `context.df.Task.all` (JavaScript), we get back a task that won't complete *until all the copy operations have completed*. If you're familiar with the Task Parallel Library (TPL) in .NET or [`Promise.all`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all) in JavaScript, then this is not new to you. The difference is that these tasks could be running on multiple VMs concurrently, and the Durable Functions extension ensures that the end-to-end execution is resilient to process recycling.
6567

6668
> [!NOTE]
6769
> Although tasks are conceptually similar to JavaScript promises, orchestrator functions should use `context.df.Task.all` and `context.df.Task.any` instead of `Promise.all` and `Promise.race` to manage task parallelization.
@@ -80,7 +82,7 @@ And here is the implementation:
8082

8183
[!code-csharp[Main](~/samples-durable-functions/samples/csx/E2_GetFileList/run.csx)]
8284

83-
### JavaScript (Functions 2.x only)
85+
### JavaScript (Functions 2.0 only)
8486

8587
[!code-javascript[Main](~/samples-durable-functions/samples/javascript/E2_GetFileList/index.js)]
8688

@@ -93,13 +95,13 @@ The *function.json* file for `E2_CopyFileToBlob` is similarly simple:
9395

9496
[!code-json[Main](~/samples-durable-functions/samples/csx/E2_CopyFileToBlob/function.json)]
9597

96-
The C# implementation is also pretty straightforward. It happens to use some advanced features of Azure Functions bindings (that is, the use of the `Binder` parameter), but you don't need to worry about those details for the purpose of this walkthrough.
98+
The C# implementation is also straightforward. It happens to use some advanced features of Azure Functions bindings (that is, the use of the `Binder` parameter), but you don't need to worry about those details for the purpose of this walkthrough.
9799

98100
### C#
99101

100102
[!code-csharp[Main](~/samples-durable-functions/samples/csx/E2_CopyFileToBlob/run.csx)]
101103

102-
### JavaScript (Functions 2.x only)
104+
### JavaScript (Functions 2.0 only)
103105

104106
The JavaScript implementation does not have access to the `Binder` feature of Azure Functions, so the [Azure Storage SDK for Node](https://github.com/Azure/azure-storage-node) takes its place.
105107

@@ -131,24 +133,24 @@ This HTTP request triggers the `E2_BackupSiteContent` orchestrator and passes th
131133
HTTP/1.1 202 Accepted
132134
Content-Length: 719
133135
Content-Type: application/json; charset=utf-8
134-
Location: http://{host}/admin/extensions/DurableTaskExtension/instances/b4e9bdcc435d460f8dc008115ff0a8a9?taskHub=DurableFunctionsHub&connection=Storage&code={systemKey}
136+
Location: http://{host}/runtime/webhooks/durabletask/instances/b4e9bdcc435d460f8dc008115ff0a8a9?taskHub=DurableFunctionsHub&connection=Storage&code={systemKey}
135137
136138
(...trimmed...)
137139
```
138140

139141
Depending on how many log files you have in your function app, this operation could take several minutes to complete. You can get the latest status by querying the URL in the `Location` header of the previous HTTP 202 response.
140142

141143
```
142-
GET http://{host}/admin/extensions/DurableTaskExtension/instances/b4e9bdcc435d460f8dc008115ff0a8a9?taskHub=DurableFunctionsHub&connection=Storage&code={systemKey}
144+
GET http://{host}/runtime/webhooks/durabletask/instances/b4e9bdcc435d460f8dc008115ff0a8a9?taskHub=DurableFunctionsHub&connection=Storage&code={systemKey}
143145
```
144146

145147
```
146148
HTTP/1.1 202 Accepted
147149
Content-Length: 148
148150
Content-Type: application/json; charset=utf-8
149-
Location: http://{host}/admin/extensions/DurableTaskExtension/instances/b4e9bdcc435d460f8dc008115ff0a8a9?taskHub=DurableFunctionsHub&connection=Storage&code={systemKey}
151+
Location: http://{host}/runtime/webhooks/durabletask/instances/b4e9bdcc435d460f8dc008115ff0a8a9?taskHub=DurableFunctionsHub&connection=Storage&code={systemKey}
150152
151-
{"runtimeStatus":"Running","input":"D:\\home\\LogFiles","output":null,"createdTime":"2017-06-29T18:50:55Z","lastUpdatedTime":"2017-06-29T18:51:16Z"}
153+
{"runtimeStatus":"Running","input":"D:\\home\\LogFiles","output":null,"createdTime":"2019-06-29T18:50:55Z","lastUpdatedTime":"2019-06-29T18:51:16Z"}
152154
```
153155

154156
In this case, the function is still running. You are able to see the input that was saved into the orchestrator state and the last updated time. You can continue to use the `Location` header values to poll for completion. When the status is "Completed", you see an HTTP response value similar to the following:
@@ -158,7 +160,7 @@ HTTP/1.1 200 OK
158160
Content-Length: 152
159161
Content-Type: application/json; charset=utf-8
160162
161-
{"runtimeStatus":"Completed","input":"D:\\home\\LogFiles","output":452071,"createdTime":"2017-06-29T18:50:55Z","lastUpdatedTime":"2017-06-29T18:51:26Z"}
163+
{"runtimeStatus":"Completed","input":"D:\\home\\LogFiles","output":452071,"createdTime":"2019-06-29T18:50:55Z","lastUpdatedTime":"2019-06-29T18:51:26Z"}
162164
```
163165

164166
Now you can see that the orchestration is complete and approximately how much time it took to complete. You also see a value for the `output` field, which indicates that around 450 KB of logs were uploaded.
@@ -168,7 +170,7 @@ Now you can see that the orchestration is complete and approximately how much ti
168170
Here is the orchestration as a single C# file in a Visual Studio project:
169171

170172
> [!NOTE]
171-
> You will need to install the `Microsoft.Azure.WebJobs.Extensions.Storage` Nuget package to run the sample code below.
173+
> You will need to install the `Microsoft.Azure.WebJobs.Extensions.Storage` NuGet package to run the sample code below.
172174
173175
[!code-csharp[Main](~/samples-durable-functions/samples/precompiled/BackupSiteContent.cs)]
174176

0 commit comments

Comments
 (0)