You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/azure-functions/durable/durable-functions-cloud-backup.md
+56-48Lines changed: 56 additions & 48 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,15 +10,13 @@ ms.author: azfuncdf
10
10
11
11
*Fan-out/fan-in* refers to the pattern of executing multiple functions concurrently and then performing some aggregation on the results. This article explains a sample that uses [Durable Functions](durable-functions-overview.md) to implement a fan-in/fan-out scenario. The sample is a durable function that backs up all or some of an app's site content into Azure Storage.
In this sample, the functions upload all files under a specified directory recursively into blob storage. They also count the total number of bytes that were uploaded.
20
18
21
-
It's possible to write a single function that takes care of everything. The main problem you would run into is **scalability**. A single function execution can only run on a single VM, so the throughput will be limited by the throughput of that single VM. Another problem is **reliability**. If there's a failure midway through, or if the entire process takes more than 5 minutes, the backup could fail in a partially completed state. It would then need to be restarted.
19
+
It's possible to write a single function that takes care of everything. The main problem you would run into is **scalability**. A single function execution can only run on a single virtual machine, so the throughput will be limited by the throughput of that single VM. Another problem is **reliability**. If there's a failure midway through, or if the entire process takes more than 5 minutes, the backup could fail in a partially completed state. It would then need to be restarted.
22
20
23
21
A more robust approach would be to write two regular functions: one would enumerate the files and add the file names to a queue, and another would read from the queue and upload the files to blob storage. This approach is better in terms of throughput and reliability, but it requires you to provision and manage a queue. More importantly, significant complexity is introduced in terms of **state management** and **coordination** if you want to do anything more, like report the total number of bytes uploaded.
24
22
@@ -28,84 +26,103 @@ A Durable Functions approach gives you all of the mentioned benefits with very l
28
26
29
27
This article explains the following functions in the sample app:
30
28
31
-
*`E2_BackupSiteContent`
32
-
*`E2_GetFileList`
33
-
*`E2_CopyFileToBlob`
29
+
*`E2_BackupSiteContent`: An [orchestrator function](durable-functions-bindings.md#orchestration-trigger) that calls `E2_GetFileList` to obtain a list of files to back up, then calls `E2_CopyFileToBlob` to back up each file.
30
+
*`E2_GetFileList`: An [activity function](durable-functions-bindings.md#activity-trigger) that returns a list of files in a directory.
31
+
*`E2_CopyFileToBlob`: An activity function that backs up a single file to Azure Blob Storage.
34
32
35
-
The following sections explain the configuration and code that is used for C# scripting. The code for Visual Studio development is shown at the end of the article.
33
+
### E2_BackupSiteContent orchestrator function
36
34
37
-
## The cloud backup orchestration (Visual Studio Code and Azure portal sample code)
35
+
This orchestrator function essentially does the following:
38
36
39
-
The `E2_BackupSiteContent` function uses the standard *function.json* for orchestrator functions.
37
+
1. Takes a `rootDirectory` value as an input parameter.
38
+
2. Calls a function to get a recursive list of files under `rootDirectory`.
39
+
3. Makes multiple parallel function calls to upload each file into Azure Blob Storage.
40
+
4. Waits for all uploads to complete.
41
+
5. Returns the sum total bytes that were uploaded to Azure Blob Storage.
Notice the `await Task.WhenAll(tasks);` line. All the individual calls to the `E2_CopyFileToBlob` function were *not* awaited, which allows them to run in parallel. When we pass this array of tasks to `Task.WhenAll`, we get back a task that won't complete *until all the copy operations have completed*. If you're familiar with the Task Parallel Library (TPL) in .NET, then this is not new to you. The difference is that these tasks could be running on multiple virtual machines concurrently, and the Durable Functions extension ensures that the end-to-end execution is resilient to process recycling.
48
50
49
-
### JavaScript (Functions 2.0 only)
51
+
After awaiting from `Task.WhenAll`, we know that all function calls have completed and have returned values back to us. Each call to `E2_CopyFileToBlob` returns the number of bytes uploaded, so calculating the sum total byte count is a matter of adding all those return values together.
Notice the `await Task.WhenAll(tasks);` (C#) and `yield context.df.Task.all(tasks);` (JavaScript) lines. All the individual calls to the `E2_CopyFileToBlob` function were *not* awaited, which allows them to run in parallel. When we pass this array of tasks to `Task.WhenAll` (C#) or `context.df.Task.all` (JavaScript), we get back a task that won't complete *until all the copy operations have completed*. If you're familiar with the Task Parallel Library (TPL) in .NET or [`Promise.all`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all) in JavaScript, then this is not new to you. The difference is that these tasks could be running on multiple VMs concurrently, and the Durable Functions extension ensures that the end-to-end execution is resilient to process recycling.
59
+
Here is the code that implements the orchestrator function:
Notice the `yield context.df.Task.all(tasks);` line. All the individual calls to the `E2_CopyFileToBlob` function were *not* yielded, which allows them to run in parallel. When we pass this array of tasks to `context.df.Task.all`, we get back a task that won't complete *until all the copy operations have completed*. If you're familiar with [`Promise.all`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all) in JavaScript, then this is not new to you. The difference is that these tasks could be running on multiple virtual machines concurrently, and the Durable Functions extension ensures that the end-to-end execution is resilient to process recycling.
62
64
63
65
> [!NOTE]
64
66
> Although tasks are conceptually similar to JavaScript promises, orchestrator functions should use `context.df.Task.all` and `context.df.Task.any` instead of `Promise.all` and `Promise.race` to manage task parallelization.
65
67
66
-
After awaiting from `Task.WhenAll` (or yielding from `context.df.Task.all`), we know that all function calls have completed and have returned values back to us. Each call to `E2_CopyFileToBlob` returns the number of bytes uploaded, so calculating the sum total byte count is a matter of adding all those return values together.
68
+
After yielding from `context.df.Task.all`, we know that all function calls have completed and have returned values back to us. Each call to `E2_CopyFileToBlob` returns the number of bytes uploaded, so calculating the sum total byte count is a matter of adding all those return values together.
69
+
70
+
---
67
71
68
-
## Helper activity functions
72
+
###Helper activity functions
69
73
70
-
The helper activity functions, as with other samples, are just regular functions that use the `activityTrigger` trigger binding. For example, the *function.json* file for `E2_GetFileList` looks like the following:
74
+
The helper activity functions, as with other samples, are just regular functions that use the `activityTrigger` trigger binding.
The JavaScript implementation of `E2_GetFileList` uses the `readdirp` module to recursively read the directory structure.
92
+
The function uses the `readdirp` module (version 2.x) to recursively read the directory structure.
93
+
94
+
---
85
95
86
96
> [!NOTE]
87
-
> You might be wondering why you couldn't just put this code directly into the orchestrator function. You could, but this would break one of the fundamental rules of orchestrator functions, which is that they should never do I/O, including local file system access.
97
+
> You might be wondering why you couldn't just put this code directly into the orchestrator function. You could, but this would break one of the fundamental rules of orchestrator functions, which is that they should never do I/O, including local file system access. For more information, see [Orchestrator function code constraints](durable-functions-code-constraints.md).
88
98
89
-
The *function.json* file for `E2_CopyFileToBlob` is similarly simple:
The C# implementation is also straightforward. It happens to use some advanced features of Azure Functions bindings (that is, the use of the `Binder` parameter), but you don't need to worry about those details for the purpose of this walkthrough.
The function uses some advanced features of Azure Functions bindings (that is, the use of the [`Binder` parameter](../functions-dotnet-class-library.md#binding-at-runtime)), but you don't need to worry about those details for the purpose of this walkthrough.
98
109
99
-
### JavaScript (Functions 2.0 only)
110
+
#[JavaScript](#tab/javascript)
100
111
101
-
The JavaScript implementation does not have access to the `Binder` feature of Azure Functions, so the [Azure Storage SDK for Node](https://github.com/Azure/azure-storage-node) takes its place.
112
+
The *function.json* file for `E2_CopyFileToBlob` is similarly simple:
The JavaScript implementation uses the [Azure Storage SDK for Node](https://github.com/Azure/azure-storage-node) to upload the files to Azure Blob Storage.
The implementation loads the file from disk and asynchronously streams the contents into a blob of the same name in the "backups" container. The return value is the number of bytes copied to storage, that is then used by the orchestrator function to compute the aggregate sum.
106
123
107
124
> [!NOTE]
108
-
> This is a perfect example of moving I/O operations into an `activityTrigger` function. Not only can the work be distributed across many different VMs, but you also get the benefits of checkpointing the progress. If the host process gets terminated for any reason, you know which uploads have already completed.
125
+
> This is a perfect example of moving I/O operations into an `activityTrigger` function. Not only can the work be distributed across many different machines, but you also get the benefits of checkpointing the progress. If the host process gets terminated for any reason, you know which uploads have already completed.
Now you can see that the orchestration is complete and approximately how much time it took to complete. You also see a value for the `output` field, which indicates that around 450 KB of logs were uploaded.
162
179
163
-
## Visual Studio sample code
164
-
165
-
Here is the orchestration as a single C# file in a Visual Studio project:
166
-
167
-
> [!NOTE]
168
-
> You will need to install the `Microsoft.Azure.WebJobs.Extensions.Storage` NuGet package to run the sample code below.
This sample has shown how to implement the fan-out/fan-in pattern. The next sample shows how to implement the monitor pattern using [durable timers](durable-functions-timers.md).
0 commit comments