You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/batch/tutorial-parallel-dotnet.md
+53-52Lines changed: 53 additions & 52 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,75 +1,79 @@
1
1
---
2
-
title: Tutorial - Run a parallel workload using the .NET API
3
-
description: Tutorial - Transcode media files in parallel with ffmpeg in Azure Batch using the Batch .NET client library
2
+
title: "Tutorial: Run a parallel workload using the .NET API"
3
+
description: Learn how to transcode media files in parallel using ffmpeg in Azure Batch with the Batch .NET client library.
4
4
ms.devlang: csharp
5
5
ms.topic: tutorial
6
-
ms.date: 06/22/2022
6
+
ms.date: 04/19/2023
7
7
ms.custom: "mvc, devx-track-csharp"
8
8
---
9
9
10
10
# Tutorial: Run a parallel workload with Azure Batch using the .NET API
11
11
12
-
Use Azure Batch to run large-scale parallel and high-performance computing (HPC) batch jobs efficiently in Azure. This tutorial walks through a C# example of running a parallel workload using Batch. You learn a common Batch application workflow and how to interact programmatically with Batch and Storage resources. You learn how to:
12
+
Use Azure Batch to run large-scale parallel and high-performance computing (HPC) batch jobs efficiently in Azure. This tutorial walks through a C# example of running a parallel workload using Batch. You learn a common Batch application workflow and how to interact programmatically with Batch and Storage resources.
13
13
14
14
> [!div class="checklist"]
15
-
> * Add an application package to your Batch account
16
-
> * Authenticate with Batch and Storage accounts
17
-
> * Upload input files to Storage
18
-
> * Create a pool of compute nodes to run an application
19
-
> * Create a job and tasks to process input files
20
-
> * Monitor task execution
21
-
> * Retrieve output files
15
+
> * Add an application package to your Batch account.
16
+
> * Authenticate with Batch and Storage accounts.
17
+
> * Upload input files to Storage.
18
+
> * Create a pool of compute nodes to run an application.
19
+
> * Create a job and tasks to process input files.
20
+
> * Monitor task execution.
21
+
> * Retrieve output files.
22
22
23
-
In this tutorial, you convert MP4 media files in parallel to MP3 formatusing the [ffmpeg](https://ffmpeg.org/) open-source tool.
23
+
In this tutorial, you convert MP4 media files to MP3 format, in parallel, by using the [ffmpeg](https://ffmpeg.org) open-source tool.
*[Visual Studio 2017 or later](https://www.visualstudio.com/vs), or [.NET Core 2.1 SDK](https://dotnet.microsoft.com/download/dotnet/2.1) for Linux, macOS, or Windows.
29
+
*[Visual Studio 2017 or later](https://www.visualstudio.com/vs), or [.NET Core SDK](https://dotnet.microsoft.com/download/dotnet) for Linux, macOS, or Windows.
30
30
31
-
* A Batch account and a linked Azure Storage account. To create these accounts, see the Batch quickstarts using the [Azure portal](quick-create-portal.md) or [Azure CLI](quick-create-cli.md).
31
+
* A Batch account and a linked Azure Storage account. To create these accounts, see the Batch quickstart guides for the [Azure portal](quick-create-portal.md) or [Azure CLI](quick-create-cli.md).
32
32
33
-
* Download the appropriate version of ffmpeg for your use case to your local computer. This tutorial and the related sample app use the [Windows 64-bit version of ffmpeg 4.3.1](https://github.com/GyanD/codexffmpeg/releases/tag/4.3.1-2020-11-08). For this tutorial, you only need the zip file. You do not need to unzip the file or install it locally.
33
+
* Download the appropriate version of ffmpeg for your use case to your local computer. This tutorial and the related sample app use the [Windows 64-bit full-build version of ffmpeg 4.3.1](https://github.com/GyanD/codexffmpeg/releases/tag/4.3.1-2020-11-08). For this tutorial, you only need the zip file. You do not need to unzip the file or install it locally.
34
34
35
35
## Sign in to Azure
36
36
37
-
Sign in to the Azure portal at [https://portal.azure.com](https://portal.azure.com).
37
+
Sign in to [the Azure portal](https://portal.azure.com).
38
38
39
39
## Add an application package
40
40
41
41
Use the Azure portal to add ffmpeg to your Batch account as an [application package](batch-application-packages.md). Application packages help you manage task applications and their deployment to the compute nodes in your pool.
42
42
43
-
1. In the Azure portal, click **More services** > **Batch accounts**, and click the name of your Batch account.
44
-
3. Click **Applications** > **Add**.
45
-
4. For **Application id** enter *ffmpeg*, and a package version of *4.3.1*. Select the ffmpeg zip file you downloaded previously, and then click **OK**. The ffmpeg application package is added to your Batch account.
43
+
1. In the Azure portal, click **More services** > **Batch accounts**, and select the name of your Batch account.
:::image type="content" source="./media/tutorial-parallel-dotnet/add-application.png" alt-text="Screenshot of the Applications section of the batch account.":::
48
+
49
+
1. Enter *ffmpeg* in the **Application Id** field, and a package version of *4.3.1* in the **Version** field. Select the ffmpeg zip file that you downloaded, and then select **Submit**. The ffmpeg application package is added to your Batch account.
50
+
51
+
:::image type="content" source="./media/tutorial-parallel-dotnet/new-batch-application.png" alt-text="Screenshot of the ID and version fields in the Add application section.":::
[Download or clone the sample app](https://github.com/Azure-Samples/batch-dotnet-ffmpeg-tutorial) from GitHub. To clone the sample app repo with a Git client, use the following command:
Navigate to the directory that contains the Visual Studio solution file `BatchDotNetFfmpegTutorial.sln`.
65
+
Navigate to the directory that contains the Visual Studio solution file *BatchDotNetFfmpegTutorial.sln*.
62
66
63
-
Open the solution file in Visual Studio, and update the credential strings in `Program.cs` with the values you obtained for your accounts. For example:
67
+
Open the solution file in Visual Studio, and update the credential strings in *Program.cs* with the values you obtained for your accounts. For example:
Build and run the application in Visual Studio, or at the command line with the `dotnet build` and `dotnet run` commands. After running the application, review the code to learn what each part of the application does. For example, in Visual Studio:
88
92
89
-
* Right-click the solution in Solution Explorer and click**Build Solution**.
93
+
1. Right-click the solution in Solution Explorer and select**Build Solution**.
90
94
91
-
* Confirm the restoration of any NuGet packages, if you're prompted. If you need to download missing packages, ensure the [NuGet Package Manager](https://docs.nuget.org/consume/installing-nuget) is installed.
95
+
1. Confirm the restoration of any NuGet packages, if you're prompted. If you need to download missing packages, ensure the [NuGet Package Manager](https://docs.nuget.org/consume/installing-nuget) is installed.
92
96
93
-
Then run it. When you run the sample application, the console output is similar to the following. During execution, you experience a pause at `Monitoring all tasks for 'Completed' state, timeout in 00:30:00...` while the pool's compute nodes are started.
97
+
1. Run the solution. When you run the sample application, the console output is similar to the following. During execution, you experience a pause at `Monitoring all tasks for 'Completed' state, timeout in 00:30:00...` while the pool's compute nodes are started.
Go to your Batch account in the Azure portal to monitor the pool, compute nodes, job, and tasks. For example, to see a heat map of the compute nodes in your pool, click **Pools** > *WinFFmpegPool*.
120
+
Go to your Batch account in the Azure portal to monitor the pool, compute nodes, job, and tasks. For example, to see a heat map of the compute nodes in your pool, click **Pools** > **WinFFmpegPool**.
117
121
118
122
When tasks are running, the heat map is similar to the following:
The following sections break down the sample application into the steps that it performs to process a workload in the Batch service. Refer to the file `Program.cs` in the solution while you read the rest of this article, since not every line of code in the sample is discussed.
132
+
The following sections break down the sample application into the steps that it performs to process a workload in the Batch service. Refer to the file *Program.cs* in the solution while you read the rest of this article, since not every line of code in the sample is discussed.
Then, files are uploaded to the input container from the local `InputFiles` folder. The files in storage are defined as Batch [ResourceFile](/dotnet/api/microsoft.azure.batch.resourcefile) objects that Batch can later download to compute nodes.
167
+
Then, files are uploaded to the input container from the local *InputFiles* folder. The files in storage are defined as Batch [ResourceFile](/dotnet/api/microsoft.azure.batch.resourcefile) objects that Batch can later download to compute nodes.
164
168
165
-
Two methods in `Program.cs` are involved in uploading the files:
169
+
Two methods in *Program.cs* are involved in uploading the files:
166
170
167
-
*`UploadFilesToContainerAsync`: Returns a collection of ResourceFile objects and internally calls `UploadResourceFileToContainerAsync` to upload each file that is passed in the `inputFilePaths` parameter.
168
-
*`UploadResourceFileToContainerAsync`: Uploads each file as a blob to the input container. After uploading the file, it obtains a shared access signature (SAS) for the blob and returns a ResourceFile object to represent it.
171
+
*`UploadFilesToContainerAsync`: Returns a collection of `ResourceFile` objects and internally calls `UploadResourceFileToContainerAsync` to upload each file that is passed in the `inputFilePaths` parameter.
172
+
*`UploadResourceFileToContainerAsync`: Uploads each file as a blob to the input container. After uploading the file, it obtains a shared access signature (SAS) for the blob and returns a `ResourceFile` object to represent it.
@@ -188,7 +192,7 @@ Next, the sample creates a pool of compute nodes in the Batch account with a cal
188
192
The number of nodes and VM size are set using defined constants. Batch supports dedicated nodes and [Spot nodes](batch-spot-vms.md), and you can use either or both in your pools. Dedicated nodes are reserved for your pool. Spot nodes are offered at a reduced price from surplus VM capacity in Azure. Spot nodes become unavailable if Azure does not have enough capacity. The sample by default creates a pool containing only 5 Spot nodes in size *Standard_A1_v2*.
189
193
190
194
>[!Note]
191
-
>Be sure you check your node quotas. See [Batch service quotas and limits](batch-quota-limit.md#increase-a-quota) for instructions on how to create a quota request."
195
+
>Be sure you check your node quotas. See [Batch service quotas and limits](batch-quota-limit.md#increase-a-quota) for instructions on how to create a quota request.
192
196
193
197
The ffmpeg application is deployed to the compute nodes by adding an [ApplicationPackageReference](/dotnet/api/microsoft.azure.batch.applicationpackagereference) to the pool configuration.
194
198
@@ -244,7 +248,7 @@ The sample creates an [OutputFile](/dotnet/api/microsoft.azure.batch.outputfile)
244
248
245
249
Then, the sample adds tasks to the job with the [AddTaskAsync](/dotnet/api/microsoft.azure.batch.joboperations.addtaskasync) method, which queues them to run on the compute nodes.
246
250
247
-
Replace the executable's file path with the name of the version that you downloaded. This sample code uses the example `ffmpeg-4.3.1-2020-09-21-full_build`.
251
+
Replace the executable's file path with the name of the version that you downloaded. This sample code uses the example `ffmpeg-4.3.1-2020-11-08-full_build`.
248
252
249
253
```csharp
250
254
// Create a collection to hold the tasks added to the job.
@@ -317,15 +321,12 @@ When no longer needed, delete the resource group, Batch account, and storage acc
317
321
In this tutorial, you learned how to:
318
322
319
323
> [!div class="checklist"]
320
-
> * Add an application package to your Batch account
321
-
> * Authenticate with Batch and Storage accounts
322
-
> * Upload input files to Storage
323
-
> * Create a pool of compute nodes to run an application
324
-
> * Create a job and tasks to process input files
325
-
> * Monitor task execution
326
-
> * Retrieve output files
327
-
328
-
For more examples of using the .NET API to schedule and process Batch workloads, see the samples on GitHub.
> * Add an application package to your Batch account.
325
+
> * Authenticate with Batch and Storage accounts.
326
+
> * Upload input files to Storage.
327
+
> * Create a pool of compute nodes to run an application.
328
+
> * Create a job and tasks to process input files.
329
+
> * Monitor task execution.
330
+
> * Retrieve output files.
331
+
332
+
For more examples of using the .NET API to schedule and process Batch workloads, see the [Batch C# samples on GitHub](https://github.com/Azure-Samples/azure-batch-samples/tree/master/CSharp).
0 commit comments