Skip to content

Commit c515a09

Browse files
authored
Merge pull request #234913 from cdpark/parallel-workload-dotnet
Freshness Pass for User Story: 79612 Dotnet workloads
2 parents dfa1982 + 182e6d7 commit c515a09

File tree

3 files changed

+53
-52
lines changed

3 files changed

+53
-52
lines changed
-108 KB
Loading
15.8 KB
Loading

articles/batch/tutorial-parallel-dotnet.md

Lines changed: 53 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -1,75 +1,79 @@
11
---
2-
title: Tutorial - Run a parallel workload using the .NET API
3-
description: Tutorial - Transcode media files in parallel with ffmpeg in Azure Batch using the Batch .NET client library
2+
title: "Tutorial: Run a parallel workload using the .NET API"
3+
description: Learn how to transcode media files in parallel using ffmpeg in Azure Batch with the Batch .NET client library.
44
ms.devlang: csharp
55
ms.topic: tutorial
6-
ms.date: 06/22/2022
6+
ms.date: 04/19/2023
77
ms.custom: "mvc, devx-track-csharp"
88
---
99

1010
# Tutorial: Run a parallel workload with Azure Batch using the .NET API
1111

12-
Use Azure Batch to run large-scale parallel and high-performance computing (HPC) batch jobs efficiently in Azure. This tutorial walks through a C# example of running a parallel workload using Batch. You learn a common Batch application workflow and how to interact programmatically with Batch and Storage resources. You learn how to:
12+
Use Azure Batch to run large-scale parallel and high-performance computing (HPC) batch jobs efficiently in Azure. This tutorial walks through a C# example of running a parallel workload using Batch. You learn a common Batch application workflow and how to interact programmatically with Batch and Storage resources.
1313

1414
> [!div class="checklist"]
15-
> * Add an application package to your Batch account
16-
> * Authenticate with Batch and Storage accounts
17-
> * Upload input files to Storage
18-
> * Create a pool of compute nodes to run an application
19-
> * Create a job and tasks to process input files
20-
> * Monitor task execution
21-
> * Retrieve output files
15+
> * Add an application package to your Batch account.
16+
> * Authenticate with Batch and Storage accounts.
17+
> * Upload input files to Storage.
18+
> * Create a pool of compute nodes to run an application.
19+
> * Create a job and tasks to process input files.
20+
> * Monitor task execution.
21+
> * Retrieve output files.
2222
23-
In this tutorial, you convert MP4 media files in parallel to MP3 format using the [ffmpeg](https://ffmpeg.org/) open-source tool.
23+
In this tutorial, you convert MP4 media files to MP3 format, in parallel, by using the [ffmpeg](https://ffmpeg.org) open-source tool.
2424

2525
[!INCLUDE [quickstarts-free-trial-note.md](../../includes/quickstarts-free-trial-note.md)]
2626

2727
## Prerequisites
2828

29-
* [Visual Studio 2017 or later](https://www.visualstudio.com/vs), or [.NET Core 2.1 SDK](https://dotnet.microsoft.com/download/dotnet/2.1) for Linux, macOS, or Windows.
29+
* [Visual Studio 2017 or later](https://www.visualstudio.com/vs), or [.NET Core SDK](https://dotnet.microsoft.com/download/dotnet) for Linux, macOS, or Windows.
3030

31-
* A Batch account and a linked Azure Storage account. To create these accounts, see the Batch quickstarts using the [Azure portal](quick-create-portal.md) or [Azure CLI](quick-create-cli.md).
31+
* A Batch account and a linked Azure Storage account. To create these accounts, see the Batch quickstart guides for the [Azure portal](quick-create-portal.md) or [Azure CLI](quick-create-cli.md).
3232

33-
* Download the appropriate version of ffmpeg for your use case to your local computer. This tutorial and the related sample app use the [Windows 64-bit version of ffmpeg 4.3.1](https://github.com/GyanD/codexffmpeg/releases/tag/4.3.1-2020-11-08). For this tutorial, you only need the zip file. You do not need to unzip the file or install it locally.
33+
* Download the appropriate version of ffmpeg for your use case to your local computer. This tutorial and the related sample app use the [Windows 64-bit full-build version of ffmpeg 4.3.1](https://github.com/GyanD/codexffmpeg/releases/tag/4.3.1-2020-11-08). For this tutorial, you only need the zip file. You do not need to unzip the file or install it locally.
3434

3535
## Sign in to Azure
3636

37-
Sign in to the Azure portal at [https://portal.azure.com](https://portal.azure.com).
37+
Sign in to [the Azure portal](https://portal.azure.com).
3838

3939
## Add an application package
4040

4141
Use the Azure portal to add ffmpeg to your Batch account as an [application package](batch-application-packages.md). Application packages help you manage task applications and their deployment to the compute nodes in your pool.
4242

43-
1. In the Azure portal, click **More services** > **Batch accounts**, and click the name of your Batch account.
44-
3. Click **Applications** > **Add**.
45-
4. For **Application id** enter *ffmpeg*, and a package version of *4.3.1*. Select the ffmpeg zip file you downloaded previously, and then click **OK**. The ffmpeg application package is added to your Batch account.
43+
1. In the Azure portal, click **More services** > **Batch accounts**, and select the name of your Batch account.
4644

47-
![Add application package](./media/tutorial-parallel-dotnet/add-application.png)
45+
1. Click **Applications** > **Add**.
46+
47+
:::image type="content" source="./media/tutorial-parallel-dotnet/add-application.png" alt-text="Screenshot of the Applications section of the batch account.":::
48+
49+
1. Enter *ffmpeg* in the **Application Id** field, and a package version of *4.3.1* in the **Version** field. Select the ffmpeg zip file that you downloaded, and then select **Submit**. The ffmpeg application package is added to your Batch account.
50+
51+
:::image type="content" source="./media/tutorial-parallel-dotnet/new-batch-application.png" alt-text="Screenshot of the ID and version fields in the Add application section.":::
4852

4953
[!INCLUDE [batch-common-credentials](../../includes/batch-common-credentials.md)]
5054

51-
## Download and run the sample
55+
## Download and run the sample app
5256

53-
### Download the sample
57+
### Download the sample app
5458

5559
[Download or clone the sample app](https://github.com/Azure-Samples/batch-dotnet-ffmpeg-tutorial) from GitHub. To clone the sample app repo with a Git client, use the following command:
5660

5761
```
5862
git clone https://github.com/Azure-Samples/batch-dotnet-ffmpeg-tutorial.git
5963
```
6064

61-
Navigate to the directory that contains the Visual Studio solution file `BatchDotNetFfmpegTutorial.sln`.
65+
Navigate to the directory that contains the Visual Studio solution file *BatchDotNetFfmpegTutorial.sln*.
6266

63-
Open the solution file in Visual Studio, and update the credential strings in `Program.cs` with the values you obtained for your accounts. For example:
67+
Open the solution file in Visual Studio, and update the credential strings in *Program.cs* with the values you obtained for your accounts. For example:
6468

6569
```csharp
6670
// Batch account credentials
67-
private const string BatchAccountName = "mybatchaccount";
71+
private const string BatchAccountName = "yourbatchaccount";
6872
private const string BatchAccountKey = "xxxxxxxxxxxxxxxxE+yXrRvJAqT9BlXwwo1CwF+SwAYOxxxxxxxxxxxxxxxx43pXi/gdiATkvbpLRl3x14pcEQ==";
69-
private const string BatchAccountUrl = "https://mybatchaccount.mybatchregion.batch.azure.com";
73+
private const string BatchAccountUrl = "https://yourbatchaccount.yourbatchregion.batch.azure.com";
7074

7175
// Storage account credentials
72-
private const string StorageAccountName = "mystorageaccount";
76+
private const string StorageAccountName = "yourstorageaccount";
7377
private const string StorageAccountKey = "xxxxxxxxxxxxxxxxy4/xxxxxxxxxxxxxxxxfwpbIC5aAWA8wDu+AFXZB827Mt9lybZB1nUcQbQiUrkPtilK5BQ==";
7478
```
7579

@@ -86,11 +90,11 @@ const string appPackageVersion = "4.3.1";
8690

8791
Build and run the application in Visual Studio, or at the command line with the `dotnet build` and `dotnet run` commands. After running the application, review the code to learn what each part of the application does. For example, in Visual Studio:
8892

89-
* Right-click the solution in Solution Explorer and click **Build Solution**.
93+
1. Right-click the solution in Solution Explorer and select **Build Solution**.
9094

91-
* Confirm the restoration of any NuGet packages, if you're prompted. If you need to download missing packages, ensure the [NuGet Package Manager](https://docs.nuget.org/consume/installing-nuget) is installed.
95+
1. Confirm the restoration of any NuGet packages, if you're prompted. If you need to download missing packages, ensure the [NuGet Package Manager](https://docs.nuget.org/consume/installing-nuget) is installed.
9296

93-
Then run it. When you run the sample application, the console output is similar to the following. During execution, you experience a pause at `Monitoring all tasks for 'Completed' state, timeout in 00:30:00...` while the pool's compute nodes are started.
97+
1. Run the solution. When you run the sample application, the console output is similar to the following. During execution, you experience a pause at `Monitoring all tasks for 'Completed' state, timeout in 00:30:00...` while the pool's compute nodes are started.
9498

9599
```
96100
Sample start: 11/19/2018 3:20:21 PM
@@ -113,19 +117,19 @@ Sample end: 11/19/2018 3:29:36 PM
113117
Elapsed time: 00:09:14.3418742
114118
```
115119

116-
Go to your Batch account in the Azure portal to monitor the pool, compute nodes, job, and tasks. For example, to see a heat map of the compute nodes in your pool, click **Pools** > *WinFFmpegPool*.
120+
Go to your Batch account in the Azure portal to monitor the pool, compute nodes, job, and tasks. For example, to see a heat map of the compute nodes in your pool, click **Pools** > **WinFFmpegPool**.
117121

118122
When tasks are running, the heat map is similar to the following:
119123

120-
![Pool heat map](./media/tutorial-parallel-dotnet/pool.png)
124+
:::image type="content" source="./media/tutorial-parallel-dotnet/pool.png" alt-text="Screenshot of the pool heat map in the Azure portal.":::
121125

122-
Typical execution time is approximately **10 minutes** when you run the application in its default configuration. Pool creation takes the most time.
126+
Typical execution time is approximately *10 minutes* when you run the application in its default configuration. Pool creation takes the most time.
123127

124128
[!INCLUDE [batch-common-tutorial-download](../../includes/batch-common-tutorial-download.md)]
125129

126130
## Review the code
127131

128-
The following sections break down the sample application into the steps that it performs to process a workload in the Batch service. Refer to the file `Program.cs` in the solution while you read the rest of this article, since not every line of code in the sample is discussed.
132+
The following sections break down the sample application into the steps that it performs to process a workload in the Batch service. Refer to the file *Program.cs* in the solution while you read the rest of this article, since not every line of code in the sample is discussed.
129133

130134
### Authenticate Blob and Batch clients
131135

@@ -160,12 +164,12 @@ CreateContainerIfNotExistAsync(blobClient, inputContainerName);
160164
CreateContainerIfNotExistAsync(blobClient, outputContainerName);
161165
```
162166

163-
Then, files are uploaded to the input container from the local `InputFiles` folder. The files in storage are defined as Batch [ResourceFile](/dotnet/api/microsoft.azure.batch.resourcefile) objects that Batch can later download to compute nodes.
167+
Then, files are uploaded to the input container from the local *InputFiles* folder. The files in storage are defined as Batch [ResourceFile](/dotnet/api/microsoft.azure.batch.resourcefile) objects that Batch can later download to compute nodes.
164168

165-
Two methods in `Program.cs` are involved in uploading the files:
169+
Two methods in *Program.cs* are involved in uploading the files:
166170

167-
* `UploadFilesToContainerAsync`: Returns a collection of ResourceFile objects and internally calls `UploadResourceFileToContainerAsync` to upload each file that is passed in the `inputFilePaths` parameter.
168-
* `UploadResourceFileToContainerAsync`: Uploads each file as a blob to the input container. After uploading the file, it obtains a shared access signature (SAS) for the blob and returns a ResourceFile object to represent it.
171+
* `UploadFilesToContainerAsync`: Returns a collection of `ResourceFile` objects and internally calls `UploadResourceFileToContainerAsync` to upload each file that is passed in the `inputFilePaths` parameter.
172+
* `UploadResourceFileToContainerAsync`: Uploads each file as a blob to the input container. After uploading the file, it obtains a shared access signature (SAS) for the blob and returns a `ResourceFile` object to represent it.
169173

170174
```csharp
171175
string inputPath = Path.Combine(Environment.CurrentDirectory, "InputFiles");
@@ -188,7 +192,7 @@ Next, the sample creates a pool of compute nodes in the Batch account with a cal
188192
The number of nodes and VM size are set using defined constants. Batch supports dedicated nodes and [Spot nodes](batch-spot-vms.md), and you can use either or both in your pools. Dedicated nodes are reserved for your pool. Spot nodes are offered at a reduced price from surplus VM capacity in Azure. Spot nodes become unavailable if Azure does not have enough capacity. The sample by default creates a pool containing only 5 Spot nodes in size *Standard_A1_v2*.
189193

190194
>[!Note]
191-
>Be sure you check your node quotas. See [Batch service quotas and limits](batch-quota-limit.md#increase-a-quota) for instructions on how to create a quota request."
195+
>Be sure you check your node quotas. See [Batch service quotas and limits](batch-quota-limit.md#increase-a-quota) for instructions on how to create a quota request.
192196
193197
The ffmpeg application is deployed to the compute nodes by adding an [ApplicationPackageReference](/dotnet/api/microsoft.azure.batch.applicationpackagereference) to the pool configuration.
194198

@@ -244,7 +248,7 @@ The sample creates an [OutputFile](/dotnet/api/microsoft.azure.batch.outputfile)
244248

245249
Then, the sample adds tasks to the job with the [AddTaskAsync](/dotnet/api/microsoft.azure.batch.joboperations.addtaskasync) method, which queues them to run on the compute nodes.
246250

247-
Replace the executable's file path with the name of the version that you downloaded. This sample code uses the example `ffmpeg-4.3.1-2020-09-21-full_build`.
251+
Replace the executable's file path with the name of the version that you downloaded. This sample code uses the example `ffmpeg-4.3.1-2020-11-08-full_build`.
248252

249253
```csharp
250254
// Create a collection to hold the tasks added to the job.
@@ -317,15 +321,12 @@ When no longer needed, delete the resource group, Batch account, and storage acc
317321
In this tutorial, you learned how to:
318322

319323
> [!div class="checklist"]
320-
> * Add an application package to your Batch account
321-
> * Authenticate with Batch and Storage accounts
322-
> * Upload input files to Storage
323-
> * Create a pool of compute nodes to run an application
324-
> * Create a job and tasks to process input files
325-
> * Monitor task execution
326-
> * Retrieve output files
327-
328-
For more examples of using the .NET API to schedule and process Batch workloads, see the samples on GitHub.
329-
330-
> [!div class="nextstepaction"]
331-
> [Batch C# samples](https://github.com/Azure-Samples/azure-batch-samples/tree/master/CSharp)
324+
> * Add an application package to your Batch account.
325+
> * Authenticate with Batch and Storage accounts.
326+
> * Upload input files to Storage.
327+
> * Create a pool of compute nodes to run an application.
328+
> * Create a job and tasks to process input files.
329+
> * Monitor task execution.
330+
> * Retrieve output files.
331+
332+
For more examples of using the .NET API to schedule and process Batch workloads, see the [Batch C# samples on GitHub](https://github.com/Azure-Samples/azure-batch-samples/tree/master/CSharp).

0 commit comments

Comments
 (0)