You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: learn-pr/azure/run-parallel-tasks-in-azure-batch-with-the-azure-cli/includes/1-introduction.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,9 +2,9 @@ You can use Azure Batch to execute large-scale, high-intensity computation jobs
2
2
3
3
Imagine you're a software developer at a nonprofit organization whose mission is to give every human on the planet access to clean water. To reach this goal, every citizen is asked to take a picture of their water purification meter and text it to you. Each day, you have to scan pictures from more than 500,000 households and record each reading against the sender's phone number. The data is used to detect water-quality trends and to dispatch the mobile water-quality team to investigate the worst cases across each region. Time is of the essence, but processing each image with Optical Character Recognition (OCR) is time-intensive. With Azure Batch, you can scale out the amount of compute needed to handle this task on a daily basis, saving your nonprofit the expense of fixed resources.
4
4
5
-
In this module, you'll learn about using Azure Batch to create and run parallel tasks with the Azure CLI, and how to use the CLI to check the status of Batch jobs and tasks. This module also describes how to use the standalone Batch Explorer tool to monitor ongoing jobs.
5
+
In this module, you learn about using Azure Batch to create and run parallel tasks with the Azure CLI, and how to use the CLI to check the status of Batch jobs and tasks. This module also describes how to use the standalone Batch Explorer tool to monitor ongoing jobs.
6
6
7
-
By the end of this module, you'll be able to create, execute, and monitor batch jobs.
7
+
By the end of this module, you're able to create, execute, and monitor batch jobs.
8
8
9
9
## Learning objectives
10
10
@@ -20,4 +20,4 @@ In this module, you will:
20
20
- Familiarity with Azure and the Azure CLI
21
21
22
22
>[!IMPORTANT]
23
-
>The exercises in this module are optional. To complete the exercises, you'll need your own Azure subscription.
23
+
>The exercises in this module are optional. To complete the exercises, you need your own Azure subscription.
Copy file name to clipboardExpand all lines: learn-pr/azure/run-parallel-tasks-in-azure-batch-with-the-azure-cli/includes/2-what-is-azure-batch.md
+7-6Lines changed: 7 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,22 +1,23 @@
1
1
Few organizations have the resources to support permanent deployments of super-powerful compute platforms that might only occasionally be used to capacity. More typically, you need a flexible and scalable compute solution such as Azure Batch to provide the computational power.
2
2
3
-
As part of the process of performing OCR on the uploaded pictures of water-purification meters using Azure Batch, you need to have an understanding of the main components of an Azure Batch workflow.
3
+
As part of the process of performing Optical Character Recognition (OCR) on the uploaded pictures of water-purification meters using Azure Batch, you need to have an understanding of the main components of an Azure Batch workflow.
4
4
5
-
Here, you'll look at what Azure Batch is designed to do, and how Batch components are used in a typical workflow.
5
+
Here, you look at what Azure Batch is designed to do, and how Batch components are used in a typical workflow.
6
6
7
7
## Compute-intensive tasks and parallel workloads
8
8
9
-
Some workloads require massive computational power. Examples include financial risk modeling, 3D image rendering, media transcoding, and genetic sequence analysis. In some cases, these workloads can be broken down into separate subtasks and can be run in parallel so that the tasks are performed in much less time. Three Virtual Machines (VMs) can do the work of one in a third of the time.
9
+
Some workloads require massive computational power. Examples include financial risk modeling, 3D image rendering, media transcoding, and genetic sequence analysis. In some cases, these workloads can be broken down into separate subtasks and can be run in parallel so that the tasks are performed in much less time. Three virtual machines (VMs) can do the work of one in a third of the time.
10
+
11
+
As part of the OCR project for the nonprofit, you can use Azure Batch to partition the work of performing the OCR, allowing the processing to be done in parallel so that it takes much less time to complete.
10
12
11
-
As part of the OCR project for the nonprofit, you can use Azure Batch to partition up the work of performing the OCR so that the processing is done in parallel and takes much less time to complete.
12
13
13
14
## The components of Azure Batch
14
15
15
16
You can use Azure Batch for these large-scale parallel and computationally intensive tasks on Azure. At a high level, Azure Batch involves many components working together. Everything needs to take place within the context of an Azure Batch account, which acts as a container for all Batch resources.
16
17
17
-
For some use cases, you might wish to associate this account with an Azure Storage account to download input files and applications for use during execution and storing their results. Within your Azure Batch account, you can create pools of virtual compute nodes, running either Windows or Linux, of the number and CPU and memory size that you specify, which the Azure batch service manages and schedules for you. A Batch account can contain many Batch pools.
18
+
For some use cases, you might wish to associate this account with an Azure Storage account to download input files and applications for use during execution and storing their results. Within your Azure Batch account, you can create pools of virtual compute nodes, running either Windows or Linux. You specify the number, CPU type, and memory size of the nodes, which the Azure batch service manages and schedules for you. A Batch account can contain many Batch pools.
18
19
19
-
Once you've created one or more pools, you create individual jobs, which act like logical containers for all the tasks you schedule and can share common properties. Again, it's perfectly possible to have many jobs.
20
+
Once you create one or more pools, you create individual jobs, which act like logical containers for all the tasks you schedule and can share common properties. Again, it's perfectly possible to have many jobs.
20
21
21
22
And finally, you create tasks, which describe how the work actually gets done. You can either use tasks to directly invoke the command line, or the tasks can run applications that you upload to Azure Storage. We'll see all the main components of Azure Batch working together in the next exercise.
Copy file name to clipboardExpand all lines: learn-pr/azure/run-parallel-tasks-in-azure-batch-with-the-azure-cli/includes/3-design-an-azure-batch-job-to-run-parallel-task.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,20 +1,20 @@
1
1
Azure Batch uses parallel tasks to split a job across compute nodes.
2
2
3
-
As the solution architect for the nonprofit, you want to understand how to design an Azure Batch job and how to enable parallel processing so that these capabilities can be used for the OCR scenario.
3
+
As the solution architect for the nonprofit, you want to first understand how to design an Azure Batch job and how to enable parallel processing. Then, you can use these capabilities for the Optical Character Recognition (OCR) scenario.
4
4
5
-
Here, you'll learn the key elements of a Batch parallel task.
5
+
Here, you learn the key elements of a Batch parallel task.
6
6
7
7
## Using Azure Batch with Azure CLI
8
8
9
-
You decide to create the solution to help the nonprofit with their missionusing Azure Batch and the CLI, so you can get a quick framework in place and incrementally improve it without spending time scaffolding all the foundational code you'd otherwise need to get started. It's true that other methods of creating Azure Batch solutions like .NET and Node.js offer richer and more granular control of Batch and other Azure APIs, but none of those methods can match the agility and immediate feedback you get by issuing commands in the CLI and observing their results.
9
+
You decide to create the solution to help the nonprofit with their mission, by using Azure Batch and the CLI. You want to get a quick framework in place and incrementally improve it without spending time scaffolding all the foundational code you'd otherwise need to get started. It's true that other methods of creating Azure Batch solutions like .NET and Node.js offer richer and more granular control of Batch and other Azure APIs. However, none of those methods can match the agility and immediate feedback you get by issuing commands in the CLI and observing their results.
10
10
11
11
## A sample parallel task
12
12
13
-
To get to grips with Azure Batch and the CLI, you decide on a simple proof-of-concept to demonstrate the different nodes working together in parallel. You'll loop several times in the CLI, add a numbered task per iteration of the loop, and later download and look at the metadata each task generates. This metadata shows the Azure Batch service scheduling tasks as they're created onto different nodes in sequential fashion, so that they all execute their work in parallel.
13
+
To get to grips with Azure Batch and the CLI, you decide on a simple proof-of-concept to demonstrate the different nodes working together in parallel. You plan to loop several times in the CLI, add a numbered task per iteration of the loop, and later download and look at the metadata each task generates. This metadata shows the Azure Batch service scheduling tasks as they're created onto different nodes in sequential fashion, so that they all execute their work in parallel.
14
14
15
-
This kind of proof-of-concept actually underlies many real-world applications of Azure Batch. For example, in the OCR scenario, tasks would also install software like Imagemagick in order to convert the uploaded water-purification images to the TIF format, and would then install Tesseract to perform the conversion work. These tasks would be partitioned in such a way that each worker node would perform a portion of the OCR work in parallel with the others in order to complete faster.
15
+
This kind of proof-of-concept actually underlies many real-world applications of Azure Batch. For example, in the OCR scenario, tasks would install software like Imagemagick to convert the uploaded water-purification images to the TIF format, and would then install Tesseract to perform the conversion work. These tasks would be partitioned so that each worker node would perform a portion of the OCR work in parallel with the others, allowing the work to complete faster.
16
16
17
-
Even this proof of concept will demonstrate important components of Azure Batch working in concert together. You'll create a pool, you'll create worker nodes, you'll create a job, and you'll create tasks, all using the Azure CLI to issue commands and get immediate feedback.
17
+
Even this proof of concept demonstrates important components of Azure Batch working in concert together. You create a pool, create worker nodes, create a job, and create tasks, all using the Azure CLI to issue commands and get immediate feedback.
18
18
19
19

Copy file name to clipboardExpand all lines: learn-pr/azure/run-parallel-tasks-in-azure-batch-with-the-azure-cli/includes/4-exercise-create-azure-batch-job-in-cli-to-run-parallel-task.md
+8-8Lines changed: 8 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,15 +1,15 @@
1
1
Azure CLI lets you create and deploy Azure Batch resources from the command line.
2
2
3
-
As the solution architect for the non-profit, you want to start creating a Batch solution for the OCR scenario on Azure. As a first step, you're going to test creating a Batch account, a pool of worker nodes, a job, and tasks, all using the Azure CLI.
3
+
As the solution architect for the non-profit, you want to start creating a Batch solution for the Optical Character Recognition (OCR) scenario on Azure. As a first step, you're going to test creating a Batch account, a pool of worker nodes, a job, and tasks, all using the Azure CLI.
4
4
5
5
The Azure CLI is a cross-platform command line for creating and managing Azure resources. It uses a consistent ```az noun verb --properties``` syntax. CLI commands to manage Azure Batch follow this convention, and so all commands take the form of ```az batch verb --properties```.
6
6
7
7
>[!IMPORTANT]
8
-
>The exercises in this module are optional. To complete the exercises, you'll need your own Azure subscription.
8
+
>The exercises in this module are optional. To complete the exercises, you need your own Azure subscription.
9
9
10
10
## Create the Azure Batch resources
11
11
12
-
First, you'll create the Azure Batch account and pool of worker nodes for your test scenario. You'll then create a Batch job and tasks to run in the pool.
12
+
First, you create the Azure Batch account and pool of worker nodes for your test scenario. Then, you create a Batch job and tasks to run in the pool.
13
13
14
14
### Create a new resource group
15
15
@@ -23,7 +23,7 @@ First, you'll create the Azure Batch account and pool of worker nodes for your t
23
23
24
24
### Create the Azure Batch account
25
25
26
-
1. In portal, open the Cloud Shell by selecting the Cloud Shell icon (**>_**) in the top menu. Use the drop-down menu at the top of the terminal window to make sure you're in **Bash**.
26
+
1. In the portal, open the Cloud Shell by selecting the Cloud Shell icon (**>_**) in the top menu. Use the drop-down menu at the top of the terminal window to make sure you're in **Bash**.
27
27
28
28
1. Create environment variables for your resource group and batch resource.
29
29
@@ -32,7 +32,7 @@ First, you'll create the Azure Batch account and pool of worker nodes for your t
32
32
BATCH_ACCOUNT=batchaccount$RANDOM
33
33
```
34
34
35
-
1. Run the following command in the Cloud Shell to create an Azure Batch account. Choose from one of the following locations:
35
+
1. Create an Azure Batch account by running the following command in the Cloud Shell. Choose from one of the following locations:
@@ -73,7 +73,7 @@ First, you'll create the Azure Batch account and pool of worker nodes for your t
73
73
74
74
When this query shows that the nodes are "steady," they're ready to work. While they show as "resizing," they're actually being provisioned. You can use the up arrow key to repeat the most recent command in the Cloud Shell if you want to query the status of the nodes more than once.
75
75
76
-
1. Run the following command to create a *Batch job*, which is a logical grouping for all the tasks that will run on your nodes:
76
+
1. Run the following command to create a *Batch job*, which is a logical grouping for all the tasks that run on your nodes:
77
77
78
78
```azurecli
79
79
az batch job create \
@@ -95,9 +95,9 @@ First, you'll create the Azure Batch account and pool of worker nodes for your t
95
95
96
96
This loop creates a series of numbered Batch tasks, each of which outputs information about the task ID and node ID involved. By examining what's happening here closely, you can see quite a few characteristics of the way Azure Batch works. Tasks get automatically scheduled onto nodes with available capacity. However, the environment variables we're using (`AZ_BATCH_TASK_ID` and `AZ_BATCH_NODE_ID`), to show information about which task is being run on which node, are only available in the context of the *task user*, a default account created on each node in order to run tasks. Also, the `--command-line` argument doesn't directly run under a shell, so you have to explicitly invoke the shell using `/bin/bash -c` in order to use commands like `printenv` and `grep`.
97
97
98
-
This loop completes the Azure Batch job implementation. We'll leave the Azure Batch account and pool of worker nodes in place for use in the next module, but we'll delete the Batch job. Because the Batch job acts as a container for its Batch tasks, the tasks are also deleted.
98
+
This loop completes the Azure Batch job implementation. Leave the Azure Batch account and pool of worker nodes in place for use in the next module, but delete the Batch job. Because the Batch job acts as a container for its Batch tasks, the tasks are also deleted.
99
99
100
-
1. Run the following command in the Cloud Shell in order to delete the Batch job:
100
+
1. Delete the Batch job by running the following command in the Cloud Shell:
Copy file name to clipboardExpand all lines: learn-pr/azure/run-parallel-tasks-in-azure-batch-with-the-azure-cli/includes/5-exercise-monitor-azure-batch-job-in-cli.md
+8-8Lines changed: 8 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,33 +1,33 @@
1
-
If you're using Azure Batch to run intensive tasks, you'll often want to know how the tasks are going.
1
+
If you're using Azure Batch to run intensive tasks, you often want to know how the tasks are going.
2
2
3
-
As the solution architect for the nonprofit, you'll need ways to access what's happening in Azure Batch. You also need to determine whether the nodes and tasks are working, how far they've progressed, and whether they've produced downloadable output files.
3
+
As the solution architect for the nonprofit, you need ways to access what's happening in Azure Batch. You also need to determine whether the nodes and tasks are working, how far they progressed, and whether they produced downloadable output files.
4
4
5
-
In this unit, you'll create a job similar to the one in the previous unit, monitor the progress of its tasks, and download the files that the tasks produce as output, all using the Azure CLI.
5
+
In this unit, you create a job similar to the one in the previous unit, monitor the progress of its tasks, and download the files that the tasks produce as output, all using the Azure CLI.
6
6
7
7
>[!IMPORTANT]
8
-
>The exercises in this module are optional. To complete the exercises, you'll need your own Azure subscription.
8
+
>The exercises in this module are optional. To complete the exercises, you need your own Azure subscription.
9
9
10
10
## Monitor tasks using the CLI
11
11
12
12
Azure Batch handles all the details of creating and managing compute nodes and scheduling jobs to run on those nodes.
13
13
14
14
When you created tasks in the last unit, Azure Batch queued them to run in the pool. As nodes were determined to have available capacity, Azure Batch scheduled the tasks to run on those nodes.
15
15
16
-
At the end of the last unit, we deleted the job, which had the effect of deleting all its tasks. We'll now create a new job and tasks. It's worth noting that deleting and recreating tasks is a fairly quick operation, and allows you to easily separate out different experiments and projects in Azure Batch. By contrast, deleting and recreating higher-level Batch entities (like accounts) takes more time, because the Batch pool has to be recreated and virtual machines allocated.
16
+
At the end of the last unit, we deleted the job, which had the effect of deleting all its tasks. Now, we want to create a new job and tasks. It's worth noting that deleting and recreating tasks is a fairly quick operation, and allows you to easily separate out different experiments and projects in Azure Batch. By contrast, deleting and recreating higher-level Batch entities (like accounts) takes more time, because the Batch pool has to be recreated and virtual machines allocated.
17
17
18
18
1. Sign into the [Azure portal](https://portal.azure.com/).
19
19
20
20
1. Open the Cloud Shell by selecting the Cloud Shell icon (**>_**) in the top menu.
21
21
22
-
1.Run the following command in the Cloud Shell in order to create a new Batch job:
22
+
1.Create a new Batch job by running the following command in the Cloud Shell:
23
23
24
24
```azurecli
25
25
az batch job create \
26
26
--id myjob2 \
27
27
--pool-id mypool
28
28
```
29
29
30
-
We named it differently to avoid any conflict with the job from the previous unit, which has been scheduled for deletion but not necessarily deleted yet.
30
+
We named it differently to avoid any conflict with the job from the previous unit, which was scheduled for deletion but not necessarily deleted yet.
31
31
32
32
1. Run the following command to create Batch tasks within this new job:
33
33
@@ -53,7 +53,7 @@ At the end of the last unit, we deleted the job, which had the effect of deletin
53
53
54
54
To check that the batch job executed correctly, you can download and examine the files it created.
55
55
56
-
1. Run the following command in the Cloud Shell to show a table of all the files generated by one of the tasks you've created:
56
+
1. Run the following command in the Cloud Shell to show a table of all the files generated by one of the tasks you created:
Copy file name to clipboardExpand all lines: learn-pr/azure/run-parallel-tasks-in-azure-batch-with-the-azure-cli/includes/6-introducing-batch-explorer.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
Many administrators prefer GUI tools, and you can use the Azure Batch Explorer to graphically analyze the status of your batch jobs.
2
2
3
-
Here, you'll learn about the Azure Batch Explorer tool and how it can help you to manage your jobs.
3
+
Here, you learn about the Azure Batch Explorer tool and how it can help you to manage your jobs.
0 commit comments