Skip to content

Commit c2d6166

Browse files
v-thepetv-thepet
committed
quick run python
1 parent b961c5f commit c2d6166

File tree

1 file changed

+86
-81
lines changed

1 file changed

+86
-81
lines changed

‎articles/batch/quick-run-python.md

Lines changed: 86 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -9,66 +9,78 @@ ms.custom: seo-python-october2019, mvc, devx-track-python, mode-api
99

1010
# Quickstart: Use Python API to run an Azure Batch job
1111

12-
Get started with Azure Batch by using the Python API to run an Azure Batch job from an app. The app uploads input data files to Azure Storage and creates a pool of Batch compute nodes (virtual machines). It then creates a job that runs tasks to process each input file in the pool using a basic command.
12+
This quickstart shows you how to get started with Azure Batch by using the [Azure Batch libraries for Python](/python/api/overview/azure/batch). The quickstart uses a Python app to do the following actions:
1313

14-
After completing this quickstart, you'll understand key concepts of the Batch service and be ready to try Batch with more realistic workloads at larger scale.
14+
> [!div class="checklist"]
15+
> - Upload three text files to a blob container in Azure Storage as inputs for Batch task processing.
16+
> - Create a *pool* of two compute *nodes* running Ubuntu 20.04 LTS OS.
17+
> - Create a *job* and three *tasks* to run on the nodes. Each task processes one of the input files by using a Bash shell command line.
18+
> - Display the output files that the tasks return.
1519
16-
![Overview of the Azure Batch workflow](./media/quick-run-python/overview-of-the-azure-batch-workflow.png)
20+
After you complete this quickstart, you understand the [key concepts of the Batch service](batch-service-workflow-features.md) and are ready to use Batch with more realistic, larger scale workloads.
1721

1822
## Prerequisites
1923

20-
- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).
24+
- An Azure account with an active subscription. If you don't have one, [create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).
2125

22-
- A Batch account and a linked Azure Storage account. To create these accounts, see the Batch quickstarts using the [Azure portal](quick-create-portal.md) or [Azure CLI](quick-create-cli.md).
26+
- A Batch account with a linked Azure Storage account. You can create the accounts by using any of the following methods:<br>[Azure CLI](quick-create-cli.md) | [Azure portal](quick-create-portal.md) | [Bicep](quick-create-bicep.md) | [ARM template](quick-create-template.md) | [Terraform](quick-create-terraform.md)
2327

24-
- [Python](https://python.org/downloads) version 3.6 or later, including the [pip](https://pip.pypa.io/en/stable/installing/) package manager.
28+
- [Python](https://python.org/downloads) version 3.6 or later, which includes the [pip](https://pip.pypa.io/en/stable/installing) package manager.
2529

26-
## Sign in to Azure
30+
## Run the app
2731

28-
Sign in to the Azure portal at [https://portal.azure.com](https://portal.azure.com).
32+
To run the Python app, you need to provide your Batch and Storage account names, account keys, and Batch account endpoint. You can get this information from the Azure portal, Azure APIs, or command-line tools. To get these values from the Azure portal:
33+
34+
1. From the Azure search bar, search for and select your Batch account name.
35+
1. On your Batch account page, select **Keys** from the left navigation.
36+
1. On the **Keys** page, copy the following values:
37+
38+
- **Batch account**
39+
- **Account endpoint**
40+
- **Primary access key**
41+
- **Storage account name**
42+
- **Key1**
2943

30-
[!INCLUDE [batch-common-credentials](../../includes/batch-common-credentials.md)]
44+
To run the app:
3145

32-
## Download the sample
46+
1. Download or clone the [Azure Batch Python Quickstart](https://github.com/Azure-Samples/batch-python-quickstart) app from GitHub. Use the following command to clone the app repo with a Git client:
3347

34-
[Download or clone the sample app](https://github.com/Azure-Samples/batch-python-quickstart) from GitHub. To clone the sample app repo with a Git client, use the following command:
48+
```bash
49+
git clone https://github.com/Azure-Samples/batch-python-quickstart.git
50+
```
3551

36-
```bash
37-
git clone https://github.com/Azure-Samples/batch-python-quickstart.git
38-
```
52+
1. Switch to the directory that contains the Python script file *python_quickstart_client.py*, and install the required packages by using `pip`.
3953

40-
Go to the directory that contains the Python script `python_quickstart_client.py`.
54+
```bash
55+
pip install -r requirements.txt
56+
```
4157

42-
In your Python development environment, install the required packages using `pip`.
58+
1. Open the file *config.py*, and update the following strings, replacing the placeholders with the values you copied from your accounts.:
4359

44-
```bash
45-
pip install -r requirements.txt
46-
```
60+
```python
61+
BATCH_ACCOUNT_NAME = '<batch account name>'
62+
BATCH_ACCOUNT_KEY = '<primary access key>'
63+
BATCH_ACCOUNT_URL = '<account endpoint>'
64+
STORAGE_ACCOUNT_NAME = '<storage account name>'
65+
STORAGE_ACCOUNT_KEY = '<key1>'
66+
```
4767

48-
Open the file `config.py`. Update the Batch and storage account credential strings with the values you obtained for your accounts. For example:
49-
50-
```Python
51-
BATCH_ACCOUNT_NAME = 'mybatchaccount'
52-
BATCH_ACCOUNT_KEY = 'xxxxxxxxxxxxxxxxE+yXrRvJAqT9BlXwwo1CwF+SwAYOxxxxxxxxxxxxxxxx43pXi/gdiATkvbpLRl3x14pcEQ=='
53-
BATCH_ACCOUNT_URL = 'https://mybatchaccount.mybatchregion.batch.azure.com'
54-
STORAGE_ACCOUNT_NAME = 'mystorageaccount'
55-
STORAGE_ACCOUNT_KEY = 'xxxxxxxxxxxxxxxxy4/xxxxxxxxxxxxxxxxfwpbIC5aAWA8wDu+AFXZB827Mt9lybZB1nUcQbQiUrkPtilK5BQ=='
56-
```
68+
1. Run the script to see the Batch workflow in action.
5769

58-
## Run the app
70+
```bash
71+
python python_quickstart_client.py
72+
```
5973

60-
To see the Batch workflow in action, run the script:
74+
Typical execution time is approximately three minutes. Initial pool node setup takes the most time.
6175

62-
```bash
63-
python python_quickstart_client.py
64-
```
76+
### App output
6577

66-
After running the script, review the code to learn what each part of the application does.
78+
During app execution, there's a pause at `Monitoring all tasks for 'Completed' state, timeout in 00:30:00...` while the pool's compute nodes are started. Tasks are queued to run as soon as the first compute node is running. You can monitor node, task, and job status from your Batch account page in the Azure portal.
6779

68-
When you run the sample application, the console output is similar to the following. During execution, you experience a pause at `Monitoring all tasks for 'Completed' state, timeout in 00:30:00...` while the pool's compute nodes are started. Tasks are queued to run as soon as the first compute node is running. Go to your Batch account in the [Azure portal](https://portal.azure.com) to monitor the pool, compute nodes, job, and tasks in your Batch account.
80+
The sample application returns output similar to the following example:
6981

7082
```output
71-
Sample start: 11/26/2018 4:02:54 PM
83+
Sample start: 11/26/2012 4:02:54 PM
7284
7385
Uploading file taskdata0.txt to container [input]...
7486
Uploading file taskdata1.txt to container [input]...
@@ -79,7 +91,7 @@ Adding 3 tasks to job [PythonQuickstartJob]...
7991
Monitoring all tasks for 'Completed' state, timeout in 00:30:00...
8092
```
8193

82-
After tasks complete, you see output similar to the following for each task:
94+
After each task completes, you see output similar to the following example:
8395

8496
```output
8597
Printing task output...
@@ -90,58 +102,49 @@ Batch processing began with mainframe computers and punch cards. Today it still
90102
...
91103
```
92104

93-
Typical execution time is approximately 3 minutes when you run the application in its default configuration. Initial pool setup takes the most time.
94-
95105
## Review the code
96106

97-
The Python app in this quickstart does the following:
98-
99-
- Uploads three small text files to a blob container in your Azure storage account. These files are inputs for processing by Batch tasks.
100-
- Creates a pool of two compute nodes running Ubuntu 20.04 LTS.
101-
- Creates a job and three tasks to run on the nodes. Each task processes one of the input files using a Bash shell command line.
102-
- Displays files returned by the tasks.
103-
104-
See the file `python_quickstart_client.py` and the following sections for details.
107+
The Python app in this quickstart takes the following steps:
105108

106-
### Preliminaries
109+
### Upload resource files
107110

108-
To interact with a storage account, the app creates a [BlobServiceClient](/python/api/azure-storage-blob/azure.storage.blob.blobserviceclient) object.
111+
1. To interact with the Storage account, the app creates a [BlobServiceClient](/python/api/azure-storage-blob/azure.storage.blob.blobserviceclient) object.
109112

110-
```python
111-
blob_service_client = BlobServiceClient(
112-
account_url=f"https://{config.STORAGE_ACCOUNT_NAME}.{config.STORAGE_ACCOUNT_DOMAIN}/",
113-
credential=config.STORAGE_ACCOUNT_KEY
114-
)
115-
```
116-
117-
The app uses the `blob_service_client` reference to create a container in the storage account and to upload data files to the container. The files in storage are defined as Batch [ResourceFile](/python/api/azure-batch/azure.batch.models.resourcefile) objects that Batch can later download to compute nodes.
118-
119-
```python
120-
input_file_paths = [os.path.join(sys.path[0], 'taskdata0.txt'),
121-
os.path.join(sys.path[0], 'taskdata1.txt'),
122-
os.path.join(sys.path[0], 'taskdata2.txt')]
113+
```python
114+
blob_service_client = BlobServiceClient(
115+
account_url=f"https://{config.STORAGE_ACCOUNT_NAME}.{config.STORAGE_ACCOUNT_DOMAIN}/",
116+
credential=config.STORAGE_ACCOUNT_KEY
117+
)
118+
```
123119

124-
input_files = [
125-
upload_file_to_container(blob_service_client, input_container_name, file_path)
126-
for file_path in input_file_paths]
127-
```
120+
1. The app uses the `blob_service_client` reference to create a container in the Storage account and upload data files to the container. The files in storage are defined as Batch [ResourceFile](/python/api/azure-batch/azure.batch.models.resourcefile) objects that Batch can later download to compute nodes.
128121

129-
The app creates a [BatchServiceClient](/python/api/azure.batch.batchserviceclient) object to create and manage pools, jobs, and tasks in the Batch service. The Batch client in the sample uses shared key authentication. Batch also supports Azure Active Directory authentication.
122+
```python
123+
input_file_paths = [os.path.join(sys.path[0], 'taskdata0.txt'),
124+
os.path.join(sys.path[0], 'taskdata1.txt'),
125+
os.path.join(sys.path[0], 'taskdata2.txt')]
126+
127+
input_files = [
128+
upload_file_to_container(blob_service_client, input_container_name, file_path)
129+
for file_path in input_file_paths]
130+
```
130131

131-
```python
132-
credentials = SharedKeyCredentials(config.BATCH_ACCOUNT_NAME,
133-
config.BATCH_ACCOUNT_KEY)
132+
1. The app creates a [BatchServiceClient](/python/api/azure.batch.batchserviceclient) object to create and manage pools, jobs, and tasks in the Batch account. The Batch client uses shared key authentication. Batch also supports Azure Active Directory (Azure AD) authentication.
134133

135-
batch_client = BatchServiceClient(
136-
credentials,
137-
batch_url=config.BATCH_ACCOUNT_URL)
138-
```
134+
```python
135+
credentials = SharedKeyCredentials(config.BATCH_ACCOUNT_NAME,
136+
config.BATCH_ACCOUNT_KEY)
137+
138+
batch_client = BatchServiceClient(
139+
credentials,
140+
batch_url=config.BATCH_ACCOUNT_URL)
141+
```
139142

140143
### Create a pool of compute nodes
141144

142-
To create a Batch pool, the app uses the [PoolAddParameter](/python/api/azure-batch/azure.batch.models.pooladdparameter) class to set the number of nodes, VM size, and a pool configuration. Here, a [VirtualMachineConfiguration](/python/api/azure-batch/azure.batch.models.virtualmachineconfiguration) object specifies an [ImageReference](/python/api/azure-batch/azure.batch.models.imagereference) to an Ubuntu Server 20.04 LTS image published in the Azure Marketplace. Batch supports a wide range of Linux and Windows Server images in the Azure Marketplace, as well as custom VM images.
145+
To create a Batch pool, the app uses the [PoolAddParameter](/python/api/azure-batch/azure.batch.models.pooladdparameter) class to set the number of nodes, virtual machine (VM) size, and pool configuration. The following [VirtualMachineConfiguration](/python/api/azure-batch/azure.batch.models.virtualmachineconfiguration) object specifies an [ImageReference](/python/api/azure-batch/azure.batch.models.imagereference) to an Ubuntu Server 20.04 LTS Azure Marketplace image. Batch supports a wide range of Linux and Windows Server Marketplace images as well as custom VM images.
143146

144-
The number of nodes (`POOL_NODE_COUNT`) and VM size (`POOL_VM_SIZE`) are defined constants. The sample by default creates a pool of 2 size *Standard_DS1_v2* nodes. The size suggested offers a good balance of performance versus cost for this quick example.
147+
The number of nodes (`POOL_NODE_COUNT`) and VM size (`POOL_VM_SIZE`) are defined constants. The app by default creates a pool of two size *Standard_DS1_v2* nodes. This size offers a good balance of performance versus cost for this quickstart.
145148

146149
The [pool.add](/python/api/azure-batch/azure.batch.operations.pooloperations) method submits the pool to the Batch service.
147150

@@ -164,7 +167,9 @@ new_pool = batchmodels.PoolAddParameter(
164167

165168
### Create a Batch job
166169

167-
A Batch job is a logical grouping of one or more tasks. A job includes settings common to the tasks, such as priority and the pool to run tasks on. The app uses the [JobAddParameter](/python/api/azure-batch/azure.batch.models.jobaddparameter) class to create a job on your pool. The [job.add](/python/api/azure-batch/azure.batch.operations.joboperations) method adds a job to the specified Batch account. Initially the job has no tasks.
170+
A Batch job is a logical grouping of one or more tasks. A job includes settings common to the tasks, such as priority and the pool to run tasks on.
171+
172+
The app uses the [JobAddParameter](/python/api/azure-batch/azure.batch.models.jobaddparameter) class to create a job on the pool. The [job.add](/python/api/azure-batch/azure.batch.operations.joboperations) method adds a job to the specified Batch account. Initially the job has no tasks.
168173

169174
```python
170175
job = batchmodels.JobAddParameter(
@@ -176,9 +181,9 @@ batch_service_client.job.add(job)
176181

177182
### Create tasks
178183

179-
The app creates a list of task objects using the [TaskAddParameter](/python/api/azure-batch/azure.batch.models.taskaddparameter) class. Each task processes an input `resource_files` object using a `command_line` parameter. In the sample, the command line runs the Bash shell `cat` command to display the text file. This command is a simple example for demonstration purposes. When you use Batch, the command line is where you specify your app or script. Batch provides a number of ways to deploy apps and scripts to compute nodes.
184+
The app creates a list of task objects by using the [TaskAddParameter](/python/api/azure-batch/azure.batch.models.taskaddparameter) class. Each task uses a `command_line` parameter to process an input `resource_files` object. The command line is where you specify your app or script. Batch provides a number of ways to deploy apps and scripts to compute nodes.
180185

181-
Then, the app adds tasks to the job with the [task.add_collection](/python/api/azure-batch/azure.batch.operations.taskoperations) method, which queues them to run on the compute nodes.
186+
In the current example, the command line runs the Bash shell `cat` command to display the text file. Then, the app adds tasks to the job with the [task.add_collection](/python/api/azure-batch/azure.batch.operations.taskoperations) method, which queues the tasks to run on the compute nodes.
182187

183188
```python
184189
tasks = []
@@ -197,7 +202,7 @@ batch_service_client.task.add_collection(job_id, tasks)
197202

198203
### View task output
199204

200-
The app monitors task state to make sure the tasks complete. Then, the app displays the `stdout.txt` file generated by each completed task. When the task runs successfully, the output of the task command is written to `stdout.txt`:
205+
The app monitors task state to make sure the tasks complete. When the task runs successfully, the output of the task command writes to the *stdout.txt* file. The app then displays the *stdout.txt* file each completed task generates.
201206

202207
```python
203208
tasks = batch_service_client.task.list(job_id)
@@ -233,7 +238,7 @@ When no longer needed, delete the resource group, Batch account, and storage acc
233238

234239
## Next steps
235240

236-
In this quickstart, you ran a small app built using the Batch Python API to create a Batch pool and a Batch job. The job ran sample tasks, and downloaded output created on the nodes. Now that you understand the key concepts of the Batch service, you are ready to try Batch with more realistic workloads at larger scale. To learn more about Azure Batch, and walk through a parallel workload with a real-world application, continue to the Batch Python tutorial.
241+
In this quickstart, you ran an app that uses the Batch Python API to create a Batch pool, nodes, job, and tasks. The job uploaded resource files to an storage container, ran tasks on the nodes, and downloaded output created on the nodes. Now that you understand the key concepts of the Batch service, you're ready to use Batch with more realistic, larger-scale workloads. To learn more about Azure Batch and walk through a parallel workload with a real-world application, continue to the Batch Python tutorial.
237242

238243
> [!div class="nextstepaction"]
239244
> [Process a parallel workload with Python](tutorial-parallel-python.md)

0 commit comments

Comments
 (0)