Skip to content

Commit 8fa3cf1

Browse files
ttorblettorble
authored
Merge pull request #207793 from lgayhardt/pipelineschedule0822
Pipeline Schedule v2
2 parents 9fb9b12 + 48f7339 commit 8fa3cf1

File tree

4 files changed

+602
-0
lines changed

4 files changed

+602
-0
lines changed

‎articles/machine-learning/how-to-schedule-pipeline-job.md

Lines changed: 326 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,326 @@
1+
---
2+
title: Schedule Azure Machine Learning pipeline jobs (preview)
3+
titleSuffix: Azure Machine Learning
4+
description: Learn how to schedule pipeline jobs that allow you to automate routine, time-consuming tasks such as data processing, training, and monitoring.
5+
services: machine-learning
6+
ms.service: machine-learning
7+
ms.subservice: mlops
8+
ms.author: lochen
9+
author: cloga
10+
ms.date: 08/15/2022
11+
ms.topic: how-to
12+
ms.custom: devx-track-python
13+
14+
---
15+
16+
# Schedule machine learning pipeline jobs (preview)
17+
18+
[!INCLUDE [dev v2](../../includes/machine-learning-dev-v2.md)]
19+
20+
> [!IMPORTANT]
21+
> SDK v2 is currently in public preview.
22+
> The preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.
23+
> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
24+
25+
In this article, you'll learn how to programmatically schedule a pipeline to run on Azure. You can create a schedule based on elapsed time. Time-based schedules can be used to take care of routine tasks, such as retrain models or do batch predictions regularly to keep them up-to-date. After learning how to create schedules, you'll learn how to retrieve, update and deactivate them via CLI and SDK.
26+
27+
## Prerequisites
28+
29+
- You must have an Azure subscription to use Azure Machine Learning. If you don't have an Azure subscription, create a free account before you begin. Try the [free or paid version of Azure Machine Learning](https://azure.microsoft.com/free/) today.
30+
31+
# [Azure CLI](#tab/cliv2)
32+
33+
- Install the Azure CLI and the `ml` extension. Follow the installation steps in [Install, set up, and use the CLI (v2)](how-to-configure-cli.md).
34+
35+
- Create an Azure Machine Learning workspace if you don't have one. For workspace creation, see [Install, set up, and use the CLI (v2)](how-to-configure-cli.md).
36+
37+
# [Python](#tab/python)
38+
39+
- Create an Azure Machine Learning workspace if you don't have one.
40+
- The [Azure Machine Learning SDK v2 for Python](/python/api/overview/azure/ml/installv2).
41+
42+
---
43+
44+
## Schedule a pipeline job
45+
46+
To run a pipeline job on a recurring basis, you'll need to create a schedule. A `Schedule` associates a job, and a trigger. The trigger can either be `cron` that use cron expression to describe the wait between runs or `recurrence` that specify using what frequency to trigger job. In each case, you need to define a pipeline job first, it can be existing pipeline jobs or a pipeline job define inline, refer to [Create a pipeline job in CLI](how-to-create-component-pipelines-cli.md) and [Create a pipeline job in SDK](how-to-create-component-pipeline-python.md).
47+
48+
You can schedule a pipeline job yaml in local or an existing pipeline job in workspace.
49+
50+
## Create a schedule
51+
52+
### Create a time-based schedule with recurrence pattern
53+
54+
# [Azure CLI](#tab/cliv2)
55+
56+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
57+
58+
:::code language="yaml"source="~/azureml-examples-main/cli/schedules/recurrence-job-schedule.yml":::
59+
60+
`trigger` contains the following properties:
61+
62+
- **(Required)** `type` specifies the schedule type is `recurrence`. It can also be `cron`, see details in the next section.
63+
64+
# [Python](#tab/python)
65+
66+
[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
67+
68+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=create_schedule_recurrence)]
69+
70+
`RecurrenceTrigger` contains following properties:
71+
72+
- **(Required)** To provide better coding experience, we use `RecurrenceTrigger` for recurrence schedule.
73+
74+
---
75+
76+
- **(Required)** `frequency` specifies the unit of time that describes how often the schedule fires. Can be `minute`, `hour`, `day`, `week`, `month`.
77+
78+
- **(Required)** `interval` specifies how often the schedule fires based on the frequency, which is the number of time units to wait until the schedule fires again.
79+
80+
- (Optional) `schedule` defines the recurrence pattern, containing `hours`, `minutes`, and `weekdays`.
81+
- When `frequency` is `day`, pattern can specify `hours` and `minutes`.
82+
- When `frequency` is `week` and `month`, pattern can specify `hours`, `minutes` and `weekdays`.
83+
- `hours` should be an integer or a list, from 0 to 23.
84+
- `minutes` should be an integer or a list, from 0 to 59.
85+
- `weekdays` can be a string or list from `monday` to `sunday`.
86+
- If `schedule` is omitted, the job(s) will be triggered according to the logic of `start_time`, `frequency` and `interval`.
87+
88+
- (Optional) `start_time` describes the start date and time with timezone. If `start_time` is omitted, start_time will be equal to the job created time. If the start time is in the past, the first job will run at the next calculated run time.
89+
90+
- (Optional) `end_time` describes the end date and time with timezone. If `end_time` is omitted, the schedule will continue trigger jobs until the schedule is manually disabled.
91+
92+
- (Optional) `time_zone` specifies the time zone of the recurrence. If omitted, by default is UTC. To learn more about timezone values, see [appendix for timezone values](reference-yaml-schedule.md#appendix).
93+
94+
### Create a time-based schedule with cron expression
95+
96+
# [Azure CLI](#tab/cliv2)
97+
98+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
99+
100+
:::code language="yaml" source="~/azureml-examples-main/cli/schedules/cron-job-schedule.yml":::
101+
102+
The `trigger` section defines the schedule details and contains following properties:
103+
104+
- **(Required)** `type` specifies the schedule type is `cron`.
105+
106+
# [Python](#tab/python)
107+
108+
[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
109+
110+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=create_schedule_cron)]
111+
112+
The `CronTrigger` section defines the schedule details and contains following properties:
113+
114+
- **(Required)** To provide better coding experience, we use `CronTrigger` for recurrence schedule.
115+
116+
---
117+
118+
- **(Required)** `expression` uses standard crontab expression to express a recurring schedule. A single expression is composed of five space-delimited fields:
119+
120+
`MINUTES HOURS DAYS MONTHS DAYS-OF-WEEK`
121+
122+
- A single wildcard (`*`), which covers all values for the field. So a `*` in days means all days of a month (which varies with month and year).
123+
- The `expression: "15 16 * * 1"` in the sample above means the 16:15PM on every Monday.
124+
- The table below lists the valid values for each field:
125+
126+
| Field | Range | Comment |
127+
|----------------|----------|-----------------------------------------------------------|
128+
| `MINUTES` | 0-59 | - |
129+
| `HOURS` | 0-23 | - |
130+
| `DAYS` | - | Not supported. The value will be ignored and treat as `*`. |
131+
| `MONTHS` | - | Not supported. The value will be ignored and treat as `*`. |
132+
| `DAYS-OF-WEEK` | 0-6 | Zero (0) means Sunday. Names of days also accepted. |
133+
134+
- To learn more about how to use crontab expression, see [Crontab Expression wiki on GitHub ](https://github.com/atifaziz/NCrontab/wiki/Crontab-Expression).
135+
136+
> [!IMPORTANT]
137+
> `DAYS` and `MONTH` are not supported. If you pass a value, it will be ignored and treat as `*`.
138+
139+
- (Optional) `start_time` specifies the start date and time with timezone of the schedule. `start_time: "2022-05-10T10:15:00-04:00"` means the schedule starts from 10:15:00AM on 2022-05-10 in UTC-4 timezone. If `start_time` is omitted, the `start_time` will be equal to schedule creation time. If the start time is in the past, the first job will run at the next calculated run time.
140+
141+
- (Optional) `end_time` describes the end date and time with timezone. If `end_time` is omitted, the schedule will continue trigger jobs until the schedule is manually disabled.
142+
143+
- (Optional) `time_zone`specifies the time zone of the expression. If omitted, by default is UTC. See [appendix for timezone values](reference-yaml-schedule.md#appendix).
144+
145+
### Change runtime settings when defining schedule
146+
147+
When defining a schedule using an existing job, you can change the runtime settings of the job. Using this approach, you can define multi-schedules using the same job with different inputs.
148+
149+
# [Azure CLI](#tab/cliv2)
150+
151+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
152+
153+
:::code language="yaml" source="~/azureml-examples-main/cli/schedules/cron-with-settings-job-schedule.yml":::
154+
155+
# [Python](#tab/python)
156+
157+
[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
158+
159+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=change_run_settings)]
160+
161+
---
162+
163+
Following properties can be changed when defining schedule:
164+
165+
| Property | Description |
166+
| --- | --- |
167+
|settings| A dictionary of settings to be used when running the pipeline job. |
168+
|inputs| A dictionary of inputs to be used when running the pipeline job. |
169+
|outputs| A dictionary of inputs to be used when running the pipeline job. |
170+
|experiment_name|Experiment name of triggered job.|
171+
172+
### Expressions supported in schedule
173+
174+
When define schedule, we support following expression that will be resolved to real value during job runtime.
175+
176+
| Expression | Description |Supported properties|
177+
|----------------|----------------|-------------|
178+
|`${{create_context.trigger_time}}`|The time when the schedule is triggered.|String type inputs of pipeline job|
179+
|`${{name}}`|The name of job.|outputs.path of pipeline job|
180+
181+
## Manage schedule
182+
183+
### Create schedule
184+
185+
# [Azure CLI](#tab/cliv2)
186+
187+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
188+
189+
After you create the schedule yaml, you can use the following command to create a schedule via CLI.
190+
191+
:::code language="azurecli" source="~/azureml-examples-main/cli/schedules/schedule.sh" ID="create_schedule":::
192+
193+
# [Python](#tab/python)
194+
195+
[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
196+
197+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=create_schedule)]
198+
199+
---
200+
201+
### Check schedule detail
202+
203+
# [Azure CLI](#tab/cliv2)
204+
205+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
206+
207+
:::code language="azurecli" source="~/azureml-examples-main/cli/schedules/schedule.sh" ID="show_schedule":::
208+
209+
# [Python](#tab/python)
210+
211+
[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
212+
213+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=show_schedule)]
214+
215+
---
216+
217+
### List schedules in a workspace
218+
219+
# [Azure CLI](#tab/cliv2)
220+
221+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
222+
223+
:::code language="azurecli" source="~/azureml-examples-main/cli/schedules/schedule.sh" ID="list_schedule":::
224+
225+
# [Python](#tab/python)
226+
227+
[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
228+
229+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=list_schedule)]
230+
231+
---
232+
233+
### Update a schedule
234+
235+
# [Azure CLI](#tab/cliv2)
236+
237+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
238+
239+
:::code language="azurecli" source="~/azureml-examples-main/cli/schedules/schedule.sh" ID="update_schedule":::
240+
241+
# [Python](#tab/python)
242+
243+
[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
244+
245+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=create_schedule)]
246+
247+
---
248+
249+
### Disable a schedule
250+
251+
# [Azure CLI](#tab/cliv2)
252+
253+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
254+
255+
:::code language="azurecli" source="~/azureml-examples-main/cli/schedules/schedule.sh" ID="disable_schedule":::
256+
257+
# [Python](#tab/python)
258+
259+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=disable_schedule)]
260+
261+
---
262+
263+
### Enable a schedule
264+
265+
# [Azure CLI](#tab/cliv2)
266+
267+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
268+
269+
:::code language="azurecli" source="~/azureml-examples-main/cli/schedules/schedule.sh" ID="enable_schedule":::
270+
271+
# [Python](#tab/python)
272+
273+
[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
274+
275+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=enable_schedule)]
276+
277+
---
278+
279+
## Query triggered jobs from a schedule
280+
281+
All the display name of jobs triggered by schedule will have the display name as <schedule_name>-YYYYMMDDThhmmssZ. For example, if a schedule with a name of named-schedule is created with a scheduled run every 12 hours starting at 6 AM on Jan 1 2021, then the display names of the jobs created will be as follows:
282+
283+
- named-schedule-20210101T060000Z
284+
- named-schedule-20210101T180000Z
285+
- named-schedule-20210102T060000Z
286+
- named-schedule-20210102T180000Z, and so on
287+
288+
:::image type="content" source="media/how-to-schedule-pipeline-job/schedule-triggered-pipeline-jobs.png" alt-text="Screenshot of the jobs tab in the Azure Machine Learning studio filtering by job display name." lightbox= "media/how-to-schedule-pipeline-job/schedule-triggered-pipeline-jobs.png":::
289+
290+
You can also apply [Azure CLI JMESPath query](/cli/azure/query-azure-cli) to query the jobs triggered by a schedule name.
291+
292+
:::code language="azurecli" source="~/azureml-examples-main/CLI/schedules/schedule.sh" ID="query_triggered_jobs":::
293+
294+
---
295+
296+
## Delete a schedule
297+
298+
> [!IMPORTANT]
299+
> A schedule must be disabled to be deleted.
300+
301+
# [Azure CLI](#tab/cliv2)
302+
303+
[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
304+
305+
:::code language="azurecli" source="~/azureml-examples-main/cli/schedules/schedule.sh" ID="delete_schedule":::
306+
307+
# [Python](#tab/python)
308+
309+
[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
310+
311+
[!notebook-python[] (~/azureml-examples-main/sdk/schedules/job-schedule.ipynb?name=delete_schedule)]
312+
313+
---
314+
315+
## Next steps
316+
317+
* Learn more about the [CLI (v2) schedule YAML schema](./reference-yaml-schedule.md).
318+
* Learn how to [create pipeline job in CLI v2](how-to-create-component-pipelines-cli.md).
319+
* Learn how to [create pipeline job in SDK v2](how-to-create-component-pipeline-python.md).
320+
* Learn more about [CLI (v2) core YAML syntax](reference-yaml-core-syntax.md).
321+
* Learn more about [Pipelines](concept-ml-pipelines.md).
322+
* Learn more about [Component](concept-component.md).
323+
324+
325+
> [!NOTE]
326+
> Information the user should notice even if skimming

‎articles/machine-learning/media/how-to-schedule-pipeline-job/schedule-triggered-pipeline-jobs.png

141 KB
Loading

0 commit comments

Comments
 (0)