Skip to content

Commit b91ece9

Browse files
zhengfeiwangzhengfeiwang
authored
[ML][Pipelines] Test: add more E2E tests on serverless compute in pipeline (Azure#29055)
* add node-level serverless test * test: add node serverless cases * fix test parameterize bug * add more test cases
1 parent f9ddf92 commit b91ece9

16 files changed

+284
-15
lines changed

‎sdk/ml/azure-ai-ml/tests/pipeline_job/e2etests/test_pipeline_job.py

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1747,17 +1747,26 @@ def check_name_version_and_register_succeed(output, asset_name):
17471747

17481748
@pytest.mark.skipif(condition=not is_live(), reason="Task 2177353: component version changes across tests.")
17491749
@pytest.mark.parametrize(
1750-
"yaml_path",
1750+
"test_path",
17511751
[
1752-
"./tests/test_configs/pipeline_jobs/serverless_compute/all_types/command/pipeline.yml",
1753-
"./tests/test_configs/pipeline_jobs/serverless_compute/all_types/sweep/pipeline.yml",
1754-
"./tests/test_configs/pipeline_jobs/serverless_compute/all_types/pipeline/pipeline.yml",
1755-
"./tests/test_configs/pipeline_jobs/serverless_compute/all_types/automl/pipeline.yml",
1756-
"./tests/test_configs/pipeline_jobs/serverless_compute/all_types/parallel/pipeline.yml",
1757-
"./tests/test_configs/pipeline_jobs/serverless_compute/all_types/spark/pipeline.yml",
1752+
"command/pipeline_serverless_compute.yml",
1753+
"command/node_serverless_compute.yml",
1754+
"command/node_serverless_compute_no_default.yml",
1755+
"sweep/pipeline_serverless_compute.yml",
1756+
"sweep/node_serverless_compute.yml",
1757+
"sweep/node_serverless_compute_no_default.yml",
1758+
"pipeline/pipeline_serverless_compute.yml",
1759+
"pipeline/node_serverless_compute.yml",
1760+
"automl/pipeline_with_instance_type.yml",
1761+
"automl/pipeline_without_instance_type.yml",
1762+
"automl/pipeline_with_instance_type_no_default.yml",
1763+
"parallel/pipeline_serverless_compute.yml",
1764+
"spark/pipeline_serverless_compute.yml",
1765+
"spark/node_serverless_compute_no_default.yml",
17581766
],
17591767
)
1760-
def test_serverless_compute_in_pipeline(self, client: MLClient, yaml_path: str) -> None:
1768+
def test_serverless_compute_in_pipeline(self, client: MLClient, test_path: str) -> None:
1769+
yaml_path = "./tests/test_configs/pipeline_jobs/serverless_compute/all_types/" + test_path
17611770
pipeline_job = load_job(yaml_path)
17621771
assert_job_cancel(pipeline_job, client)
17631772

‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/automl/pipeline.yml renamed to ‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/automl/pipeline_with_instance_type.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
22
type: pipeline
3-
display_name: Serverless in Pipeline - AutoML
3+
display_name: AutoML with instance type - pipeline serverless compute
4+
experiment_name: Serverless in Pipeline
45
settings:
56
default_compute: azureml:serverless
67
inputs:
@@ -36,7 +37,6 @@ jobs:
3637
outputs:
3738
best_model:
3839
type: mlflow_model
39-
# BUG 2246569: resources should be optional, explicitly specify here with NC/ND-family for dnn tasks.
4040
resources:
4141
instance_type: Standard_NC6
4242
register_model_node:

‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/automl/pipeline_with_instance_type_no_default.yml

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
2+
type: pipeline
3+
display_name: AutoML with instance type - pipeline serverless compute with no default
4+
experiment_name: Serverless in Pipeline
5+
inputs:
6+
text_ner_training_data:
7+
type: mltable
8+
path: ./training-mltable-folder
9+
text_ner_validation_data:
10+
type: mltable
11+
path: ./validation-mltable-folder
12+
jobs:
13+
preprocessing_node:
14+
type: command
15+
component: file:./components/component_preprocessing.yaml
16+
inputs:
17+
train_data: ${{parent.inputs.text_ner_training_data}}
18+
validation_data: ${{parent.inputs.text_ner_validation_data}}
19+
outputs:
20+
preprocessed_train_data:
21+
type: mltable
22+
preprocessed_validation_data:
23+
type: mltable
24+
compute: azureml:cpu-cluster
25+
text_ner_node:
26+
type: automl
27+
task: text_ner
28+
log_verbosity: info
29+
primary_metric: accuracy
30+
limits:
31+
max_trials: 1
32+
timeout_minutes: 60
33+
target_column_name: label
34+
training_data: ${{parent.jobs.preprocessing_node.outputs.preprocessed_train_data}}
35+
validation_data: ${{parent.jobs.preprocessing_node.outputs.preprocessed_validation_data}}
36+
outputs:
37+
best_model:
38+
type: mlflow_model
39+
resources:
40+
instance_type: Standard_NC6
41+
register_model_node:
42+
type: command
43+
component: file:./components/component_register_model.yaml
44+
inputs:
45+
model_input_path: ${{parent.jobs.text_ner_node.outputs.best_model}}
46+
model_base_name: paper_categorization_model
47+
compute: azureml:cpu-cluster

‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/automl/pipeline_without_instance_type.yml

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
2+
type: pipeline
3+
display_name: AutoML without instance type - pipeline serverless compute
4+
experiment_name: Serverless in Pipeline
5+
settings:
6+
default_compute: azureml:serverless
7+
inputs:
8+
text_ner_training_data:
9+
type: mltable
10+
path: ./training-mltable-folder
11+
text_ner_validation_data:
12+
type: mltable
13+
path: ./validation-mltable-folder
14+
jobs:
15+
preprocessing_node:
16+
type: command
17+
component: file:./components/component_preprocessing.yaml
18+
inputs:
19+
train_data: ${{parent.inputs.text_ner_training_data}}
20+
validation_data: ${{parent.inputs.text_ner_validation_data}}
21+
outputs:
22+
preprocessed_train_data:
23+
type: mltable
24+
preprocessed_validation_data:
25+
type: mltable
26+
text_ner_node:
27+
type: automl
28+
task: text_ner
29+
log_verbosity: info
30+
primary_metric: accuracy
31+
limits:
32+
max_trials: 1
33+
timeout_minutes: 60
34+
target_column_name: label
35+
training_data: ${{parent.jobs.preprocessing_node.outputs.preprocessed_train_data}}
36+
validation_data: ${{parent.jobs.preprocessing_node.outputs.preprocessed_validation_data}}
37+
outputs:
38+
best_model:
39+
type: mlflow_model
40+
register_model_node:
41+
type: command
42+
component: file:./components/component_register_model.yaml
43+
inputs:
44+
model_input_path: ${{parent.jobs.text_ner_node.outputs.best_model}}
45+
model_base_name: paper_categorization_model

‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/command/node_serverless_compute.yml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
2+
type: pipeline
3+
display_name: Command - node serverless compute
4+
experiment_name: Serverless in Pipeline
5+
settings:
6+
default_compute: azureml:cpu-cluster
7+
jobs:
8+
hello_world_compute:
9+
command: echo "Hello World"
10+
environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
11+
hello_world_serverless:
12+
command: echo "Hello World"
13+
environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
14+
compute: azureml:serverless
15+
resources:
16+
instance_type: Standard_D3_v2

‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/command/node_serverless_compute_no_default.yml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
2+
type: pipeline
3+
display_name: Command - node serverless compute with no default
4+
experiment_name: Serverless in Pipeline
5+
jobs:
6+
hello_world_compute:
7+
command: echo "Hello World"
8+
environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
9+
compute: azureml:cpu-cluster
10+
hello_world_serverless:
11+
command: echo "Hello World"
12+
environment: azureml:AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest
13+
compute: azureml:serverless
14+
resources:
15+
instance_type: Standard_D3_v2

‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/command/pipeline.yml renamed to ‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/command/pipeline_serverless_compute.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
22
type: pipeline
3-
display_name: Serverless in Pipeline - Command
3+
display_name: Command - pipeline serverless compute
4+
experiment_name: Serverless in Pipeline
45
settings:
56
default_compute: azureml:serverless
67
jobs:

‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/parallel/pipeline.yml renamed to ‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/parallel/pipeline_serverless_compute.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
22
type: pipeline
3-
display_name: Serverless in Pipeline - Parallel
3+
display_name: Parallel - pipeline serverless compute
4+
experiment_name: Serverless in Pipeline
45
settings:
56
default_compute: azureml:serverless
67
jobs:

‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/pipeline/node_serverless_compute.yml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
2+
type: pipeline
3+
display_name: Pipeline - node serverless compute
4+
experiment_name: Serverless in Pipeline
5+
settings:
6+
default_compute: azureml:cpu-cluster
7+
jobs:
8+
pipeline_node_compute:
9+
type: pipeline
10+
component: ./pipeline_component.yml
11+
pipeline_node_serverless:
12+
type: pipeline
13+
component: ./pipeline_component_serverless.yml

‎sdk/ml/azure-ai-ml/tests/test_configs/pipeline_jobs/serverless_compute/all_types/pipeline/pipeline_component_serverless.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
$schema: https://azuremlschemas.azureedge.net/development/pipelineComponent.schema.json
2+
type: pipeline
3+
name: simple_pipeline_component_with_serverless
4+
jobs:
5+
node_serverless:
6+
type: command
7+
component: ./component.yml
8+
compute: azureml:serverless
9+
resources:
10+
instance_type: Standard_D3_v2

0 commit comments

Comments
 (0)