Skip to content

Commit d987cda

Browse files
Merge pull request #232382 from Blackmist/65566-known-issue
documenting v1 known issue
2 parents ecc49d7 + 2bc3f5e commit d987cda

File tree

2 files changed

+86
-2
lines changed

2 files changed

+86
-2
lines changed

articles/machine-learning/v1/how-to-debug-pipelines.md

Lines changed: 82 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -128,7 +128,7 @@ file_path = os.path.join(script_dir, "<file_name>")
128128
- `run_invocation_timeout`: The `run()` method invocation timeout in seconds. (optional; default value is `60`)
129129
- `run_max_try`: Maximum try count of `run()` for a mini-batch. A `run()` is failed if an exception is thrown, or nothing is returned when `run_invocation_timeout` is reached (optional; default value is `3`).
130130

131-
You can specify `mini_batch_size`, `node_count`, `process_count_per_node`, `logging_level`, `run_invocation_timeout`, and `run_max_try` as `PipelineParameter`, so that when you resubmit a pipeline run, you can fine-tune the parameter values. In this example, you use `PipelineParameter` for `mini_batch_size` and `Process_count_per_node` and you will change these values when resubmit a run later.
131+
You can specify `mini_batch_size`, `node_count`, `process_count_per_node`, `logging_level`, `run_invocation_timeout`, and `run_max_try` as `PipelineParameter`, so that when you resubmit a pipeline run, you can fine-tune the parameter values. In this example, you use `PipelineParameter` for `mini_batch_size` and `Process_count_per_node` and you will change these values when you resubmit a run later.
132132

133133
### Parameters for creating the ParallelRunStep
134134

@@ -267,6 +267,86 @@ For more information on using the OpenCensus Python library in this manner, see
267267

268268
In some cases, you may need to interactively debug the Python code used in your ML pipeline. By using Visual Studio Code (VS Code) and debugpy, you can attach to the code as it runs in the training environment. For more information, visit the [interactive debugging in VS Code guide](../how-to-debug-visual-studio-code.md#debug-and-troubleshoot-machine-learning-pipelines).
269269

270+
## HyperdriveStep and AutoMLStep fail with network isolation
271+
272+
After using HyperdriveStep and AutoMLStep, when you attempt to register the model you may receive an error.
273+
274+
* You are using Azure Machine Learning SDK v1.
275+
* Your Azure Machine Learning workspace is configured for network isolation (VNet).
276+
* Your pipeline attempts to register the model generated by the previous step. For example, in the following code the `inputs` parameter is the saved_model from a HyperdriveStep:
277+
278+
```python
279+
register_model_step = PythonScriptStep(script_name='register_model.py',
280+
name="register_model_step01",
281+
inputs=[saved_model],
282+
compute_target=cpu_cluster,
283+
arguments=["--saved-model", saved_model],
284+
allow_reuse=True,
285+
runconfig=rcfg)
286+
```
287+
288+
### Workaround
289+
290+
> [!IMPORTANT]
291+
> This behavior does not occur when using Azure Machine Learning SDK v2.
292+
293+
To work around this error, use the [Run](/python/api/azureml-core/azureml.core.run.run) class to get the model created from the HyperdriveStep or AutoMLStep. The following is an example script that gets the output model from a HyperdriveStep:
294+
295+
```python
296+
%%writefile $script_folder/model_download9.py
297+
import argparse
298+
from azureml.core import Run
299+
from azureml.pipeline.core import PipelineRun
300+
from azureml.core.experiment import Experiment
301+
from azureml.train.hyperdrive import HyperDriveRun
302+
from azureml.pipeline.steps import HyperDriveStepRun
303+
304+
if __name__ == "__main__":
305+
parser = argparse.ArgumentParser()
306+
parser.add_argument(
307+
'--hd_step_name',
308+
type=str, dest='hd_step_name',
309+
help='The name of the step that runs AutoML training within this pipeline')
310+
311+
312+
313+
args = parser.parse_args()
314+
315+
current_run = Run.get_context()
316+
317+
pipeline_run = PipelineRun(current_run.experiment, current_run.experiment.name)
318+
319+
hd_step_run = HyperDriveStepRun((pipeline_run.find_step_run(args.hd_step_name))[0])
320+
hd_best_run = hd_step_run.get_best_run_by_primary_metric()
321+
322+
print(hd_best_run)
323+
hd_best_run.download_file("outputs/model/saved_model.pb", "saved_model.pb")
324+
325+
326+
print("Successfully downloaded model")
327+
```
328+
329+
The file can then be used from a PythonScriptStep:
330+
331+
```python
332+
from azureml.pipeline.steps import PythonScriptStep
333+
conda_dep = CondaDependencies()
334+
conda_dep.add_pip_package("azureml-sdk")
335+
conda_dep.add_pip_package("azureml-pipeline")
336+
337+
rcfg = RunConfiguration(conda_dependencies=conda_dep)
338+
339+
model_download_step = PythonScriptStep(
340+
name="Download Model 9",
341+
script_name="model_download9.py",
342+
arguments=["--hd_step_name", hd_step_name],
343+
compute_target=compute_target,
344+
source_directory=script_folder,
345+
allow_reuse=False,
346+
runconfig=rcfg
347+
)
348+
```
349+
270350
## Next steps
271351

272352
* For a complete tutorial using `ParallelRunStep`, see [Tutorial: Build an Azure Machine Learning pipeline for batch scoring](../tutorial-pipeline-batch-scoring-classification.md).
@@ -275,4 +355,4 @@ In some cases, you may need to interactively debug the Python code used in your
275355

276356
* See the SDK reference for help with the [azureml-pipelines-core](/python/api/azureml-pipeline-core/) package and the [azureml-pipelines-steps](/python/api/azureml-pipeline-steps/) package.
277357

278-
* See the list of [designer exceptions and error codes](../algorithm-module-reference/designer-error-codes.md).
358+
* See the list of [designer exceptions and error codes](../algorithm-module-reference/designer-error-codes.md).

articles/machine-learning/v1/how-to-use-automlstep-in-pipelines.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -366,6 +366,10 @@ print("Registered version {0} of model {1}".format(model.version, model.name))
366366

367367
### Write the PythonScriptStep code
368368

369+
370+
> [!WARNING]
371+
> If you are using the Azure Machine Learning SDK v1, and your workspace is configured for network isolation (VNet), you may receive an error when running this step. For more information, see [HyperdriveStep and AutoMLStep fail with network isolation](how-to-debug-pipelines.md#hyperdrivestep-and-automlstep-fail-with-network-isolation).
372+
369373
The model-registering `PythonScriptStep` uses a `PipelineParameter` for one of its arguments. Pipeline parameters are arguments to pipelines that can be easily set at run-submission time. Once declared, they're passed as normal arguments.
370374

371375
```python

0 commit comments

Comments
 (0)