|
| 1 | +--- |
| 2 | +title: Tracking for ML experiments with MLflow and CLI (v2) |
| 3 | +titleSuffix: Azure Machine Learning |
| 4 | +description: Set up MLflow Tracking with Azure Machine Learning to log metrics and artifacts from ML models with MLflow or the Azure Machine Learning CLI (v2) |
| 5 | +services: machine-learning |
| 6 | +author: abeomor |
| 7 | +ms.author: osomorog |
| 8 | +ms.service: machine-learning |
| 9 | +ms.subservice: mlops |
| 10 | +ms.reviewer: nibaccam |
| 11 | +ms.date: 12/16/2021 |
| 12 | +ms.topic: how-to |
| 13 | +ms.custom: devx-track-python, mlflow |
| 14 | +--- |
| 15 | + |
| 16 | +# Track ML experiments and models with MLflow or the Azure Machine Learning CLI (v2) (preview) |
| 17 | + |
| 18 | + |
| 19 | +In this article, learn how to enable MLflow's tracking URI and logging API, collectively known as [MLflow Tracking](https://mlflow.org/docs/latest/quickstart.html#using-the-tracking-api), to connect Azure Machine Learning as the backend of your MLflow experiments. You can accomplish this connection with either the MLflow Python API or the [Azure Machine Learning CLI (v2) (preview)](how-to-train-cli.md) in your terminal. You also learn how to use [MLflow's Model Registry](https://mlflow.org/docs/latest/model-registry.html) capabilities with Azure Machine Learning. |
| 20 | + |
| 21 | +[MLflow](https://www.mlflow.org) is an open-source library for managing the lifecycle of your machine learning experiments. MLflow Tracking is a component of MLflow that logs and tracks your training run metrics and model artifacts, no matter your experiment's environment--locally on your computer, on a remote compute target, a virtual machine, or an [Azure Databricks cluster](how-to-use-mlflow-azure-databricks.md). |
| 22 | + |
| 23 | +See [MLflow and Azure Machine Learning](concept-mlflow.md) for all supported MLflow and Azure Machine Learning functionality including MLflow Project support (preview) and model deployment. |
| 24 | + |
| 25 | +> [!TIP] |
| 26 | +> The information in this document is primarily for data scientists and developers who want to monitor the model training process. If you are an administrator interested in monitoring resource usage and events from Azure Machine Learning, such as quotas, completed training runs, or completed model deployments, see [Monitoring Azure Machine Learning](monitor-azure-machine-learning.md). |
| 27 | +
|
| 28 | +> [!NOTE] |
| 29 | +> You can use the [MLflow Skinny client](https://github.com/mlflow/mlflow/blob/master/README_SKINNY.rst) which is a lightweight MLflow package without SQL storage, server, UI, or data science dependencies. This is recommended for users who primarily need the tracking and logging capabilities without importing the full suite of MLflow features including deployments. |
| 30 | +
|
| 31 | +## Prerequisites |
| 32 | + |
| 33 | +* Install the `azureml-mlflow` package. |
| 34 | + * This package automatically brings in `azureml-core` of the [The Azure Machine Learning Python SDK](/python/api/overview/azure/ml/install), which provides the connectivity for MLflow to access your workspace. |
| 35 | + |
| 36 | +* [Create an Azure Machine Learning Workspace](how-to-manage-workspace.md). |
| 37 | + * See which [access permissions you need to perform your MLflow operations with your workspace](how-to-assign-roles.md#mlflow-operations). |
| 38 | + |
| 39 | +* Install and [set up CLI (v2)](how-to-configure-cli.md#prerequisites) and make sure you install the ml extension. |
| 40 | +[!INCLUDE [preview disclaimer](../../includes/machine-learning-preview-generic-disclaimer.md)] |
| 41 | + |
| 42 | + |
| 43 | +## Track runs from your local machine |
| 44 | + |
| 45 | +MLflow Tracking with Azure Machine Learning lets you store the logged metrics and artifacts runs that were executed on your local machine into your Azure Machine Learning workspace. |
| 46 | + |
| 47 | +### Set up tracking environment |
| 48 | + |
| 49 | +To track a local run, you need to point your local machine to the Azure Machine Learning MLflow Tracking URI. |
| 50 | + |
| 51 | +>[!IMPORTANT] |
| 52 | +> Make sure you are logged in to your Azure account, otherwise the tracking URI returns an empty string. |
| 53 | +
|
| 54 | +# [MLflow SDK](#tab/mlflow) |
| 55 | + |
| 56 | +The following code uses `mlflow` and the [`subprocess`](https://docs.python.org/3/library/subprocess.html) classes in Python to run the Azure Machine Learning CLI (v2) command to retrieve the unique MLFLow tracking URI associated with your workspace. Then the method [`set_tracking_uri()`](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.set_tracking_uri) points the MLflow tracking URI to that URI. |
| 57 | + |
| 58 | +```Python |
| 59 | +import mlflow |
| 60 | +import subprocess |
| 61 | + |
| 62 | +#Get MLfLow URI through the Azure ML CLI (v2) and convert to string |
| 63 | +MLFLOW_TRACKING_URI = subprocess.run(["az", "ml", "workspace", "show", "--query", "mlflow_tracking_uri", "-o", "tsv"], stdout=subprocess.PIPE, text=True) |
| 64 | + |
| 65 | +MLFLOW_TRACKING_URI = str(MLFLOW_TRACKING_URI.stdout).strip() |
| 66 | + |
| 67 | +## Set the MLFLOW TRACKING URI |
| 68 | +mlflow.set_tracking_uri(MLFLOW_TRACKING_URI) |
| 69 | + |
| 70 | +## Make sure the MLflow URI looks something like this: |
| 71 | +## azureml://westus.api.azureml.ms/mlflow/v1.0/subscriptions/<Sub-ID>/resourceGroups/<RG>/providers/Microsoft.MachineLearningServices/workspaces/<WS> |
| 72 | + |
| 73 | +print("MLFlow Tracking URI:",MLFLOW_TRACKING_URI) |
| 74 | +``` |
| 75 | + |
| 76 | +# [Terminal](#tab/terminal) |
| 77 | + |
| 78 | +Another option is to set one of the MLflow environment variables [MLFLOW_TRACKING_URI](https://mlflow.org/docs/latest/tracking.html#logging-to-a-tracking-server) directly in your terminal. |
| 79 | + |
| 80 | +```Azure CLI |
| 81 | +# Configure MLflow to communicate with a Azure Machine Learning-hosted tracking server |
| 82 | +
|
| 83 | +export MLFLOW_TRACKING_URI=$(az ml workspace show --query mlflow_tracking_uri | sed 's/"//g') |
| 84 | +``` |
| 85 | +--- |
| 86 | + |
| 87 | + |
| 88 | +### Set experiment name |
| 89 | + |
| 90 | +All MLflow runs are logged to the active experiment, which can be set with the MLflow SDK or Azure CLI. |
| 91 | + |
| 92 | +# [MLflow SDK](#tab/mlflow) |
| 93 | + |
| 94 | +With MLflow you can use the [`mlflow.set_experiment()`](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.set_experiment) command. |
| 95 | + |
| 96 | +```Python |
| 97 | +experiment_name = 'experiment_with_mlflow' |
| 98 | +mlflow.set_experiment(experiment_name) |
| 99 | +``` |
| 100 | + |
| 101 | +# [Terminal](#tab/terminal) |
| 102 | + |
| 103 | +You can set one of the MLflow environment variables [MLFLOW_EXPERIMENT_NAME or MLFLOW_EXPERIMENT_ID](https://mlflow.org/docs/latest/cli.html#cmdoption-mlflow-run-arg-uri) with the experiment name. |
| 104 | + |
| 105 | +```Azure CLI |
| 106 | +# Configure MLflow to communicate with a Azure Machine Learning-hosted tracking server |
| 107 | +export MLFLOW_EXPERIMENT_NAME="experiment_with_mlflow" |
| 108 | +``` |
| 109 | +--- |
| 110 | + |
| 111 | +### Start training run |
| 112 | + |
| 113 | +After you set the MLflow experiment name, you can start your training run with `start_run()`. Then use `log_metric()` to activate the MLflow logging API and begin logging your training run metrics. |
| 114 | + |
| 115 | +```Python |
| 116 | +import os |
| 117 | +from random import random |
| 118 | + |
| 119 | +with mlflow.start_run() as mlflow_run: |
| 120 | + mlflow.log_param("hello_param", "world") |
| 121 | + mlflow.log_metric("hello_metric", random()) |
| 122 | + os.system(f"echo 'hello world' > helloworld.txt") |
| 123 | + mlflow.log_artifact("helloworld.txt") |
| 124 | +``` |
| 125 | + |
| 126 | +## Track remote runs with Azure Machine Learning CLI (v2) (preview) |
| 127 | + |
| 128 | +[!INCLUDE [preview disclaimer](../../includes/machine-learning-preview-generic-disclaimer.md)] |
| 129 | + |
| 130 | +Remote runs (jobs) let you train your models on more powerful computes, such as GPU enabled virtual machines, or Machine Learning Compute clusters. See [Use compute targets for model training](how-to-set-up-training-targets.md) to learn about different compute options. |
| 131 | + |
| 132 | +MLflow Tracking with Azure Machine Learning lets you store the logged metrics and artifacts from your remote runs into your Azure Machine Learning workspace. Any run with MLflow Tracking code in it logs metrics automatically to the workspace. |
| 133 | + |
| 134 | +First, you should create a `src` subdirectory and create a file with your training code in a `train.py` file in the `src` subdirectory. All your training code will go into the `src` subdirectory, including `train.py`. |
| 135 | + |
| 136 | +The training code is taken from this [MLfLow example](https://github.com/Azure/azureml-examples/blob/main/cli/jobs/basics/src/hello-mlflow.py) in the Azure Machine Learning example repo. |
| 137 | + |
| 138 | +Copy this code into the file: |
| 139 | + |
| 140 | +```Python |
| 141 | +# imports |
| 142 | +import os |
| 143 | +import mlflow |
| 144 | + |
| 145 | +from random import random |
| 146 | + |
| 147 | +# define functions |
| 148 | +def main(): |
| 149 | + mlflow.log_param("hello_param", "world") |
| 150 | + mlflow.log_metric("hello_metric", random()) |
| 151 | + os.system(f"echo 'hello world' > helloworld.txt") |
| 152 | + mlflow.log_artifact("helloworld.txt") |
| 153 | + |
| 154 | + |
| 155 | +# run functions |
| 156 | +if __name__ == "__main__": |
| 157 | + # run main function |
| 158 | + main() |
| 159 | +``` |
| 160 | + |
| 161 | +Use the [Azure Machine Learning CLI (v2)](how-to-train-cli.md) to submit a remote run. When using the Azure Machine Learning CLI (v2), the MLflow tracking URI and experiment name are set automatically and directs the logging from MLflow to your workspace. Learn more about [logging Azure Machine Learning CLI (v2) experiments with MLflow](how-to-train-cli.md#model-tracking-with-mlflow) |
| 162 | + |
| 163 | +Create a YAML file with your job definition in a `job.yml` file. This file should be created outside the `src` directory. Copy this code into the file: |
| 164 | + |
| 165 | +```YAML |
| 166 | +$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json |
| 167 | +experiment_name: experiment_with_mlflow |
| 168 | +command: >- |
| 169 | + pip install mlflow azureml-mlflow |
| 170 | + && |
| 171 | + python train.py |
| 172 | +code: |
| 173 | + local_path: src |
| 174 | +environment: |
| 175 | + image: python:3.8 |
| 176 | +compute: azureml:MyCluster |
| 177 | +``` |
| 178 | +
|
| 179 | +Open your terminal and use the following to submit the job. |
| 180 | +
|
| 181 | +```Azure CLI |
| 182 | +az ml job create -f job.yml --web |
| 183 | +``` |
| 184 | + |
| 185 | +## View metrics and artifacts in your workspace |
| 186 | + |
| 187 | +The metrics and artifacts from MLflow logging are tracked in your workspace. To view them anytime, navigate to your workspace and find the experiment by name in your workspace in [Azure Machine Learning studio](https://ml.azure.com). Or run the below code. |
| 188 | + |
| 189 | +Retrieve run metric using MLflow [get_run()](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.get_run). |
| 190 | + |
| 191 | +```Python |
| 192 | +from mlflow.tracking import MlflowClient |
| 193 | + |
| 194 | +# Use MlFlow to retrieve the run that was just completed |
| 195 | +client = MlflowClient() |
| 196 | +run_id = mlflow_run.info.run_id |
| 197 | +finished_mlflow_run = MlflowClient().get_run(run_id) |
| 198 | + |
| 199 | +metrics = finished_mlflow_run.data.metrics |
| 200 | +tags = finished_mlflow_run.data.tags |
| 201 | +params = finished_mlflow_run.data.params |
| 202 | + |
| 203 | +print(metrics,tags,params) |
| 204 | +``` |
| 205 | + |
| 206 | +### Retrieve artifacts with MLFLow |
| 207 | + |
| 208 | +To view the artifacts of a run, you can use [MlFlowClient.list_artifacts()](https://mlflow.org/docs/latest/python_api/mlflow.tracking.html#mlflow.tracking.MlflowClient.list_artifacts) |
| 209 | + |
| 210 | +```Python |
| 211 | +client.list_artifacts(run_id) |
| 212 | +``` |
| 213 | + |
| 214 | +To download an artifact to the current directory, you can use [MLFlowClient.download_artifacts()](https://www.mlflow.org/docs/latest/python_api/mlflow.tracking.html#mlflow.tracking.MlflowClient.download_artifacts) |
| 215 | + |
| 216 | +```Python |
| 217 | +client.download_artifacts(run_id, "helloworld.txt", ".") |
| 218 | +``` |
| 219 | + |
| 220 | +### Compare and query |
| 221 | + |
| 222 | +Compare and query all MLflow runs in your Azure Machine Learning workspace with the following code. |
| 223 | +[Learn more about how to query runs with MLflow](https://mlflow.org/docs/latest/search-syntax.html#programmatically-searching-runs). |
| 224 | + |
| 225 | +```Python |
| 226 | +from mlflow.entities import ViewType |
| 227 | + |
| 228 | +all_experiments = [exp.experiment_id for exp in MlflowClient().list_experiments()] |
| 229 | +query = "metrics.hello_metric > 0" |
| 230 | +runs = mlflow.search_runs(experiment_ids=all_experiments, filter_string=query, run_view_type=ViewType.ALL) |
| 231 | + |
| 232 | +runs.head(10) |
| 233 | +``` |
| 234 | + |
| 235 | +## Manage models |
| 236 | + |
| 237 | +Register and track your models with the [Azure Machine Learning model registry](concept-model-management-and-deployment.md#register-package-and-deploy-models-from-anywhere), which supports the MLflow model registry. Azure Machine Learning models are aligned with the MLflow model schema making it easy to export and import these models across different workflows. The MLflow-related metadata, such as run ID, is also tracked with the registered model for traceability. Users can submit training runs, register, and deploy models produced from MLflow runs. |
| 238 | + |
| 239 | +If you want to deploy and register your production ready model in one step, see [Deploy and register MLflow models](how-to-deploy-mlflow-models.md). |
| 240 | + |
| 241 | +To register and view a model from a run, use the following steps: |
| 242 | + |
| 243 | +1. Once a run is complete, call the [`register_model()`](https://mlflow.org/docs/latest/python_api/mlflow.html#mlflow.register_model) method. |
| 244 | + |
| 245 | + ```Python |
| 246 | + # the model folder produced from a run is registered. This includes the MLmodel file, model.pkl and the conda.yaml. |
| 247 | + model_path = "model" |
| 248 | + model_uri = 'runs:/{}/{}'.format(run_id, model_path) |
| 249 | + mlflow.register_model(model_uri,"registered_model_name") |
| 250 | + ``` |
| 251 | + |
| 252 | +1. View the registered model in your workspace with [Azure Machine Learning studio](overview-what-is-machine-learning-studio.md). |
| 253 | + |
| 254 | + In the following example the registered model, `my-model` has MLflow tracking metadata tagged. |
| 255 | + |
| 256 | +  |
| 257 | + |
| 258 | +1. Select the **Artifacts** tab to see all the model files that align with the MLflow model schema (conda.yaml, MLmodel, model.pkl). |
| 259 | + |
| 260 | +  |
| 261 | + |
| 262 | +1. Select MLmodel to see the MLmodel file generated by the run. |
| 263 | + |
| 264 | +  |
| 265 | + |
| 266 | + |
| 267 | +## Example notebooks |
| 268 | + |
| 269 | +[Use MLflow and CLI (v2)](https://github.com/Azure/azureml-examples/blob/main/cli/jobs/basics/hello-mlflow.yml) |
| 270 | + |
| 271 | +## Next steps |
| 272 | + |
| 273 | +* [Deploy MLflow models to managed online endpoint (preview)](how-to-deploy-mlflow-models-online-endpoints.md). |
| 274 | +* [Manage your models](concept-model-management-and-deployment.md). |
0 commit comments