Merge pull request #246395 from Blackmist/expressions

adding info about expressions

Author: v-regandowner

articles/machine-learning/concept-expressions.md

@@ -0,0 +1,50 @@
+---
+title: 'SDK and CLI v2 expressions'
+titleSuffix: Azure Machine Learning
+description: SDK and CLI v2 use expressions when a value may not be known when authoring a job or component.
+services: machine-learning
+ms.service: machine-learning
+ms.subservice: core
+ms.topic: conceptual
+ms.author: zhanxia
+author: xiaoharper
+ms.reviewer: larryfr
+ms.date: 07/26/2023
+ms.custom: cliv2, sdkv2
+---
+
+# Expressions in Azure Machine Learning SDK and CLI v2
+
+With Azure Machine Learning SDK and CLI v2, you can use _expressions_ when a value may not be known when you're authoring a job or component. When you submit a job or call a component, the expression is evaluated and the value is substituted.
+
+The format for an expression is `${{ <expression> }}`. Some expressions are evaluated on the _client_, when submitting the job or component. Other expressions are evaluated on the _server_ (the compute where the job or component is running.)
+
+## Client expressions
+
+> [!NOTE]
+> The "client" that evaluates the expression is where the job is submitted or component is ran. For example, your local machine or a compute instance.
+
+| Expression | Description | Scope |
+| ---- | ---- | ---- |
+| `${{inputs.<input_name>}}` | References to an input data asset or model. | Works for all jobs. |
+| `${{outputs.<output_name>}}` | References to an output data asset or model. | Works for all jobs. |
+| `${{search_space.<hyperparameter>}}` | References the hyperparameters to use in a sweep job. The hyperparameter values for each trial are selected based on the `search_space`. | Sweep jobs only. |
+| `${{parent.inputs.<input_name>}}` | Binds the inputs of a child job (pipeline step) in a pipeline to the inputs of the top-level parent pipeline job. | Pipeline jobs only. |
+| `${{parent.outputs.<output_name>}}` | Binds the outputs of a child job (pipeline step) in a pipeline to the outputs of the top-level parent pipeline job. | Pipeline jobs only. |
+| `${{parent.jobs.<step-name>.inputs.<input-name>}}` | Binds to the inputs of another step in the pipeline. | Pipeline jobs only. |
+| `${{parent.jobs.<step-name>.outputs.<output-name>}}` | Binds to the outputs of another step in the pipeline. | Pipeline jobs only. |
+
+## Server expressions
+
+[!INCLUDE [output-path-expressions](includes/output-path-expressions.md)]
+
+## Next steps
+
+For more information on these expressions, see the following articles and examples:
+
+* [CLI v2 core YAML syntax](reference-yaml-core-syntax.md#expression-syntax-for-configuring-azure-machine-learning-jobs-and-components)
+* [Hyperparameter tuning a model](how-to-tune-hyperparameters.md)
+* [Tutorial: ML pipelines with Python SDK v2](tutorial-pipeline-python-sdk.md)
+* [Create and run component-based ML pipelines (CLI)](how-to-create-component-pipelines-cli.md)
+* [Example: Iris batch prediction notebook](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/parallel/2a_iris_batch_prediction/iris_batch_prediction.ipynb)
+* [Example: Pipeline YAML file](https://github.com/Azure/azureml-examples/blob/main/cli/jobs/pipelines-with-components/nyc_taxi_data_regression/pipeline.yml)

articles/machine-learning/concept-ml-pipelines.md

@@ -78,6 +78,7 @@ Azure Machine Learning pipelines are a powerful facility that begins delivering
 + [Define pipelines with Designer](./how-to-create-component-pipelines-ui.md)
 + Try out [CLI v2 pipeline example](https://github.com/Azure/azureml-examples/tree/sdk-preview/cli/jobs/pipelines-with-components)
 + Try out [Python SDK v2 pipeline example](https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/pipelines)
++ Learn about [SDK and CLI v2 expressions](concept-expressions.md) that can be used in a pipeline.
 :::moniker-end
 :::moniker range="azureml-api-1"
 + [Create and run machine learning pipelines](v1/how-to-create-machine-learning-pipelines.md)

articles/machine-learning/how-to-train-model.md

@@ -180,7 +180,7 @@ curl -X PUT \
 
 # [Python SDK](#tab/python)
 
-To run this script, you'll use a `command` that executes main.py Python script located under ./sdk/python/jobs/single-step/lightgbm/iris/src/. The command will be run by submitting it as a `job` to Azure ML. 
+To run this script, you'll use a `command` that executes main.py Python script located under ./sdk/python/jobs/single-step/lightgbm/iris/src/. The command will be run by submitting it as a `job` to Azure Machine Learning. 
 
 > [!NOTE]
 > To use [serverless compute (preview)](./how-to-use-serverless-compute.md), delete `compute="cpu-cluster"` in this code.
@@ -193,7 +193,7 @@ In the above examples, you configured:
 - `code` - path where the code to run the command is located
 - `command` -  command that needs to be run
 - `environment` - the environment needed to run the training script. In this example, we use a curated or ready-made environment provided by Azure Machine Learning called `AzureML-lightgbm-3.2-ubuntu18.04-py37-cpu`. We use the latest version of this environment by using the `@latest` directive. You can also use custom environments by specifying a base docker image and specifying a conda yaml on top of it.
-- `inputs` - dictionary of inputs using name value pairs to the command. The key is a name for the input within the context of the job and the value is the input value. Inputs are referenced in the `command` using the `${{inputs.<input_name>}}` expression. To use files or folders as inputs, you can use the `Input` class.
+- `inputs` - dictionary of inputs using name value pairs to the command. The key is a name for the input within the context of the job and the value is the input value. Inputs are referenced in the `command` using the `${{inputs.<input_name>}}` expression. To use files or folders as inputs, you can use the `Input` class. For more information, see [SDK and CLI v2 expressions](concept-expressions.md).
 
 For more information, see the [reference documentation](/python/api/azure-ai-ml/azure.ai.ml#azure-ai-ml-command).
 
@@ -212,7 +212,7 @@ The `az ml job create` command used in this example requires a YAML job definiti
 In the above, you configured:
 - `code` - path where the code to run the command is located
 - `command` - command that needs to be run
-- `inputs` - dictionary of inputs using name value pairs to the command. The key is a name for the input within the context of the job and the value is the input value. Inputs are referenced in the `command` using the `${{inputs.<input_name>}}` expression. 
+- `inputs` - dictionary of inputs using name value pairs to the command. The key is a name for the input within the context of the job and the value is the input value. Inputs are referenced in the `command` using the `${{inputs.<input_name>}}` expression. For more information, see [SDK and CLI v2 expressions](concept-expressions.md).
 - `environment` - the environment needed to run the training script. In this example, we use a curated or ready-made environment provided by Azure Machine Learning called `AzureML-sklearn-0.24-ubuntu18.04-py37-cpu`. We use the latest version of this environment by using the `@latest` directive. You can also use custom environments by specifying a base docker image and specifying a conda yaml on top of it.
 To submit the job, use the following command. The run ID (name) of the training job is stored in the `$run_id` variable:
 

articles/machine-learning/includes/output-path-expressions.md

@@ -0,0 +1,22 @@
+---
+title: "include file"
+description: "include file"
+services: machine-learning
+author: blackmist
+ms.service: machine-learning
+ms.author: larryfr
+ms.custom: "include file"
+ms.topic: "include"
+ms.date: 07/26/2023
+---
+
+> [!IMPORTANT]
+> The following expressions are resolved on the _server_ side, not the _client_ side. For scheduled jobs where the job _creation time_ and job _submission time_ are different, the expressions are resolved when the job is submitted. Since these expressions are resolved on the server side, they use the _current_ state of the workspace, not the state of the workspace when the scheduled job was created. For example, if you change the default datastore of the workspace after you create a scheduled job, the expression `${{default_datastore}}` is resolved to the new default datastore, not the default datastore when the scheduled job was created.
+
+| Expression | Description | Scope |
+| --- | --- | --- |
+| `${{default_datastore}}` | If pipeline default datastore is configured, is resolved as pipeline default datastore name; otherwise is resolved as workspace default datastore name. <br><br> Pipeline default datastore can be controlled using `pipeline_job.settings.default_datastore`. | Works for all jobs. <br><br> Pipeline jobs have a configurable pipeline default datastore. | 
+| `${{name}}` | The job name. For pipelines, it's the step job name, not the pipeline job name. | Works for all jobs |
+| `${{output_name}}` | The job output name | Works for all jobs |
+
+For example, if `azureml://datastores/${{default_datastore}}/paths/{{$name}}/${{output_name}}` is used as the output path, at runtime it's resolved as a path of `azureml://datastores/workspaceblobstore/paths/<job-name>/model_path`.

articles/machine-learning/reference-yaml-core-syntax.md

@@ -213,7 +213,7 @@ Similar to the `command` for a job, the `command` for a component can also be pa
 
 #### Define optional inputs in command line
 When the input is set as `optional = true`, you need use `$[[]]` to embrace the command line with inputs. For example `$[[--input1 ${{inputs.input1}}]`. The command line at runtime may have different inputs.
-- If you are using only specify the required `training_data` and `model_output` parameters, the command line will look like:
+- If you are using only the required `training_data` and `model_output` parameters, the command line will look like:
 
 ```cli
 python train.py --training_data some_input_path --learning_rate 0.01 --learning_rate_schedule time-based --model_output some_output_path
@@ -226,6 +226,12 @@ If no value is specified at runtime, `learning_rate` and `learning_rate_schedule
 python train.py --training_data some_input_path --max_epocs 10 --learning_rate 0.01 --learning_rate_schedule time-based --model_output some_output_path
 ```
 
+### Output path expressions
+
+The following expressions can be used in the output path of your job:
+
+[!INCLUDE [output path expressions](includes/output-path-expressions.md)]
+
 ## Next steps
 
 * [Install and use the CLI (v2)](how-to-configure-cli.md)

articles/machine-learning/toc.yml

@@ -516,6 +516,8 @@
     href: how-to-train-model.md
   - name: Training with UI
     href: how-to-train-with-ui.md
+  - name: CLI and Python SDK v2 expressions
+    href: concept-expressions.md
   - name: Using secrets in training 
     href: how-to-use-secrets-in-runs.md
   - name: Custom Training

articles/machine-learning/tutorial-pipeline-python-sdk.md

@@ -326,7 +326,7 @@ if __name__ == "__main__":
 
 Now that you have a script that can perform the desired task, create an Azure Machine Learning Component from it.
 
-Use the general purpose `CommandComponent` that can run command line actions. This command line action can directly call system commands or run a script. The inputs/outputs are specified on the command line via the `${{ ... }}` notation.
+Use the general purpose `CommandComponent` that can run command line actions. This command line action can directly call system commands or run a script. The inputs/outputs are specified on the command line via the `${{ ... }}` (expression) notation. For more information, see [SDK and CLI v2 expressions](concept-expressions.md).