Skip to content

Commit c1ebad2

Browse files
authored
Merge pull request #113613 from djpmsft/docUpdates
updating parameters doc
2 parents a64d906 + 1d3ac45 commit c1ebad2

8 files changed

+46
-19
lines changed

articles/data-factory/control-flow-execute-data-flow-activity.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -93,9 +93,7 @@ If your data flow uses parameterized datasets, set the parameter values in the *
9393

9494
### Parameterized data flows
9595

96-
If your data flow is parameterized, set the dynamic values of the data flow parameters in the **Parameters** tab. You can use either the ADF pipeline expression language or the data flow expression language to assign dynamic or literal parameter values. For more information, see [Data Flow Parameters](parameters-data-flow.md). If you wish to include pipeline properties as part of your expression to pass into a data flow parameter, then choose pipeline expressions.
97-
98-
![Execute Data Flow Parameter Example](media/data-flow/parameter-example.png "Parameter Example")
96+
If your data flow is parameterized, set the dynamic values of the data flow parameters in the **Parameters** tab. You can use either the ADF pipeline expression language or the data flow expression language to assign dynamic or literal parameter values. For more information, see [Data Flow Parameters](parameters-data-flow.md).
9997

10098
### Parameterized compute properties.
10199

Binary file not shown.
112 KB
Loading
24.7 KB
Loading
56.2 KB
Loading
257 KB
Loading
51.1 KB
Loading

articles/data-factory/parameters-data-flow.md

Lines changed: 45 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,14 @@ ms.author: makromer
66
ms.reviewer: daperlov
77
ms.service: data-factory
88
ms.topic: conceptual
9-
ms.date: 01/07/2020
9+
ms.date: 05/01/2020
1010
---
1111

1212
# Parameterizing mapping data flows
1313

1414
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
1515

16-
Mapping data flows in Azure Data Factory support the use of parameters. You can define parameters inside of your data flow definition, which you can then use throughout your expressions. The parameter values can be set by the calling pipeline via the Execute Data Flow activity. You have three options for setting the values in the data flow activity expressions:
16+
Mapping data flows in Azure Data Factory support the use of parameters. Define parameters inside of your data flow definition and use them throughout your expressions. The parameter values are set by the calling pipeline via the Execute Data Flow activity. You have three options for setting the values in the data flow activity expressions:
1717

1818
* Use the pipeline control flow expression language to set a dynamic value
1919
* Use the data flow expression language to set a dynamic value
@@ -37,34 +37,63 @@ You can quickly add additional parameters by selecting **New parameter** and spe
3737

3838
![Data flow parameter expression](media/data-flow/new-parameter-expression.png "Data flow parameter expression")
3939

40-
### Passing in a column name as a parameter
40+
## Assign parameter values from a pipeline
4141

42-
A common pattern is to pass in a column name as a parameter value. To reference the column associated with the parameter, use the `byName()` function. Remember to cast the column to its appropriate type with a casting function such as `toString()`.
42+
Once you've created a data flow with parameters, you can execute it from a pipeline with the Execute Data Flow Activity. After you add the activity to your pipeline canvas, you will be presented with the available data flow parameters in the activity's **Parameters** tab.
4343

44-
For example, if you wanted to map a string column based upon a parameter `columnName`, you can add a derived column transformation equal to `toString(byName($columnName))`.
44+
When assigning parameter values, you can use either the [pipeline expression language](control-flow-expression-language-functions.md) or the [data flow expression language](data-flow-expression-functions.md) based on spark types. Each mapping data flow can have any combination of pipeline and data flow expression parameters.
4545

46-
![Passing in a column name as a parameter](media/data-flow/parameterize-column-name.png "Passing in a column name as a paramete")
46+
![Setting a Data Flow parameter](media/data-flow/parameter-assign.png "Setting a Data Flow parameter")
4747

48-
## Assign parameter values from a pipeline
48+
### Pipeline expression parameters
4949

50-
Once you've created your data flow with parameters, you can execute it from a pipeline with the Execute Data Flow Activity. After you add the activity to your pipeline canvas, you will be presented with the available data flow parameters in the activity's **Parameters** tab.
50+
Pipeline expression parameters allow you to reference system variables, functions, pipeline parameters, and variables similar to other pipeline activities. When you click **Pipeline expression**, a side-nav will open allowing you to enter an expression using the expression builder.
5151

52-
![Setting a Data Flow parameter](media/data-flow/parameter-assign.png "Setting a Data Flow parameter")
52+
![Setting a Data Flow parameter](media/data-flow/parameter-pipeline.png "Setting a Data Flow parameter")
53+
54+
When referenced, pipeline parameters are evaluated and then their value is used in the data flow expression language. The pipeline expression type doesn't need to match the data flow parameter type.
55+
56+
#### String literals vs expressions
57+
58+
When assigning a pipeline expression parameter of type string, by default quotes will be added and the value will be evaluated as a literal. To read the parameter value as a data flow expression, check the expression box next to the parameter.
59+
60+
![Setting a Data Flow parameter](media/data-flow/string-parameter.png "Setting a Data Flow parameter")
61+
62+
If data flow parameter `stringParam` references a pipeline parameter with value `upper(column1)`.
5363

54-
If your parameter data type is string, when you click the text box to set parameter values, you can choose to enter either a pipeline or a data flow expression. If you choose pipeline expression, you will be presented with the pipeline expression panel. Make sure to include pipeline functions inside string interpolation syntax using `'@{<expression>}'`, for example:
64+
- If expression is checked, `$stringParam` evaluates to the value of column1 all uppercase.
65+
- If expression is not checked (default behavior), `$stringParam` evaluates to `'upper(column1)'`
5566

56-
```'@{pipeline().RunId}'```
5767

58-
If your parameter is not of type string, you will always be presented with the Data Flow Expression Builder. Here, you can enter any expression or literal values that you wish that matches the data type of the parameter. Below are examples of data flow expression and a literal string from the expression builder:
68+
#### Pipeline parameter example
5969

60-
* ```toInteger(Role)```
61-
* ```'this is my static literal string'```
70+
Say you have an integer parameter `intParam` that is referencing a pipeline parameter of type String, `@pipeline.parameters.pipelineParam`.
6271

63-
Each mapping data flow can have any combination of pipeline and data flow expression parameters.
72+
![Setting a Data Flow parameter](media/data-flow/parameter-pipeline-2.png "Setting a Data Flow parameter")
6473

65-
![Data flow parameters sample](media/data-flow/parameter-example.png "Data flow parameters sample")
74+
`@pipeline.parameters.pipelineParam` is assigned a value of `abs(1)` at runtime.
6675

76+
![Setting a Data Flow parameter](media/data-flow/parameter-pipeline-4.png "Setting a Data Flow parameter")
77+
78+
When `$intParam` is referenced in an expression such as a derived column, it will evaluate `abs(1)` return `1`.
79+
80+
![Setting a Data Flow parameter](media/data-flow/parameter-pipeline-3.png "Setting a Data Flow parameter")
81+
82+
### Data flow expression parameters
83+
84+
Select **Data flow expression** will open up the data flow expression builder. You will be able to reference functions, other parameters and any defined schema column throughout your data flow. This expression will be evaluated as is when referenced.
85+
86+
> [!NOTE]
87+
> If you pass in an invalid expression or reference a schema column that doesn't exist in that transformation, the parameter will evaluate to null.
88+
89+
90+
### Passing in a column name as a parameter
91+
92+
A common pattern is to pass in a column name as a parameter value. If the column is defined in the data flow schema, you can reference it directly as a string expression. If the column isn't defined in the schema, use the `byName()` function. Remember to cast the column to its appropriate type with a casting function such as `toString()`.
93+
94+
For example, if you wanted to map a string column based upon a parameter `columnName`, you can add a derived column transformation equal to `toString(byName($columnName))`.
6795

96+
![Passing in a column name as a parameter](media/data-flow/parameterize-column-name.png "Passing in a column name as a parameter")
6897

6998
## Next steps
7099
* [Execute data flow activity](control-flow-execute-data-flow-activity.md)

0 commit comments

Comments
 (0)