Skip to content

Commit 2e622d9

Browse files
authored
Merge pull request #192538 from kromerm/MDFupdates
MDF updates
2 parents a96bdab + a07d0a7 commit 2e622d9

File tree

3 files changed

+9
-3
lines changed

3 files changed

+9
-3
lines changed

articles/data-factory/control-flow-execute-data-flow-activity.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ ms.subservice: data-flows
88
ms.custom: synapse
99
ms.topic: conceptual
1010
ms.author: makromer
11-
ms.date: 09/09/2021
11+
ms.date: 03/22/2022
1212
---
1313

1414
# Data Flow activity in Azure Data Factory and Azure Synapse Analytics
@@ -101,6 +101,10 @@ A minimum compute type of General Purpose with an 8+8 (16 total v-cores) configu
101101

102102
If you're using an Azure Synapse Analytics as a sink or source, you must choose a staging location for your PolyBase batch load. PolyBase allows for batch loading in bulk instead of loading the data row-by-row. PolyBase drastically reduces the load time into Azure Synapse Analytics.
103103

104+
## Checkpoint key
105+
106+
When using the change capture option for data flow sources, ADF will maintain and manage the checkpoint for you automatically. The default checkpoint key is a hash of the data flow name and the pipeline name. If you are using a dynamic pattern for your source tables or folders, you may wish to override this hash and set your own checkpoint key value here.
107+
104108
## Logging level
105109

106110
If you do not require every pipeline execution of your data flow activities to fully log all verbose telemetry logs, you can optionally set your logging level to "Basic" or "None". When executing your data flows in "Verbose" mode (default), you are requesting the service to fully log activity at each individual partition level during your data transformation. This can be an expensive operation, so only enabling verbose when troubleshooting can improve your overall data flow and pipeline performance. "Basic" mode will only log transformation durations while "None" will only provide a summary of durations.

articles/data-factory/data-flow-exists.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ ms.service: data-factory
99
ms.subservice: data-flows
1010
ms.topic: conceptual
1111
ms.custom: synapse
12-
ms.date: 09/09/2021
12+
ms.date: 03/22/2022
1313
---
1414

1515
# Exists transformation in mapping data flow
@@ -43,6 +43,8 @@ To create a free-form expression that contains operators other than "and" and "e
4343

4444
:::image type="content" source="media/data-flow/exists1.png" alt-text="Exists custom settings":::
4545

46+
If you are building dynamic patterns in your data flows by using "late binding" of columns via schema drift, you can use the ```byName()``` expression function to use the exists transformation without hardcoding (i.e. early binding) the column names. Example: ```toString(byName('ProductNumber','source1')) == toString(byName('ProductNumber','source2'))```
47+
4648
## Broadcast optimization
4749

4850
:::image type="content" source="media/data-flow/broadcast.png" alt-text="Broadcast Join":::

articles/data-factory/data-flow-expression-functions.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ In Data Factory and Synapse pipelines, use the expression language of the mappin
7070
| [hasColumn](data-flow-expressions-usage.md#hasColumn) | Checks for a column value by name in the stream. You can pass an optional stream name as the second argument. Column names known at design time should be addressed just by their name. Computed inputs aren't supported but you can use parameter substitutions. |
7171
| [hasError](data-flow-expressions-usage.md#hasError) | Checks if the assert with provided ID is marked as error. |
7272
| [iif](data-flow-expressions-usage.md#iif) | Based on a condition applies one value or the other. If other is unspecified, it's considered NULL. Both the values must be compatible(numeric, string...). |
73-
| [iifNull](data-flow-expressions-usage.md#iifNull) | Checks if the first parameter is null. If not null, the first parameter is returned. If null, the second parameter is returned. If three parameters are specified, the behavior is the same as iif(isNull(value1), value2, value3) and the third parameter is returned if the first value isn't null. |
73+
| [iifNull](data-flow-expressions-usage.md#iifNull) | Given two or more inputs, returns the first not null item. This function is equivalent to coalesce. |
7474
| [initCap](data-flow-expressions-usage.md#initCap) | Converts the first letter of every word to uppercase. Words are identified as separated by whitespace. |
7575
| [instr](data-flow-expressions-usage.md#instr) | Finds the position(1 based) of the substring within a string. 0 is returned if not found. |
7676
| [isDelete](data-flow-expressions-usage.md#isDelete) | Checks if the row is marked for delete. For transformations taking more than one input stream you can pass the (1-based) index of the stream. The stream index should be either 1 or 2 and the default value is 1. |

0 commit comments

Comments
 (0)