Skip to content

Commit ce38b59

Browse files
authored
Merge pull request #103322 from djpmsft/docUpdates
updating new features
2 parents c0aca1a + eb3b552 commit ce38b59

File tree

5 files changed

+14
-2
lines changed

5 files changed

+14
-2
lines changed

articles/data-factory/data-flow-sink.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,12 @@ Similar to a Select transformation, in the **Mapping** tab of the sink, you can
4848

4949
When you turn off auto-mapping, you'll have the option to add either fixed column-based mappings or rule-based mappings. Rule-based mappings allow you to write expressions with pattern matching while fixed mapping will map logical and physical column names. For more information on rule-based mapping, see [column patterns in mapping data flow](concepts-data-flow-column-pattern.md#rule-based-mapping-in-select-and-sink).
5050

51+
## Custom sink ordering
52+
53+
By default, data is written to multiple sinks in a nondeterministic order. The execution engine will write data in parallel as the transformation logic is completed and the sink ordering may vary each run. To specify and exact sink ordering, enable **Custom sink ordering** in the general tab of the data flow. When enabled, sinks will be written sequentially in increasing order.
54+
55+
![Custom sink ordering](media/data-flow/custom-sink-ordering.png "Custom sink ordering")
56+
5157
## Data preview in sink
5258

5359
When fetching a data preview on a debug cluster, no data will be written to your sink. A snapshot of what the data looks like will be returned, but nothing will be written to your destination. To test writing data into your sink, run a pipeline debug from the pipeline canvas.

articles/data-factory/data-flow-source.md

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ Once you have added a source, configure via the **Source Settings** tab. Here yo
3939

4040
![Source settings tab](media/data-flow/source1.png "Source settings tab")
4141

42+
**Test connection:** Test whether or not data flow's spark service can successfully connect to the linked service used in your source dataset. Debug mode must be on for this feature to be enabled.
43+
4244
**Schema drift:** [Schema Drift](concepts-data-flow-schema-drift.md) is data factory's ability to natively handle flexible schemas in your data flows without needing to explicitly define column changes.
4345

4446
* Check the **Allow schema drift** box if the source columns will change often. This setting allows all incoming source fields to flow through the transformations to the sink.
@@ -64,13 +66,17 @@ Like schemas in datasets, the projection in a source defines the data columns, t
6466

6567
![Settings on the Projection tab](media/data-flow/source3.png "Projection")
6668

67-
If your text file has no defined schema, select **Detect data type** so that Data Factory will sample and infer the data types. Select **Define default format** to autodetect the default data formats.
69+
If your text file has no defined schema, select **Detect data type** so that Data Factory will sample and infer the data types. Select **Define default format** to autodetect the default data formats.
70+
71+
**Reset schema** resets the projection to what is defined in the referenced dataset.
6872

6973
You can modify the column data types in a down-stream derived-column transformation. Use a select transformation to modify the column names.
7074

7175
### Import schema
7276

73-
Datasets like Avro and CosmosDB that support complex data structures do not require schema definitions to exist in the dataset. Therefore, you will be able to click the **Import Schema** button on the **Projection** tab for these types of sources.
77+
The **Import Schema** button on the **Projection** tab allows you to use an active debug cluster to create a schema projection. Available in every source type, importing the schema here will override the projection defined in the dataset. The dataset object will not be changed.
78+
79+
This is useful in datasets like Avro and CosmosDB that support complex data structures do not require schema definitions to exist in the dataset.
7480

7581
## Optimize the source transformation
7682

180 KB
Loading
62.9 KB
Loading
128 KB
Loading

0 commit comments

Comments
 (0)