You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/connector-microsoft-fabric-lakehouse.md
+40-40Lines changed: 40 additions & 40 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -42,7 +42,7 @@ This Microsoft Fabric Lakehouse connector is supported for the following capabil
42
42
43
43
Use the following steps to create a Microsoft Fabric Lakehouse linked service in the Azure portal UI.
44
44
45
-
1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then select New:
45
+
1. Browse to the **Manage** tab in your Azure Data Factory or Synapse workspace and select Linked Services, then select New:
46
46
47
47
# [Azure Data Factory](#tab/data-factory)
48
48
@@ -206,18 +206,18 @@ The following properties are supported for Microsoft Fabric Lakehouse Table data
206
206
207
207
```json
208
208
{
209
-
"name":"LakehouseTableDataset",
210
-
"properties":{
211
-
"type":"LakehouseTable",
212
-
"linkedServiceName":{
213
-
"referenceName":"<Microsoft Fabric Lakehouse linked service name>",
214
-
"type":"LinkedServiceReference"
215
-
},
216
-
"typeProperties":{
209
+
"name":"LakehouseTableDataset",
210
+
"properties":{
211
+
"type":"LakehouseTable",
212
+
"linkedServiceName":{
213
+
"referenceName":"<Microsoft Fabric Lakehouse linked service name>",
214
+
"type":"LinkedServiceReference"
215
+
},
216
+
"typeProperties":{
217
217
"table": "<table_name>"
218
-
},
219
-
"schema":[< physical schema, optional, retrievable during authoring >]
220
-
}
218
+
},
219
+
"schema":[< physical schema, optional, retrievable during authoring >]
220
+
}
221
221
}
222
222
```
223
223
@@ -544,39 +544,39 @@ The following properties are supported in the Mapping Data Flows **sink** sectio
544
544
| Update method | When you select "Allow insert" alone or when you write to a new delta table, the target receives all incoming rows regardless of the Row policies set. If your data contains rows of other Row policies, they need to be excluded using a preceding Filter transform. <br><br> When all Update methods are selected a Merge is performed, where rows are inserted/deleted/upserted/updated as per the Row Policies set using a preceding Alter Row transform. | yes |`true` or `false`| insertable <br> deletable <br> upsertable <br> updateable |
545
545
| Optimized Write | Achieve higher throughput for write operation via optimizing internal shuffle in Spark executors. As a result, you might notice fewer partitions and files that are of a larger size | no |`true` or `false`| optimizedWrite: true |
546
546
| Auto Compact | After any write operation has completed, Spark will automatically execute the ```OPTIMIZE``` command to reorganize the data, resulting in more partitions if necessary, for better reading performance in the future | no |`true` or `false`| autoCompact: true |
547
-
| Merge Schema | Merge schemaoption allows schema evolution, that is, any columns that are present in the current incoming stream but not in the target Delta table is automatically added to its schema. This option is supported across all update methods. | no |`true` or `false`| mergeSchema: true |
547
+
| Merge Schema | Merge schemaoption allows schema evolution, that is, any columns that are present in the current incoming stream but not in the target Delta table is automatically added to its schema. This option is supported across all update methods. | no |`true` or `false`| mergeSchema: true |
548
548
549
549
**Example: Microsoft Fabric Lakehouse Table sink**
You can use the Delete Activity in Azure Data Factory to delete files or folders from on-premises storage stores or cloud storage stores. Use this activity to clean up or archive files when they are no longer needed.
18
+
You can use the Delete Activity in Azure Data Factory to delete files or folders from on-premises storage stores or cloud storage stores. Use this activity to clean up or archive files when they're no longer needed.
19
19
20
20
> [!WARNING]
21
21
> Deleted files or folders cannot be restored (unless the storage has soft-delete enabled). Be cautious when using the Delete activity to delete files or folders.
@@ -28,32 +28,32 @@ Here are some recommendations for using the Delete activity:
28
28
29
29
- Make sure that the service has write permissions to delete folders or files from the storage store.
30
30
31
-
- Make sure you are not deleting files that are being written at the same time.
31
+
- Make sure you aren't deleting files that are being written at the same time.
32
32
33
-
- If you want to delete files or folder from an on-premises system, make sure you are using a self-hosted integration runtime with a version greater than 3.14.
33
+
- If you want to delete files or folder from an on-premises system, make sure you're using a self-hosted integration runtime with a version greater than 3.14.
To use a Delete activity in a pipeline, complete the following steps:
54
54
55
55
1. Search for _Delete_ in the pipeline Activities pane, and drag a Delete activity to the pipeline canvas.
56
-
1. Select the new Delete activity on the canvas if it is not already selected, and its **Source** tab, to edit its details.
56
+
1. Select the new Delete activity on the canvas if it isn't already selected, and its **Source** tab, to edit its details.
57
57
58
58
:::image type="content" source="media/delete-activity/delete-activity.png" alt-text="Shows the UI for a Delete activity.":::
59
59
@@ -98,10 +98,10 @@ To use a Delete activity in a pipeline, complete the following steps:
98
98
| dataset | Provides the dataset reference to determine which files or folder to be deleted | Yes |
99
99
| recursive | Indicates whether the files are deleted recursively from the subfolders or only from the specified folder. | No. The default is `false`. |
100
100
| maxConcurrentConnections | The number of the connections to connect to storage store concurrently for deleting folder or files. | No. The default is `1`. |
101
-
|enablelogging | Indicates whether you need to record the folder or file names that have been deleted. If true, you need to further provide a storage account to save the log file, so that you can track the behaviors of the Delete activity by reading the log file. | No |
102
-
| logStorageSettings | Only applicable when enablelogging = true.<br/><br/>A group of storage properties that can be specified where you want to save the log file containing the folder or file names that have been deleted by the Delete activity. | No |
103
-
| linkedServiceName | Only applicable when enablelogging = true.<br/><br/>The linked service of [Azure Storage](connector-azure-blob-storage.md#linked-service-properties), [Azure Data Lake Storage Gen1](connector-azure-data-lake-store.md#linked-service-properties), or [Azure Data Lake Storage Gen2](connector-azure-data-lake-storage.md#linked-service-properties) to store the log file that contains the folder or file names that have been deleted by the Delete activity. Be aware it must be configured with the same type of Integration Runtime from the one used by delete activity to delete files. | No |
104
-
| path | Only applicable when enablelogging = true.<br/><br/>The path to save the log file in your storage account. If you do not provide a path, the service creates a container for you. | No |
101
+
|enable logging | Indicates whether you need to record the deleted folder or file names. If true, you need to further provide a storage account to save the log file, so that you can track the behaviors of the Delete activity by reading the log file. | No |
102
+
| logStorageSettings | Only applicable when enablelogging = true.<br/><br/>A group of storage properties that can be specified where you want to save the log file containing the folder or file names deleted by the Delete activity. | No |
103
+
| linkedServiceName | Only applicable when enablelogging = true.<br/><br/>The linked service of [Azure Storage](connector-azure-blob-storage.md#linked-service-properties), [Azure Data Lake Storage Gen1](connector-azure-data-lake-store.md#linked-service-properties), or [Azure Data Lake Storage Gen2](connector-azure-data-lake-storage.md#linked-service-properties) to store the log file that contains the folder or file names deleted by the Delete activity. Be aware it must be configured with the same type of Integration Runtime from the one used by delete activity to delete files. | No |
104
+
| path | Only applicable when enablelogging = true.<br/><br/>The path to save the log file in your storage account. If you don't provide a path, the service creates a container for you. | No |
105
105
106
106
## Monitoring
107
107
@@ -144,7 +144,7 @@ The store has the following folder structure:
Now you are using the Delete activity to delete folder or files by the combination of different property value from the dataset and the Delete activity:
147
+
Now you're using the Delete activity to delete folder or files by the combination of different property value from the dataset and the Delete activity:
148
148
149
149
| folderPath | fileName | recursive | Output |
150
150
|:--- |:--- |:--- |:--- |
@@ -155,7 +155,7 @@ Now you are using the Delete activity to delete folder or files by the combinati
155
155
156
156
### Periodically clean up the time-partitioned folder or files
157
157
158
-
You can create a pipeline to periodically clean up the time partitioned folder or files. For example, the folder structure is similar as: `/mycontainer/2018/12/14/*.csv`. You can leverage the service system variable from schedule trigger to identify which folder or files should be deleted in each pipeline run.
158
+
You can create a pipeline to periodically clean up the time partitioned folder or files. For example, the folder structure is similar as: `/mycontainer/2018/12/14/*.csv`. You can use the service system variable from schedule trigger to identify which folder or files should be deleted in each pipeline run.
159
159
160
160
#### Sample pipeline
161
161
@@ -295,7 +295,7 @@ You can create a pipeline to periodically clean up the time partitioned folder o
295
295
296
296
### Clean up the expired files that were last modified before 2018.1.1
297
297
298
-
You can create a pipeline to clean up the old or expired files by leveraging file attribute filter: “LastModified” in dataset.
298
+
You can create a pipeline to clean up the old or expired files by using file attribute filter: "LastModified" in dataset.
299
299
300
300
#### Sample pipeline
301
301
@@ -376,7 +376,7 @@ You can create a pipeline to clean up the old or expired files by leveraging fil
376
376
377
377
### Move files by chaining the Copy activity and the Delete activity
378
378
379
-
You can move a file by using a Copy activity to copy a file and then a Delete activity to delete a file in a pipeline. When you want to move multiple files, you can use the GetMetadata activity + Filter activity + Foreach activity + Copy activity + Delete activity as in the following sample.
379
+
You can move a file by using a Copy activity to copy a file and then a Delete activity to delete a file in a pipeline. When you want to move multiple files, you can use the GetMetadata activity + Filter activity + Foreach activity + Copy activity + Delete activity as in the following sample.
380
380
381
381
> [!NOTE]
382
382
> If you want to move the entire folder by defining a dataset containing a folder path only, and then using a Copy activity and a Delete activity to reference to the same dataset representing a folder, you need to be very careful. You must ensure that there **will not** be any new files arriving into the folder between the copy operation and the delete operation. If new files arrive in the folder at the moment when your copy activity just completed the copy job but the Delete activity has not been started, then the Delete activity might delete the newly arriving file which has NOT been copied to the destination yet by deleting the entire folder.
@@ -772,12 +772,12 @@ You can also get the template to move files from [here](solution-template-move-f
772
772
773
773
## Known limitation
774
774
775
-
-Delete activity does not support deleting list of folders described by wildcard.
775
+
- Delete activity doesn't support deleting list of folders described by wildcard.
776
776
777
-
-When using file attribute filter in delete activity: modifiedDatetimeStart and modifiedDatetimeEnd to select files to be deleted, make sure to set "wildcardFileName": "*" in delete activity as well.
777
+
- When using file attribute filter in delete activity: modifiedDatetimeStart and modifiedDatetimeEnd to select files to be deleted, make sure to set "wildcardFileName": "*" in delete activity as well.
778
778
779
779
## Related content
780
780
781
781
Learn more about moving files in Azure Data Factory and Synapse pipelines.
0 commit comments