You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -39,7 +42,7 @@ This Microsoft Fabric Lakehouse connector is supported for the following capabil
39
42
40
43
Use the following steps to create a Microsoft Fabric Lakehouse linked service in the Azure portal UI.
41
44
42
-
1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then select New:
45
+
1. Browse to the **Manage** tab in your Azure Data Factory or Synapse workspace and select Linked Services, then select New:
43
46
44
47
# [Azure Data Factory](#tab/data-factory)
45
48
@@ -203,18 +206,18 @@ The following properties are supported for Microsoft Fabric Lakehouse Table data
203
206
204
207
```json
205
208
{
206
-
"name":"LakehouseTableDataset",
207
-
"properties":{
208
-
"type":"LakehouseTable",
209
-
"linkedServiceName":{
210
-
"referenceName":"<Microsoft Fabric Lakehouse linked service name>",
211
-
"type":"LinkedServiceReference"
212
-
},
213
-
"typeProperties":{
209
+
"name":"LakehouseTableDataset",
210
+
"properties":{
211
+
"type":"LakehouseTable",
212
+
"linkedServiceName":{
213
+
"referenceName":"<Microsoft Fabric Lakehouse linked service name>",
214
+
"type":"LinkedServiceReference"
215
+
},
216
+
"typeProperties":{
214
217
"table": "<table_name>"
215
-
},
216
-
"schema":[< physical schema, optional, retrievable during authoring >]
217
-
}
218
+
},
219
+
"schema":[< physical schema, optional, retrievable during authoring >]
220
+
}
218
221
}
219
222
```
220
223
@@ -541,39 +544,39 @@ The following properties are supported in the Mapping Data Flows **sink** sectio
541
544
| Update method | When you select "Allow insert" alone or when you write to a new delta table, the target receives all incoming rows regardless of the Row policies set. If your data contains rows of other Row policies, they need to be excluded using a preceding Filter transform. <br><br> When all Update methods are selected a Merge is performed, where rows are inserted/deleted/upserted/updated as per the Row Policies set using a preceding Alter Row transform. | yes |`true` or `false`| insertable <br> deletable <br> upsertable <br> updateable |
542
545
| Optimized Write | Achieve higher throughput for write operation via optimizing internal shuffle in Spark executors. As a result, you might notice fewer partitions and files that are of a larger size | no |`true` or `false`| optimizedWrite: true |
543
546
| Auto Compact | After any write operation has completed, Spark will automatically execute the ```OPTIMIZE``` command to reorganize the data, resulting in more partitions if necessary, for better reading performance in the future | no |`true` or `false`| autoCompact: true |
544
-
| Merge Schema | Merge schemaoption allows schema evolution, that is, any columns that are present in the current incoming stream but not in the target Delta table is automatically added to its schema. This option is supported across all update methods. | no |`true` or `false`| mergeSchema: true |
547
+
| Merge Schema | Merge schemaoption allows schema evolution, that is, any columns that are present in the current incoming stream but not in the target Delta table is automatically added to its schema. This option is supported across all update methods. | no |`true` or `false`| mergeSchema: true |
545
548
546
549
**Example: Microsoft Fabric Lakehouse Table sink**
You can use the Delete Activity in Azure Data Factory to delete files or folders from on-premises storage stores or cloud storage stores. Use this activity to clean up or archive files when they are no longer needed.
18
+
You can use the Delete Activity in Azure Data Factory to delete files or folders from on-premises storage stores or cloud storage stores. Use this activity to clean up or archive files when they're no longer needed.
19
19
20
20
> [!WARNING]
21
21
> Deleted files or folders cannot be restored (unless the storage has soft-delete enabled). Be cautious when using the Delete activity to delete files or folders.
@@ -28,31 +28,32 @@ Here are some recommendations for using the Delete activity:
28
28
29
29
- Make sure that the service has write permissions to delete folders or files from the storage store.
30
30
31
-
- Make sure you are not deleting files that are being written at the same time.
31
+
- Make sure you aren't deleting files that are being written at the same time.
32
32
33
-
- If you want to delete files or folder from an on-premises system, make sure you are using a self-hosted integration runtime with a version greater than 3.14.
33
+
- If you want to delete files or folder from an on-premises system, make sure you're using a self-hosted integration runtime with a version greater than 3.14.
To use a Delete activity in a pipeline, complete the following steps:
53
54
54
55
1. Search for _Delete_ in the pipeline Activities pane, and drag a Delete activity to the pipeline canvas.
55
-
1. Select the new Delete activity on the canvas if it is not already selected, and its **Source** tab, to edit its details.
56
+
1. Select the new Delete activity on the canvas if it isn't already selected, and its **Source** tab, to edit its details.
56
57
57
58
:::image type="content" source="media/delete-activity/delete-activity.png" alt-text="Shows the UI for a Delete activity.":::
58
59
@@ -97,10 +98,10 @@ To use a Delete activity in a pipeline, complete the following steps:
97
98
| dataset | Provides the dataset reference to determine which files or folder to be deleted | Yes |
98
99
| recursive | Indicates whether the files are deleted recursively from the subfolders or only from the specified folder. | No. The default is `false`. |
99
100
| maxConcurrentConnections | The number of the connections to connect to storage store concurrently for deleting folder or files. | No. The default is `1`. |
100
-
|enablelogging | Indicates whether you need to record the folder or file names that have been deleted. If true, you need to further provide a storage account to save the log file, so that you can track the behaviors of the Delete activity by reading the log file. | No |
101
-
| logStorageSettings | Only applicable when enablelogging = true.<br/><br/>A group of storage properties that can be specified where you want to save the log file containing the folder or file names that have been deleted by the Delete activity. | No |
102
-
| linkedServiceName | Only applicable when enablelogging = true.<br/><br/>The linked service of [Azure Storage](connector-azure-blob-storage.md#linked-service-properties), [Azure Data Lake Storage Gen1](connector-azure-data-lake-store.md#linked-service-properties), or [Azure Data Lake Storage Gen2](connector-azure-data-lake-storage.md#linked-service-properties) to store the log file that contains the folder or file names that have been deleted by the Delete activity. Be aware it must be configured with the same type of Integration Runtime from the one used by delete activity to delete files. | No |
103
-
| path | Only applicable when enablelogging = true.<br/><br/>The path to save the log file in your storage account. If you do not provide a path, the service creates a container for you. | No |
101
+
|enable logging | Indicates whether you need to record the deleted folder or file names. If true, you need to further provide a storage account to save the log file, so that you can track the behaviors of the Delete activity by reading the log file. | No |
102
+
| logStorageSettings | Only applicable when enablelogging = true.<br/><br/>A group of storage properties that can be specified where you want to save the log file containing the folder or file names deleted by the Delete activity. | No |
103
+
| linkedServiceName | Only applicable when enablelogging = true.<br/><br/>The linked service of [Azure Storage](connector-azure-blob-storage.md#linked-service-properties), [Azure Data Lake Storage Gen1](connector-azure-data-lake-store.md#linked-service-properties), or [Azure Data Lake Storage Gen2](connector-azure-data-lake-storage.md#linked-service-properties) to store the log file that contains the folder or file names deleted by the Delete activity. Be aware it must be configured with the same type of Integration Runtime from the one used by delete activity to delete files. | No |
104
+
| path | Only applicable when enablelogging = true.<br/><br/>The path to save the log file in your storage account. If you don't provide a path, the service creates a container for you. | No |
104
105
105
106
## Monitoring
106
107
@@ -143,7 +144,7 @@ The store has the following folder structure:
Now you are using the Delete activity to delete folder or files by the combination of different property value from the dataset and the Delete activity:
147
+
Now you're using the Delete activity to delete folder or files by the combination of different property value from the dataset and the Delete activity:
147
148
148
149
| folderPath | fileName | recursive | Output |
149
150
|:--- |:--- |:--- |:--- |
@@ -154,7 +155,7 @@ Now you are using the Delete activity to delete folder or files by the combinati
154
155
155
156
### Periodically clean up the time-partitioned folder or files
156
157
157
-
You can create a pipeline to periodically clean up the time partitioned folder or files. For example, the folder structure is similar as: `/mycontainer/2018/12/14/*.csv`. You can leverage the service system variable from schedule trigger to identify which folder or files should be deleted in each pipeline run.
158
+
You can create a pipeline to periodically clean up the time partitioned folder or files. For example, the folder structure is similar as: `/mycontainer/2018/12/14/*.csv`. You can use the service system variable from schedule trigger to identify which folder or files should be deleted in each pipeline run.
158
159
159
160
#### Sample pipeline
160
161
@@ -294,7 +295,7 @@ You can create a pipeline to periodically clean up the time partitioned folder o
294
295
295
296
### Clean up the expired files that were last modified before 2018.1.1
296
297
297
-
You can create a pipeline to clean up the old or expired files by leveraging file attribute filter: “LastModified” in dataset.
298
+
You can create a pipeline to clean up the old or expired files by using file attribute filter: "LastModified" in dataset.
298
299
299
300
#### Sample pipeline
300
301
@@ -375,7 +376,7 @@ You can create a pipeline to clean up the old or expired files by leveraging fil
375
376
376
377
### Move files by chaining the Copy activity and the Delete activity
377
378
378
-
You can move a file by using a Copy activity to copy a file and then a Delete activity to delete a file in a pipeline. When you want to move multiple files, you can use the GetMetadata activity + Filter activity + Foreach activity + Copy activity + Delete activity as in the following sample.
379
+
You can move a file by using a Copy activity to copy a file and then a Delete activity to delete a file in a pipeline. When you want to move multiple files, you can use the GetMetadata activity + Filter activity + Foreach activity + Copy activity + Delete activity as in the following sample.
379
380
380
381
> [!NOTE]
381
382
> If you want to move the entire folder by defining a dataset containing a folder path only, and then using a Copy activity and a Delete activity to reference to the same dataset representing a folder, you need to be very careful. You must ensure that there **will not** be any new files arriving into the folder between the copy operation and the delete operation. If new files arrive in the folder at the moment when your copy activity just completed the copy job but the Delete activity has not been started, then the Delete activity might delete the newly arriving file which has NOT been copied to the destination yet by deleting the entire folder.
@@ -771,12 +772,12 @@ You can also get the template to move files from [here](solution-template-move-f
771
772
772
773
## Known limitation
773
774
774
-
-Delete activity does not support deleting list of folders described by wildcard.
775
+
- Delete activity doesn't support deleting list of folders described by wildcard.
775
776
776
-
-When using file attribute filter in delete activity: modifiedDatetimeStart and modifiedDatetimeEnd to select files to be deleted, make sure to set "wildcardFileName": "*" in delete activity as well.
777
+
- When using file attribute filter in delete activity: modifiedDatetimeStart and modifiedDatetimeEnd to select files to be deleted, make sure to set "wildcardFileName": "*" in delete activity as well.
777
778
778
779
## Related content
779
780
780
781
Learn more about moving files in Azure Data Factory and Synapse pipelines.
![prmerger-automator[bot]](https://avatars.githubusercontent.com/u/22479449?v=4&size=40){kind=avatar}
0 commit comments