Skip to content

Commit e5f0a69

Browse files
authored
Deleting intermediate data update
1 parent b648eef commit e5f0a69

File tree

1 file changed

+8
-3
lines changed

1 file changed

+8
-3
lines changed

articles/machine-learning/v1/how-to-move-data-in-out-of-pipelines.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -235,9 +235,14 @@ step1_output_ds = step1_output_data.register_on_complete(name='processed_data',
235235

236236
Azure does not automatically delete intermediate data written with `OutputFileDatasetConfig`. To avoid storage charges for large amounts of unneeded data, you should either:
237237

238-
* Programmatically delete intermediate data at the end of a pipeline job, when it is no longer needed
239-
* Use blob storage with a short-term storage policy for intermediate data (see [Optimize costs by automating Azure Blob Storage access tiers](/azure/storage/blobs/lifecycle-management-overview))
240-
* Regularly review and delete no-longer-needed data
238+
* Programmatically delete intermediate data at the end of a pipeline job, when it is no longer needed. Data should be deleted after a 30 day period, deleting the data earlier could cause the pipeline to fail.
239+
* Use blob storage with a short-term storage policy for intermediate data (see [Optimize costs by automating Azure Blob Storage access tiers](/azure/storage/blobs/lifecycle-management-overview)). This policy can only be set to a workspace's non-default datastore. Use `OutputFileDatasetConfig` to export intermediate data to another datastore that isn't the default.
240+
```Python
241+
# Get adls gen 2 datastore already registered with the workspace
242+
datastore = workspace.datastores['my_adlsgen2']
243+
step1_output_data = OutputFileDatasetConfig(name="processed_data", destination=(datastore, "mypath/{run-id}/{output-name}")).as_upload()
244+
```
245+
* Regularly review and delete no-longer-needed data.
241246

242247
For more information, see [Plan and manage costs for Azure Machine Learning](../concept-plan-manage-cost.md).
243248

0 commit comments

Comments
 (0)