You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/storage/common/storage-use-data-movement-library.md
+45-36Lines changed: 45 additions & 36 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -32,7 +32,7 @@ If you're migrating code from the older **Microsoft.Azure.Storage.DataMovement**
32
32
33
33
## Set up your environment
34
34
35
-
If you don't have an existing project, this section shows you how to set up a project to work with the Azure Blob Storage client library for .NET. The steps include package installation, adding `using` directives, and creating an authorized client object. For details, see [Get started with Azure Blob Storage and .NET](../blobs/storage-blob-dotnet-get-started.md).
35
+
If you don't have an existing project, this section shows you how to set up a project to work with the Azure Blob Storage client library for .NET. The steps include package installation, adding `using` directives, and creating an authorized client object.
You can optionally provide an instance of [TransferManagerOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions) to the constructor, which applies certain configuration options, including maximum concurrency, to all transfers started by the `TransferManager` object.
84
+
You can optionally provide an instance of [TransferManagerOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions) to the constructor, which applies certain configuration options to all transfers started by the `TransferManager` object. The following configuration options are available:
85
+
86
+
-[CheckpointStoreOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions.checkpointstoreoptions): Optional. Defines the options for creating a checkpoint used for saving transfer state so transfers can be resumed.
87
+
-[Diagnostics](/dotnet/api/azure.storage.datamovement.transfermanageroptions.diagnostics): Gets the transfer manager diagnostic options.
88
+
-[ErrorHandling](/dotnet/api/azure.storage.datamovement.transfermanageroptions.errorhandling): Optional. Defines how errors are handled during a transfer. Default is `StopOnAnyFailure`.
89
+
-[MaximumConcurrency](/dotnet/api/azure.storage.datamovement.transfermanageroptions.maximumconcurrency): The maximum number of workers that may be used in a parallel transfer.
90
+
-[ProvidersForResuming](/dotnet/api/azure.storage.datamovement.transfermanageroptions.providersforresuming): Resource providers for the transfer manager to use in resuming a transfer. Expects one provider for each storage provider in use. To learn more, see [Resume an existing transfer](#resume-an-existing-transfer).
// Use a similar approach to get a page blob or append blob resource
133
137
```
134
138
135
139
## Start a new transfer
136
140
137
-
Transfers are defined by a source and a destination. Both the source and destination are type `StorageResource`, which can be either `StorageResourceContainer` or `StorageResourceItem`. For a given transfer, the source and destination must be of the same kind. For example, if the source is a blob container, the destination must be a blob container.
141
+
All transfers need to specify a source and a destination. Both the source and destination are type `StorageResource`, which can be either `StorageResourceContainer` or `StorageResourceItem`. For a given transfer, the source and destination must be of the same kind. For example, if the source is a blob container, the destination must be a blob container.
138
142
139
143
You can start a new transfer by calling the following method:
This method returns a [TransferOperation](/dotnet/api/azure.storage.datamovement.transferoperation) object that represents the transfer. You can use the `TransferOperation` object to monitor the transfer progress or obtain the transfer ID. The transfer ID is a unique identifier for the transfer that's needed to [resume a transfer](#resume-an-existing-transfer) or pause a transfer.
144
148
145
-
You can optionally provide an instance of [TransferOptions](/dotnet/api/azure.storage.datamovement.transferoptions) to `StartTransferAsync`, which applies certain configuration options, including creation preference and transfer size, to a specific transfer.
149
+
You can optionally provide an instance of [TransferOptions](/dotnet/api/azure.storage.datamovement.transferoptions) to `StartTransferAsync` and `ResumeTransferAsync` methods, which applies certain configuration options to a specific transfer. The following configuration options are available:
146
150
147
-
### Example: Upload a local file to a blob
151
+
-[CreationMode](/dotnet/api/azure.storage.datamovement.transferoptions.creationmode): Configures the behavior when a transfer encounters a resource that already exists. Defaults to `FailIfExists` when starting a new transfer. When resuming a transfer, the defaults can vary. For all resources successfully enumerated when the transfer started, `CreationMode` defaults to the initial value used. For any remaining resources, the regular default value applies.
152
+
-[InitialTransferSize](/dotnet/api/azure.storage.datamovement.transferoptions.initialtransfersize): The size of the first range request in bytes. Single transfer sizes smaller than this limit are uploaded or downloaded in a single request. Transfers larger than this limit will continue being downloaded or uploaded in chunks of size [MaximumTransferChunkSize](/dotnet/api/azure.storage.datamovement.transferoptions.maximumtransferchunksize). The default value is 32 MiB. When resuming a transfer, the default value is the value specified when the transfer first started.
153
+
-[MaximumTransferChunkSize](/dotnet/api/azure.storage.datamovement.transferoptions.maximumtransferchunksize): The maximum size to use for each chunk when transferring data in chunks. The default value is 4 MiB. When resuming a transfer, the default value is the value specified when the transfer first started.
154
+
-[ProgressHandlerOptions](/dotnet/api/azure.storage.datamovement.transferoptions.progresshandleroptions): Optional. Options for changing behavior of the ProgressHandler.
148
155
149
-
The following code example shows how to start a new transfer to upload a local file to a blob:
156
+
### Example: Upload a local directory to a blob container
157
+
158
+
The following code example shows how to start a new transfer to upload a local directory to a blob container:
You can use the Data Movement library to copy between two `StorageResource` instances. For blob resources, the transfer uses the [Put Blob From URL](/rest/api/storageservices/put-blob-from-url) operation, which performs a server-to-server copy.
179
+
You can use the Data Movement library to copy between two `StorageResource` instances. For blob resources, the transfer uses the [Put Blob From URL](/rest/api/storageservices/put-blob-from-url) operation, which performs a server-to-server copy.
172
180
173
-
The following code example shows how to start a new transfer to copy a source blob container to a destination blob container:
181
+
The following code example shows how to start a new transfer to copy all blobs in a source blob container to a destination blob container. The destination container must already exist. In this example, we set [CreationMode](/dotnet/api/azure.storage.datamovement.storageresourcecreationmode) to `OverwriteIfExists` to overwrite any destination blobs that already exist. You can adjust the `CreationMode` property based on the needs of your app.
The following code example shows how to start a new transfer to copy a source blob to a destination blob:
211
+
The following code example shows how to start a new transfer to copy a source blob to a destination blob. In this example, we set [CreationMode](/dotnet/api/azure.storage.datamovement.storageresourcecreationmode) to `OverwriteIfExists` to overwrite the destination blob if it already exists. You can adjust the `CreationMode` property based on the needs of your app.
201
212
202
213
```csharp
203
214
UrisourceBlobUri=newUri(
@@ -207,31 +218,30 @@ Uri destinationBlobUri = new Uri(
You can set configuration options for a transfer by providing an instance of [TransferOptions](/dotnet/api/azure.storage.datamovement.transferoptions) to the `StartTransferAsync` method. The `TransferOptions` class provides properties that allow you to set the number of parallel operations, the size of the buffer, and more.
217
-
218
230
## Resume an existing transfer
219
231
220
-
By persisting transfer progress to disk, the Data Movement library allows you to resume a transfer that failed before completion, or was otherwise canceled or paused. To resume a transfer, the `TransferManager` object must be configured with `StorageResourceProvider` instances that are capable of reassembling the transfer from the persisted data. You can use the `ResumeProviders` property of the [TransferManagerOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions) class to specify the providers.
232
+
By persisting transfer progress to disk, the Data Movement library allows you to resume a transfer that failed before completion, or was otherwise canceled or paused. To resume a transfer, the `TransferManager` object must be configured with `StorageResourceProvider` instances that are capable of reassembling the transfer from the persisted data. You can use the `ProvidersForResuming` property of the [TransferManagerOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions) class to specify the providers.
221
233
222
234
The following code example shows how to initialize a `TransferManager` object that's capable of resuming a transfer between the local file system and Blob Storage:
@@ -259,7 +269,7 @@ Transfers can be monitored and observed through several mechanisms, depending on
259
269
260
270
You can monitor transfer progress using the `TransferOperation` object returned by the `StartTransferAsync` method. You can also call [TransferManager.GetTransfersAsync](/dotnet/api/azure.storage.datamovement.transfermanager.gettransfersasync) to enumerate all transfers for a `TransferManager` object.
261
271
262
-
The following code example shows how to iterate over all transfers and write the status of each transfer to a log file:
272
+
The following code example shows how to iterate over all transfers and write the state of each transfer to a log file:
The `TransferStatus` property returns a [TransferStatus](/dotnet/api/azure.storage.datamovement.transferstatus)value. `TransferStatus` includes the following properties:
285
+
[TransferStatus](/dotnet/api/azure.storage.datamovement.transferstatus)defines the status of the transfer job. `TransferStatus` includes the following properties:
276
286
277
287
| Property | Type | Description |
278
288
| --- | --- | --- |
279
289
|`HasCompletedSuccessfully`| Boolean | Represents if the transfer completes successfully without any failure or skipped items. |
280
290
|`HasFailedItems`| Boolean | Represents if transfer has any failure items. If set to `true`, the transfer has at least one failure item. If set to `false`, the transfer currently has no failures. |
281
-
|`HasSkippedItems`| Boolean | Represents if transfer has any skipped items. If set to `true`, the transfer has at least one skipped item. If set to `false`, the transfer currently has no skipped items. It's possible to never have any items skipped if `SkipIfExists` isn't enabled in [TransferOptions.CreationPreference](/dotnet/api/azure.storage.datamovement.transferoptions.creationpreference). |
291
+
|`HasSkippedItems`| Boolean | Represents if transfer has any skipped items. If set to `true`, the transfer has at least one skipped item. If set to `false`, the transfer currently has no skipped items. It's possible to never have any items skipped if `SkipIfExists` isn't enabled in [TransferOptions.CreationMode](/dotnet/api/azure.storage.datamovement.transferoptions.creationmode). |
282
292
|`State`|[TransferState](/dotnet/api/azure.storage.datamovement.transferstate)| Defines the types of the state a transfer can have. See [TransferState](/dotnet/api/azure.storage.datamovement.transferstate) for details. |
283
293
284
294
### Example: Monitor transfer progress using `TransferOptions` events
@@ -326,7 +336,6 @@ using Azure.Storage.Blobs;
326
336
usingAzure.Storage.Blobs.Models;
327
337
```
328
338
329
-
330
339
The following code example shows how to instantiate a `BlobContainerClient` for a blob container named `sample-container`:
0 commit comments