Skip to content

Commit da87840

Browse files
Final review edits
1 parent 1aabd9f commit da87840

File tree

1 file changed

+45
-36
lines changed

1 file changed

+45
-36
lines changed

articles/storage/common/storage-use-data-movement-library.md

Lines changed: 45 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ If you're migrating code from the older **Microsoft.Azure.Storage.DataMovement**
3232

3333
## Set up your environment
3434

35-
If you don't have an existing project, this section shows you how to set up a project to work with the Azure Blob Storage client library for .NET. The steps include package installation, adding `using` directives, and creating an authorized client object. For details, see [Get started with Azure Blob Storage and .NET](../blobs/storage-blob-dotnet-get-started.md).
35+
If you don't have an existing project, this section shows you how to set up a project to work with the Azure Blob Storage client library for .NET. The steps include package installation, adding `using` directives, and creating an authorized client object.
3636

3737
#### Install packages
3838

@@ -46,7 +46,7 @@ dotnet add package Azure.Identity
4646

4747
#### Add `using` directives
4848

49-
Add these `using` directives to the top of your code file:
49+
Add these `using` directives to your code file:
5050

5151
```csharp
5252
using Azure;
@@ -81,7 +81,13 @@ The following code shows how to create a `TransferManager` object:
8181
TransferManager transferManager = new(new TransferManagerOptions());
8282
```
8383

84-
You can optionally provide an instance of [TransferManagerOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions) to the constructor, which applies certain configuration options, including maximum concurrency, to all transfers started by the `TransferManager` object.
84+
You can optionally provide an instance of [TransferManagerOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions) to the constructor, which applies certain configuration options to all transfers started by the `TransferManager` object. The following configuration options are available:
85+
86+
- [CheckpointStoreOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions.checkpointstoreoptions): Optional. Defines the options for creating a checkpoint used for saving transfer state so transfers can be resumed.
87+
- [Diagnostics](/dotnet/api/azure.storage.datamovement.transfermanageroptions.diagnostics): Gets the transfer manager diagnostic options.
88+
- [ErrorHandling](/dotnet/api/azure.storage.datamovement.transfermanageroptions.errorhandling): Optional. Defines how errors are handled during a transfer. Default is `StopOnAnyFailure`.
89+
- [MaximumConcurrency](/dotnet/api/azure.storage.datamovement.transfermanageroptions.maximumconcurrency): The maximum number of workers that may be used in a parallel transfer.
90+
- [ProvidersForResuming](/dotnet/api/azure.storage.datamovement.transfermanageroptions.providersforresuming): Resource providers for the transfer manager to use in resuming a transfer. Expects one provider for each storage provider in use. To learn more, see [Resume an existing transfer](#resume-an-existing-transfer).
8591

8692
## Create a `StorageResource` object
8793

@@ -113,7 +119,7 @@ StorageResource blockBlob = await blobsProvider.FromBlobAsync(
113119
// Use a similar approach to get a page blob or append blob resource
114120
```
115121

116-
You can also create a `StorageResource` object using a client object from **Azure.Storage.Blobs**:
122+
You can also create a `StorageResource` object using a client object from **Azure.Storage.Blobs**.
117123

118124
```csharp
119125
// Create a token credential
@@ -124,29 +130,32 @@ BlobContainerClient blobContainerClient = new(
124130
tokenCredential);
125131
StorageResource containerResource = BlobsStorageResourceProvider.FromClient(blobContainerClient);
126132

127-
BlockBlobClient blockBlobClient = new(
128-
new Uri("https://<storage-account-name>.blob.core.windows.net/sample-container/sample-block-blob"),
129-
tokenCredential);
133+
BlockBlobClient blockBlobClient = blobContainerClient.GetBlockBlobClient("sample-block-blob");
130134
StorageResource blockBlobResource = BlobsStorageResourceProvider.FromClient(blockBlobClient);
131135

132136
// Use a similar approach to get a page blob or append blob resource
133137
```
134138

135139
## Start a new transfer
136140

137-
Transfers are defined by a source and a destination. Both the source and destination are type `StorageResource`, which can be either `StorageResourceContainer` or `StorageResourceItem`. For a given transfer, the source and destination must be of the same kind. For example, if the source is a blob container, the destination must be a blob container.
141+
All transfers need to specify a source and a destination. Both the source and destination are type `StorageResource`, which can be either `StorageResourceContainer` or `StorageResourceItem`. For a given transfer, the source and destination must be of the same kind. For example, if the source is a blob container, the destination must be a blob container.
138142

139143
You can start a new transfer by calling the following method:
140144

141145
- [TransferManager.StartTransferAsync](/dotnet/api/azure.storage.datamovement.transfermanager.starttransferasync)
142146

143147
This method returns a [TransferOperation](/dotnet/api/azure.storage.datamovement.transferoperation) object that represents the transfer. You can use the `TransferOperation` object to monitor the transfer progress or obtain the transfer ID. The transfer ID is a unique identifier for the transfer that's needed to [resume a transfer](#resume-an-existing-transfer) or pause a transfer.
144148

145-
You can optionally provide an instance of [TransferOptions](/dotnet/api/azure.storage.datamovement.transferoptions) to `StartTransferAsync`, which applies certain configuration options, including creation preference and transfer size, to a specific transfer.
149+
You can optionally provide an instance of [TransferOptions](/dotnet/api/azure.storage.datamovement.transferoptions) to `StartTransferAsync` and `ResumeTransferAsync` methods, which applies certain configuration options to a specific transfer. The following configuration options are available:
146150

147-
### Example: Upload a local file to a blob
151+
- [CreationMode](/dotnet/api/azure.storage.datamovement.transferoptions.creationmode): Configures the behavior when a transfer encounters a resource that already exists. Defaults to `FailIfExists` when starting a new transfer. When resuming a transfer, the defaults can vary. For all resources successfully enumerated when the transfer started, `CreationMode` defaults to the initial value used. For any remaining resources, the regular default value applies.
152+
- [InitialTransferSize](/dotnet/api/azure.storage.datamovement.transferoptions.initialtransfersize): The size of the first range request in bytes. Single transfer sizes smaller than this limit are uploaded or downloaded in a single request. Transfers larger than this limit will continue being downloaded or uploaded in chunks of size [MaximumTransferChunkSize](/dotnet/api/azure.storage.datamovement.transferoptions.maximumtransferchunksize). The default value is 32 MiB. When resuming a transfer, the default value is the value specified when the transfer first started.
153+
- [MaximumTransferChunkSize](/dotnet/api/azure.storage.datamovement.transferoptions.maximumtransferchunksize): The maximum size to use for each chunk when transferring data in chunks. The default value is 4 MiB. When resuming a transfer, the default value is the value specified when the transfer first started.
154+
- [ProgressHandlerOptions](/dotnet/api/azure.storage.datamovement.transferoptions.progresshandleroptions): Optional. Options for changing behavior of the ProgressHandler.
148155

149-
The following code example shows how to start a new transfer to upload a local file to a blob:
156+
### Example: Upload a local directory to a blob container
157+
158+
The following code example shows how to start a new transfer to upload a local directory to a blob container:
150159

151160
```csharp
152161
// Create a token credential
@@ -156,21 +165,20 @@ TransferManager transferManager = new(new TransferManagerOptions());
156165

157166
BlobsStorageResourceProvider blobsProvider = new(tokenCredential);
158167

159-
string localFilePath = "C:/path/to/file.txt";
160-
Uri blobUri = new Uri("https://<storage-account-name>.blob.core.windows.net/sample-container/sample-blob");
168+
string localDirectoryPath = "C:/path/to/directory";
169+
Uri blobContainerUri = new Uri("https://<storage-account-name>.blob.core.windows.net/sample-container");
161170

162171
TransferOperation transferOperation = await transferManager.StartTransferAsync(
163-
sourceResource: LocalFilesStorageResourceProvider.FromFile(localFilePath),
164-
destinationResource: await blobsProvider.FromBlobAsync(
165-
new Uri(blobUri)));
172+
sourceResource: LocalFilesStorageResourceProvider.FromDirectory(localDirectoryPath),
173+
destinationResource: await blobsProvider.FromContainerAsync(blobContainerUri));
166174
await transferOperation.WaitForCompletionAsync();
167175
```
168176

169177
### Example: Copy a container or blob
170178

171-
You can use the Data Movement library to copy between two `StorageResource` instances. For blob resources, the transfer uses the [Put Blob From URL](/rest/api/storageservices/put-blob-from-url) operation, which performs a server-to-server copy.
179+
You can use the Data Movement library to copy between two `StorageResource` instances. For blob resources, the transfer uses the [Put Blob From URL](/rest/api/storageservices/put-blob-from-url) operation, which performs a server-to-server copy.
172180

173-
The following code example shows how to start a new transfer to copy a source blob container to a destination blob container:
181+
The following code example shows how to start a new transfer to copy all blobs in a source blob container to a destination blob container. The destination container must already exist. In this example, we set [CreationMode](/dotnet/api/azure.storage.datamovement.storageresourcecreationmode) to `OverwriteIfExists` to overwrite any destination blobs that already exist. You can adjust the `CreationMode` property based on the needs of your app.
174182

175183
```csharp
176184
Uri sourceContainerUri = new Uri("https://<storage-account-name>.blob.core.windows.net/source-container");
@@ -183,21 +191,24 @@ TransferOperation transferOperation = await transferManager.StartTransferAsync(
183191
{
184192
BlobPrefix = "source/directory/prefix"
185193
}),
186-
destinationResource: blobsProvider.FromContainer(
194+
destinationResource: await blobsProvider.FromContainerAsync(
187195
destinationContainerUri,
188196
new BlobStorageResourceContainerOptions()
189197
{
190198
// All source blobs are copied as a single type of destination blob
191199
// Defaults to block blob, if not specified
192200
BlobType = BlobType.Block,
193201
BlobPrefix = "destination/directory/prefix"
194-
}
195-
)
196-
);
202+
}),
203+
transferOptions: new TransferOptions()
204+
{
205+
CreationMode = StorageResourceCreationMode.OverwriteIfExists,
206+
}
207+
);
197208
await transferOperation.WaitForCompletionAsync();
198209
```
199210

200-
The following code example shows how to start a new transfer to copy a source blob to a destination blob:
211+
The following code example shows how to start a new transfer to copy a source blob to a destination blob. In this example, we set [CreationMode](/dotnet/api/azure.storage.datamovement.storageresourcecreationmode) to `OverwriteIfExists` to overwrite the destination blob if it already exists. You can adjust the `CreationMode` property based on the needs of your app.
201212

202213
```csharp
203214
Uri sourceBlobUri = new Uri(
@@ -207,31 +218,30 @@ Uri destinationBlobUri = new Uri(
207218

208219
TransferOperation transferOperation = await transferManager.StartTransferAsync(
209220
sourceResource: await blobsProvider.FromBlobAsync(sourceBlobUri),
210-
destinationResource: await blobsProvider.FromBlobAsync(destinationBlobUri, new BlockBlobStorageResourceOptions()));
221+
destinationResource: await blobsProvider.FromBlobAsync(destinationBlobUri, new BlockBlobStorageResourceOptions()),
222+
transferOptions: new TransferOptions()
223+
{
224+
CreationMode = StorageResourceCreationMode.OverwriteIfExists,
225+
}
226+
);
211227
await transferOperation.WaitForCompletionAsync();
212228
```
213229

214-
### Set configuration options for a transfer
215-
216-
You can set configuration options for a transfer by providing an instance of [TransferOptions](/dotnet/api/azure.storage.datamovement.transferoptions) to the `StartTransferAsync` method. The `TransferOptions` class provides properties that allow you to set the number of parallel operations, the size of the buffer, and more.
217-
218230
## Resume an existing transfer
219231

220-
By persisting transfer progress to disk, the Data Movement library allows you to resume a transfer that failed before completion, or was otherwise canceled or paused. To resume a transfer, the `TransferManager` object must be configured with `StorageResourceProvider` instances that are capable of reassembling the transfer from the persisted data. You can use the `ResumeProviders` property of the [TransferManagerOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions) class to specify the providers.
232+
By persisting transfer progress to disk, the Data Movement library allows you to resume a transfer that failed before completion, or was otherwise canceled or paused. To resume a transfer, the `TransferManager` object must be configured with `StorageResourceProvider` instances that are capable of reassembling the transfer from the persisted data. You can use the `ProvidersForResuming` property of the [TransferManagerOptions](/dotnet/api/azure.storage.datamovement.transfermanageroptions) class to specify the providers.
221233

222234
The following code example shows how to initialize a `TransferManager` object that's capable of resuming a transfer between the local file system and Blob Storage:
223235

224236
```csharp
225237
// Create a token credential
226238
TokenCredential tokenCredential = new DefaultAzureCredential();
227239

228-
BlobsStorageResourceProvider blobsProvider = new(tokenCredential);
229-
230240
TransferManager transferManager = new(new TransferManagerOptions()
231241
{
232242
ProvidersForResuming = new List<StorageResourceProvider>()
233243
{
234-
blobsProvider
244+
new BlobsStorageResourceProvider(tokenCredential)
235245
}
236246
});
237247
```
@@ -259,7 +269,7 @@ Transfers can be monitored and observed through several mechanisms, depending on
259269

260270
You can monitor transfer progress using the `TransferOperation` object returned by the `StartTransferAsync` method. You can also call [TransferManager.GetTransfersAsync](/dotnet/api/azure.storage.datamovement.transfermanager.gettransfersasync) to enumerate all transfers for a `TransferManager` object.
261271

262-
The following code example shows how to iterate over all transfers and write the status of each transfer to a log file:
272+
The following code example shows how to iterate over all transfers and write the state of each transfer to a log file:
263273

264274
```csharp
265275
async Task CheckTransfersAsync(TransferManager transferManager)
@@ -272,13 +282,13 @@ async Task CheckTransfersAsync(TransferManager transferManager)
272282
}
273283
```
274284

275-
The `TransferStatus` property returns a [TransferStatus](/dotnet/api/azure.storage.datamovement.transferstatus) value. `TransferStatus` includes the following properties:
285+
[TransferStatus](/dotnet/api/azure.storage.datamovement.transferstatus) defines the status of the transfer job. `TransferStatus` includes the following properties:
276286

277287
| Property | Type | Description |
278288
| --- | --- | --- |
279289
| `HasCompletedSuccessfully` | Boolean | Represents if the transfer completes successfully without any failure or skipped items. |
280290
| `HasFailedItems` | Boolean | Represents if transfer has any failure items. If set to `true`, the transfer has at least one failure item. If set to `false`, the transfer currently has no failures. |
281-
| `HasSkippedItems` | Boolean | Represents if transfer has any skipped items. If set to `true`, the transfer has at least one skipped item. If set to `false`, the transfer currently has no skipped items. It's possible to never have any items skipped if `SkipIfExists` isn't enabled in [TransferOptions.CreationPreference](/dotnet/api/azure.storage.datamovement.transferoptions.creationpreference). |
291+
| `HasSkippedItems` | Boolean | Represents if transfer has any skipped items. If set to `true`, the transfer has at least one skipped item. If set to `false`, the transfer currently has no skipped items. It's possible to never have any items skipped if `SkipIfExists` isn't enabled in [TransferOptions.CreationMode](/dotnet/api/azure.storage.datamovement.transferoptions.creationmode). |
282292
| `State` | [TransferState](/dotnet/api/azure.storage.datamovement.transferstate) | Defines the types of the state a transfer can have. See [TransferState](/dotnet/api/azure.storage.datamovement.transferstate) for details. |
283293

284294
### Example: Monitor transfer progress using `TransferOptions` events
@@ -326,7 +336,6 @@ using Azure.Storage.Blobs;
326336
using Azure.Storage.Blobs.Models;
327337
```
328338

329-
330339
The following code example shows how to instantiate a `BlobContainerClient` for a blob container named `sample-container`:
331340

332341
```csharp

0 commit comments

Comments
 (0)