You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Performance tuning for uploads and downloads with the Azure Storage client library for .NET
16
16
17
-
When an application transfers data using the Azure Storage client library for .NET, there are several factors that can affect speed, memory usage, and even the success or failure of the request. To maximize performance and reliability for data transfers, it's important to be proactive in configuring client library transfer options based on the environment your app will run in.
17
+
When an application transfers data using the Azure Storage client library for .NET, there are several factors that can affect speed, memory usage, and even the success or failure of the request. To maximize performance and reliability for data transfers, it's important to be proactive in configuring client library transfer options based on the environment your app runs in.
18
18
19
19
This article walks through several considerations for tuning data transfer options, and the guidance applies to any API that accepts `StorageTransferOptions` as a parameter. When properly tuned, the client library can efficiently distribute data across multiple requests, which can result in improved operation speed, memory usage, and network stability.
20
20
@@ -33,17 +33,17 @@ The following properties of `StorageTransferOptions` can be tuned based on the n
33
33
34
34
### InitialTransferSize
35
35
36
-
[InitialTransferSize](/dotnet/api/azure.storage.storagetransferoptions.initialtransfersize) is the size of the first range request in bytes. An HTTP range request is a partial request, with the size defined by `InitialTransferSize` in this case. Blobs smaller than this size will be transferred in a single request. Blobs larger than this size will continue being transferred in chunks of size `MaximumTransferSize`.
36
+
[InitialTransferSize](/dotnet/api/azure.storage.storagetransferoptions.initialtransfersize) is the size of the first range request in bytes. An HTTP range request is a partial request, with the size defined by `InitialTransferSize` in this case. Blobs smaller than this size are transferred in a single request. Blobs larger than this size continue to be transferred in chunks of size `MaximumTransferSize`.
37
37
38
-
It's important to note that the value you specify for `MaximumTransferSize`*does not* limit the value that you define for `InitialTransferSize`. `InitialTransferSize` defines a separate size limitation for an initial request to perform the entire operation at once, with no subtransfers. It's often the case that you'll want `InitialTransferSize` to be *at least* as large as the value you define for `MaximumTransferSize`, if not larger. Depending on the size of the data transfer, this approach can be more performant, as the transfer is completed with a single request and avoids the overhead of multiple requests.
38
+
It's important to note that the value you specify for `MaximumTransferSize`*does not* limit the value that you define for `InitialTransferSize`. `InitialTransferSize` defines a separate size limitation for an initial request to perform the entire operation at once, with no subtransfers. It's often the case that you want `InitialTransferSize` to be *at least* as large as the value you define for `MaximumTransferSize`, if not larger. Depending on the size of the data transfer, this approach can be more performant, as the transfer is completed with a single request and avoids the overhead of multiple requests.
39
39
40
40
If you're unsure of what value is best for your situation, a safe option is to set `InitialTransferSize` to the same value used for `MaximumTransferSize`.
41
41
42
42
> [!NOTE]
43
43
> When using a `BlobClient` object, uploading a blob smaller than the `InitialTransferSize` will be performed using [Put Blob](/rest/api/storageservices/put-blob), rather than [Put Block](/rest/api/storageservices/put-block).
44
44
45
45
### MaximumConcurrency
46
-
[MaximumConcurrency](/dotnet/api/azure.storage.storagetransferoptions.maximumconcurrency) is the maximum number of workers that may be used in a parallel transfer. Currently, only asynchronous operations can parallelize transfers. Synchronous operations will ignore this value and work in sequence.
46
+
[MaximumConcurrency](/dotnet/api/azure.storage.storagetransferoptions.maximumconcurrency) is the maximum number of workers that may be used in a parallel transfer. Currently, only asynchronous operations can parallelize transfers. Synchronous operations ignore this value and work in sequence.
47
47
48
48
The effectiveness of this value is subject to connection pool limits in .NET, which may restrict performance by default in certain scenarios. To learn more about connection pool limits in .NET, see [.NET Framework Connection Pool Limits and the new Azure SDK for .NET](https://devblogs.microsoft.com/azure-sdk/net-framework-connection-pool-limits/).
49
49
@@ -57,7 +57,7 @@ To keep data moving efficiently, the client libraries may not always reach the `
57
57
58
58
The client library includes overloads for the `Upload` and `UploadAsync` methods, which accept a [StorageTransferOptions](/dotnet/api/azure.storage.storagetransferoptions) instance as part of a [BlobUploadOptions](/dotnet/api/azure.storage.blobs.models.blobuploadoptions) parameter. Similar overloads also exist for the `DownloadTo` and `DownloadToAsync` methods, using a [BlobDownloadToOptions](/dotnet/api/azure.storage.blobs.models.blobdownloadoptions) parameter.
59
59
60
-
The following code example shows how to define values for a `StorageTransferOptions` instance and pass these configuration options as a parameter to `UploadAsync`. The values provided in this sample aren't intended to be a recommendation. To properly tune these values, you'll need to consider the specific needs of your app.
60
+
The following code example shows how to define values for a `StorageTransferOptions` instance and pass these configuration options as a parameter to `UploadAsync`. The values provided in this sample aren't intended to be a recommendation. To properly tune these values, you need to consider the specific needs of your app.
61
61
62
62
```csharp
63
63
// Specify the StorageTransferOptions
@@ -80,40 +80,40 @@ BlobUploadOptions options = new BlobUploadOptions
80
80
awaitblobClient.UploadAsync(stream, options);
81
81
```
82
82
83
-
In this example, we set the number of parallel transfer workers to 2, using the `MaximumConcurrency` property. This configuration opens up to 2 connections simultaneously, allowing the upload to happen in parallel. The initial HTTP range request will attempt to upload up to 8 MiB of data, as defined by the `InitialTransferSize` property. Note that `InitialTransferSize` only applies for uploads when [using a seekable stream](#initialtransfersize-on-upload). If the blob size is smaller than 8 MiB, only a single request is necessary to complete the operation. If the blob size is larger than 8 MiB, all subsequent transfer requests will have a maximum size of 4 MiB, which we set with the `MaximumTransferSize` property.
83
+
In this example, we set the number of parallel transfer workers to 2, using the `MaximumConcurrency` property. This configuration opens up to two connections simultaneously, allowing the upload to happen in parallel. The initial HTTP range request attempts to upload up to 8 MiB of data, as defined by the `InitialTransferSize` property. Note that `InitialTransferSize` only applies for uploads when [using a seekable stream](#initialtransfersize-on-upload). If the blob size is smaller than 8 MiB, only a single request is necessary to complete the operation. If the blob size is larger than 8 MiB, all subsequent transfer requests have a maximum size of 4 MiB, which we set with the `MaximumTransferSize` property.
84
84
85
85
## Performance considerations for uploads
86
86
87
-
During an upload, the Storage client libraries will split a given upload stream into multiple subuploads based on the values defined in the `StorageTransferOptions` instance. Each subupload has its own dedicated call to the REST operation. For a `BlobClient` object or `BlockBlobClient` object, this operation is [Put Block](/rest/api/storageservices/put-block). For a `DataLakeFileClient` object, this operation is [Append Data](/rest/api/storageservices/datalakestoragegen2/path/update). The Storage client library manages these REST operations in parallel (depending on transfer options) to complete the full upload.
87
+
During an upload, the Storage client libraries split a given upload stream into multiple subuploads based on the values defined in the `StorageTransferOptions` instance. Each subupload has its own dedicated call to the REST operation. For a `BlobClient` object or `BlockBlobClient` object, this operation is [Put Block](/rest/api/storageservices/put-block). For a `DataLakeFileClient` object, this operation is [Append Data](/rest/api/storageservices/datalakestoragegen2/path/update). The Storage client library manages these REST operations in parallel (depending on transfer options) to complete the full upload.
88
88
89
-
Depending on whether the upload stream is seekable or non-seekable, the client library will handle buffering and `InitialTransferSize` differently, as described in the following sections. A seekable stream is a stream that supports querying and modifying the current position within a stream. To learn more about streams in .NET, see the [Stream class](/dotnet/api/system.io.stream#remarks) reference.
89
+
Depending on whether the upload stream is seekable or non-seekable, the client library handles buffering and `InitialTransferSize` differently, as described in the following sections. A seekable stream is a stream that supports querying and modifying the current position within a stream. To learn more about streams in .NET, see the [Stream class](/dotnet/api/system.io.stream#remarks) reference.
90
90
91
91
> [!NOTE]
92
92
> Block blobs have a maximum block count of 50,000 blocks. The maximum size of your block blob, then, is 50,000 times `MaximumTransferSize`.
93
93
94
94
### Buffering during uploads
95
95
96
-
The Storage REST layer doesn’t support picking up a REST upload operation where you left off; individual transfers are either completed or lost. To ensure resiliency for non-seekable stream uploads, the Storage client libraries buffer data for each individual REST call before starting the upload. In addition to network speed limitations, this buffering behavior is a reason to consider a smaller value for `MaximumTransferSize`, even when uploading in sequence. Decreasing the value of `MaximumTransferSize` decreases the maximum amount of data that will be buffered on each request and each retry of a failed request. If you're experiencing frequent timeouts during data transfers of a certain size, reducing the value of `MaximumTransferSize`will reduce the buffering time, and may result in better performance.
96
+
The Storage REST layer doesn’t support picking up a REST upload operation where you left off; individual transfers are either completed or lost. To ensure resiliency for non-seekable stream uploads, the Storage client libraries buffer data for each individual REST call before starting the upload. In addition to network speed limitations, this buffering behavior is a reason to consider a smaller value for `MaximumTransferSize`, even when uploading in sequence. Decreasing the value of `MaximumTransferSize` decreases the maximum amount of data that is buffered on each request and each retry of a failed request. If you're experiencing frequent timeouts during data transfers of a certain size, reducing the value of `MaximumTransferSize`reduces the buffering time, and may result in better performance.
97
97
98
-
Another scenario where buffering occurs is when you're uploading data with parallel REST calls to maximize network throughput. The client libraries need sources they can read from in parallel, and since streams are sequential, the Storage client libraries will buffer the data for each individual REST call before starting the upload. This buffering behavior occurs even if the provided stream is seekable.
98
+
Another scenario where buffering occurs is when you're uploading data with parallel REST calls to maximize network throughput. The client libraries need sources they can read from in parallel, and since streams are sequential, the Storage client libraries buffer the data for each individual REST call before starting the upload. This buffering behavior occurs even if the provided stream is seekable.
99
99
100
100
To avoid buffering during an asynchronous upload call, you must provide a seekable stream and set `MaximumConcurrency` to 1. While this strategy should work in most situations, it's still possible for buffering to occur if your code is using other client library features that require buffering.
101
101
102
102
### InitialTransferSize on upload
103
103
104
-
When a seekable stream is provided for upload, the stream length will be checked against the value of `InitialTransferSize`. If the stream length is less than this value, the entire stream will be uploaded as a single REST call, regardless of other `StorageTransferOptions` values. Otherwise, the upload will be done in multiple parts as described earlier. `InitialTransferSize` has no effect on a non-seekable stream and will be ignored.
104
+
When a seekable stream is provided for upload, the stream length is checked against the value of `InitialTransferSize`. If the stream length is less than this value, the entire stream is uploaded as a single REST call, regardless of other `StorageTransferOptions` values. Otherwise, the upload is done in multiple parts as described earlier. `InitialTransferSize` has no effect on a non-seekable stream and is ignored.
105
105
106
106
## Performance considerations for downloads
107
107
108
-
During a download, the Storage client libraries will split a given download request into multiple subdownloads based on the values defined in the `StorageTransferOptions` instance. Each subdownload has its own dedicated call to the REST operation. Depending on transfer options, the client libraries manage these REST operations in parallel to complete the full download.
108
+
During a download, the Storage client libraries split a given download request into multiple subdownloads based on the values defined in the `StorageTransferOptions` instance. Each subdownload has its own dedicated call to the REST operation. Depending on transfer options, the client libraries manage these REST operations in parallel to complete the full download.
109
109
110
110
### Buffering during downloads
111
111
112
112
Receiving multiple HTTP responses simultaneously with body contents has implications for memory usage. However, the Storage client libraries don't explicitly add a buffer step for downloaded contents. Incoming responses are processed in order. The client libraries configure a 16-kilobyte buffer for copying streams from an HTTP response stream to a caller-provided destination stream or file path.
113
113
114
114
### InitialTransferSize on download
115
115
116
-
During a download, the Storage client libraries will make one download range request using `InitialTransferSize` before doing anything else. During this initial download request, the client libraries will know the total size of the resource. If the initial request successfully downloaded all of the content, the operation is complete. Otherwise, the client libraries will continue to make range requests up to `MaximumTransferSize` until the full download is complete.
116
+
During a download, the Storage client libraries make one download range request using `InitialTransferSize` before doing anything else. During this initial download request, the client libraries know the total size of the resource. If the initial request successfully downloaded all of the content, the operation is complete. Otherwise, the client libraries continue to make range requests up to `MaximumTransferSize` until the full download is complete.
Copy file name to clipboardExpand all lines: includes/storage-dev-guides/storage-dev-guide-about-blob-download.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,4 +12,6 @@ ms.custom: include file
12
12
13
13
## About downloading blobs
14
14
15
-
The client library methods covered in this article wrap the [Get Blob](/rest/api/storageservices/put-blob) operation. This operation reads or downloads a blob from Azure Storage, including its metadata and properties. To learn more about the `Get Blob` operation, including timeout parameters and error conditions, see [Get Blob remarks](/rest/api/storageservices/get-blob#remarks).
15
+
The client library methods covered in this article wrap the [Get Blob](/rest/api/storageservices/put-blob) operation. This operation reads or downloads a blob from Azure Storage, including its metadata and properties. To learn more about the `Get Blob` operation, including timeout parameters and error conditions, see [Get Blob remarks](/rest/api/storageservices/get-blob#remarks).
16
+
17
+
To maximize performance and reliability for download operations, it's important to be proactive in configuring client library transfer options based on the environment your app runs in. To learn more about considerations for tuning data transfer options, see [Performance tuning for uploads and downloads](../../articles/storage/blobs/storage-blobs-tune-upload-download.md).
Copy file name to clipboardExpand all lines: includes/storage-dev-guides/storage-dev-guide-about-blob-upload.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,4 +14,6 @@ ms.custom: include file
14
14
15
15
The client library methods covered in this article wrap the [Put Blob](/rest/api/storageservices/put-blob) operation. This operation creates a new block blob, page blob, or append blob, or updates the content of an existing block blob. If you're updating an existing block blob, any existing metadata on the blob is overwritten. To learn more about the `Put Blob` operation, including blob size limitations for write operations, see [Put Blob remarks](/rest/api/storageservices/put-blob#remarks).
16
16
17
-
Staging individual blocks of data uses the [Put Block](/rest/api/storageservices/put-block) operation. This operation creates a new block to be committed as part of a blob.
17
+
Staging individual blocks of data uses the [Put Block](/rest/api/storageservices/put-block) operation. This operation creates a new block, which can be committed as to block blob using [Put Block List](/rest/api/storageservices/put-block-list).
18
+
19
+
To maximize performance and reliability for upload operations, it's important to be proactive in configuring client library transfer options based on the environment your app runs in. To learn more about considerations for tuning data transfer options, see [Performance tuning for uploads and downloads](../../articles/storage/blobs/storage-blobs-tune-upload-download.md).
0 commit comments