Skip to content

Commit aded8aa

Browse files
v-dirichardsv-dirichards
authored
Merge pull request #219475 from pauljewellmsft/pauljewell-tuning-upload-download
Add new article for tuning uploads and downloads
2 parents bda82b1 + a15a1d9 commit aded8aa

File tree

2 files changed

+123
-0
lines changed

2 files changed

+123
-0
lines changed

‎articles/storage/blobs/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -618,6 +618,8 @@ items:
618618
href: storage-blob-append.md
619619
- name: Implement a retry policy
620620
href: storage-retry-policy.md
621+
- name: Performance tuning for uploads and downloads
622+
href: storage-blobs-tune-upload-download.md
621623
- name: JavaScript
622624
items:
623625
- name: Get started

‎articles/storage/blobs/storage-blobs-tune-upload-download.md

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
---
2+
title: Performance tuning for uploads and downloads with Azure Storage client library for .NET - Azure Storage
3+
description: Learn how to tune your uploads and downloads for better performance with Azure Storage client library for .NET.
4+
services: storage
5+
author: pauljewellmsft
6+
ms.author: pauljewell
7+
ms.service: storage
8+
ms.topic: how-to
9+
ms.date: 12/09/2022
10+
ms.subservice: blobs
11+
ms.devlang: csharp
12+
ms.custom: devx-track-csharp, devguide-csharp
13+
---
14+
15+
# Performance tuning for uploads and downloads with the Azure Storage client library for .NET
16+
17+
When an application transfers data using the Azure Storage client library for .NET, there are several factors that can affect speed, memory usage, and even the success or failure of the request. To maximize performance and reliability for data transfers, it's important to be proactive in configuring client library transfer options based on the environment your app will run in.
18+
19+
This article walks through several considerations for tuning data transfer options, and the guidance applies to any API that accepts `StorageTransferOptions` as a parameter. When properly tuned, the client library can efficiently distribute data across multiple requests, which can result in improved operation speed, memory usage, and network stability.
20+
21+
## Performance tuning with StorageTransferOptions
22+
23+
Properly tuning the values in [StorageTransferOptions](/dotnet/api/azure.storage.storagetransferoptions) is key to reliable performance for data transfer operations. Storage transfers are partitioned into several subtransfers based on the property values defined in an instance of this struct. The maximum supported transfer size varies by operation and service version, so be sure to check the documentation to determine the limits. For more information on transfer size limits for Blob storage, see [Scale targets for Blob storage](scalability-targets.md#scale-targets-for-blob-storage).
24+
25+
The following properties of `StorageTransferOptions` can be tuned based on the needs of your app:
26+
27+
- [InitialTransferSize](#initialtransfersize) - the size of the first request in bytes
28+
- [MaximumConcurrency](#maximumconcurrency) - the maximum number of subtransfers that may be used in parallel
29+
- [MaximumTransferSize](#maximumtransfersize) - the maximum length of a transfer in bytes
30+
31+
> [!NOTE]
32+
> While the `StorageTransferOptions` struct contains nullable values, the client libraries will use defaults for each individual value, if not provided. These defaults are typically performant in a data center environment, but not likely to be suitable for home consumer environments. Poorly tuned `StorageTransferOptions` can result in excessively long operations and even request timeouts. It's best to be proactive in testing the values in `StorageTransferOptions`, and tuning them based on the needs of your application and environment.
33+
34+
### InitialTransferSize
35+
36+
[InitialTransferSize](/dotnet/api/azure.storage.storagetransferoptions.initialtransfersize) is the size of the first range request in bytes. An HTTP range request is a partial request, with the size defined by `InitialTransferSize` in this case. Blobs smaller than this size will be transferred in a single request. Blobs larger than this size will continue being transferred in chunks of size `MaximumTransferSize`.
37+
38+
It's important to note that the value you specify for `MaximumTransferSize` *does not* limit the value that you define for `InitialTransferSize`. `InitialTransferSize` defines a separate size limitation for an initial request to perform the entire operation at once, with no subtransfers. It's often the case that you'll want `InitialTransferSize` to be *at least* as large as the value you define for `MaximumTransferSize`, if not larger. Depending on the size of the data transfer, this approach can be more performant, as the transfer is completed with a single request and avoids the overhead of multiple requests.
39+
40+
If you're unsure of what value is best for your situation, a safe option is to set `InitialTransferSize` to the same value used for `MaximumTransferSize`.
41+
42+
> [!NOTE]
43+
> When using a `BlobClient` object, uploading a blob smaller than the `InitialTransferSize` will be performed using [Put Blob](/rest/api/storageservices/put-blob), rather than [Put Block](/rest/api/storageservices/put-block).
44+
45+
### MaximumConcurrency
46+
[MaximumConcurrency](/dotnet/api/azure.storage.storagetransferoptions.maximumconcurrency) is the maximum number of workers that may be used in a parallel transfer. Currently, only asynchronous operations can parallelize transfers. Synchronous operations will ignore this value and work in sequence.
47+
48+
The effectiveness of this value is subject to connection pool limits in .NET, which may restrict performance by default in certain scenarios. To learn more about connection pool limits in .NET, see [.NET Framework Connection Pool Limits and the new Azure SDK for .NET](https://devblogs.microsoft.com/azure-sdk/net-framework-connection-pool-limits/).
49+
50+
### MaximumTransferSize
51+
52+
[MaximumTransferSize](/dotnet/api/azure.storage.storagetransferoptions.maximumtransfersize) is the maximum length of a transfer in bytes. As mentioned earlier, this value *does not* limit `InitialTransferSize`, which can be larger than `MaximumTransferSize`.
53+
54+
To keep data moving efficiently, the client libraries may not always reach the `MaximumTransferSize` value for every transfer. Depending on the operation, the maximum supported value for transfer size can vary. For example, block blobs calling the [Put Block](/rest/api/storageservices/put-block#remarks) operation with a service version of 2019-12-12 or later have a maximum block size of 4000 MiB. For more information on transfer size limits for Blob storage, see the chart in [Scale targets for Blob storage](scalability-targets.md#scale-targets-for-blob-storage).
55+
56+
### Code example
57+
58+
The client library includes overloads for the `Upload` and `UploadAsync` methods, which accept a [StorageTransferOptions](/dotnet/api/azure.storage.storagetransferoptions) instance as part of a [BlobUploadOptions](/dotnet/api/azure.storage.blobs.models.blobuploadoptions) parameter. Similar overloads also exist for the `DownloadTo` and `DownloadToAsync` methods, using a [BlobDownloadToOptions](/dotnet/api/azure.storage.blobs.models.blobdownloadoptions) parameter.
59+
60+
The following code example shows how to define values for a `StorageTransferOptions` instance and pass these configuration options as a parameter to `UploadAsync`. The values provided in this sample aren't intended to be a recommendation. To properly tune these values, you'll need to consider the specific needs of your app.
61+
62+
```csharp
63+
// Specify the StorageTransferOptions
64+
BlobUploadOptions options = new BlobUploadOptions
65+
{
66+
TransferOptions = new StorageTransferOptions
67+
{
68+
// Set the maximum number of parallel transfer workers
69+
MaximumConcurrency = 2,
70+
71+
// Set the initial transfer length to 8 MiB
72+
InitialTransferSize = 8 * 1024 * 1024,
73+
74+
// Set the maximum length of a transfer to 4 MiB
75+
MaximumTransferSize = 4 * 1024 * 1024;
76+
}
77+
};
78+
79+
// Upload data from a stream
80+
await blobClient.UploadAsync(stream, options);
81+
```
82+
83+
In this example, we set the number of parallel transfer workers to 2, using the `MaximumConcurrency` property. This configuration opens up to 2 connections simultaneously, allowing the upload to happen in parallel. The initial HTTP range request will attempt to upload up to 8 MiB of data, as defined by the `InitialTransferSize` property. Note that `InitialTransferSize` only applies for uploads when [using a seekable stream](#initialtransfersize-on-upload). If the blob size is smaller than 8 MiB, only a single request is necessary to complete the operation. If the blob size is larger than 8 MiB, all subsequent transfer requests will have a maximum size of 4 MiB, which we set with the `MaximumTransferSize` property.
84+
85+
## Performance considerations for uploads
86+
87+
During an upload, the Storage client libraries will split a given upload stream into multiple subuploads based on the values defined in the `StorageTransferOptions` instance. Each subupload has its own dedicated call to the REST operation. For a `BlobClient` object or `BlockBlobClient` object, this operation is [Put Block](/rest/api/storageservices/put-block). For a `DataLakeFileClient` object, this operation is [Append Data](/rest/api/storageservices/datalakestoragegen2/path/update). The Storage client library manages these REST operations in parallel (depending on transfer options) to complete the full upload.
88+
89+
Depending on whether the upload stream is seekable or non-seekable, the client library will handle buffering and `InitialTransferSize` differently, as described in the following sections. A seekable stream is a stream that supports querying and modifying the current position within a stream. To learn more about streams in .NET, see the [Stream class](/dotnet/api/system.io.stream#remarks) reference.
90+
91+
> [!NOTE]
92+
> Block blobs have a maximum block count of 50,000 blocks. The maximum size of your block blob, then, is 50,000 times `MaximumTransferSize`.
93+
94+
### Buffering during uploads
95+
96+
The Storage REST layer doesn’t support picking up a REST upload operation where you left off; individual transfers are either completed or lost. To ensure resiliency for non-seekable stream uploads, the Storage client libraries buffer data for each individual REST call before starting the upload. In addition to network speed limitations, this buffering behavior is a reason to consider a smaller value for `MaximumTransferSize`, even when uploading in sequence. Decreasing the value of `MaximumTransferSize` decreases the maximum amount of data that will be buffered on each request and each retry of a failed request. If you're experiencing frequent timeouts during data transfers of a certain size, reducing the value of `MaximumTransferSize` will reduce the buffering time, and may result in better performance.
97+
98+
Another scenario where buffering occurs is when you're uploading data with parallel REST calls to maximize network throughput. The client libraries need sources they can read from in parallel, and since streams are sequential, the Storage client libraries will buffer the data for each individual REST call before starting the upload. This buffering behavior occurs even if the provided stream is seekable.
99+
100+
To avoid buffering during an asynchronous upload call, you must provide a seekable stream and set `MaximumConcurrency` to 1. While this strategy should work in most situations, it's still possible for buffering to occur if your code is using other client library features that require buffering.
101+
102+
### InitialTransferSize on upload
103+
104+
When a seekable stream is provided for upload, the stream length will be checked against the value of `InitialTransferSize`. If the stream length is less than this value, the entire stream will be uploaded as a single REST call, regardless of other `StorageTransferOptions` values. Otherwise, the upload will be done in multiple parts as described earlier. `InitialTransferSize` has no effect on a non-seekable stream and will be ignored.
105+
106+
## Performance considerations for downloads
107+
108+
During a download, the Storage client libraries will split a given download request into multiple subdownloads based on the values defined in the `StorageTransferOptions` instance. Each subdownload has its own dedicated call to the REST operation. Depending on transfer options, the client libraries manage these REST operations in parallel to complete the full download.
109+
110+
### Buffering during downloads
111+
112+
Receiving multiple HTTP responses simultaneously with body contents has implications for memory usage. However, the Storage client libraries don't explicitly add a buffer step for downloaded contents. Incoming responses are processed in order. The client libraries configure a 16-kilobyte buffer for copying streams from an HTTP response stream to a caller-provided destination stream or file path.
113+
114+
### InitialTransferSize on download
115+
116+
During a download, the Storage client libraries will make one download range request using `InitialTransferSize` before doing anything else. During this initial download request, the client libraries will know the total size of the resource. If the initial request successfully downloaded all of the content, the operation is complete. Otherwise, the client libraries will continue to make range requests up to `MaximumTransferSize` until the full download is complete.
117+
118+
## Next steps
119+
120+
- To understand more about factors that can influence performance for Azure Storage operations, see [Latency in Blob storage](storage-blobs-latency.md).
121+
- To see a list of design considerations to optimize performance for apps using Blob storage, see [Performance and scalability checklist for Blob storage](storage-performance-checklist.md).

0 commit comments

Comments
 (0)