Perform blob download-batch concurrently

[ryancasburn-KAI]

Related command
az storage blob download-batch

Is your feature request related to a problem? Please describe.
I have a need to download a large number of small blobs. I set the "--max-connections" property thinking this would allow for multiple files to download at once, but this does not appear to be the case. I'm downloading these blobs to an Azure VM (with more than 20 cores) in the same region and network receive rate is hovering around 1 MiB/second. I'd expect that rate to feasibly be well over 100 MiB/second or more with concurrency.

Describe the solution you'd like
My understanding reading through the code repository is that this functionality is handled with this function:

azure-cli/src/azure-cli/azure/cli/command_modules/storage/operations/blob.py

Line 426 in a555430

def storage_blob_download_batch(client, source, destination, source_container_name, pattern=None, dryrun=False,

At a minimum, I'd like to see the for loop which loops over the blobs_to_download dictionary be converted into an asynchronous process. This either means that _download_blob function (and everything lower in the stack) would need to be async functions. Or an async run in thread.

It would be even more awesome if the collect_blobs function could return an async generator, which could allow the downloads to start before the full list of blobs has been collected (though I realize this has more side effects). My current download took several hours to accumulate the list of blobs to download (since the API list call is paginated).

Describe alternatives you've considered
There isn't really any great alternatives other than rewriting my own version of this api which is undesirable.

Additional context
The sample download-batch I am running is 12,024,850 blobs with an average size of 10KiB. The download is running on an Azure VM of size: NC24ads_A100_v4 in the same region (East US 2) as the storage account. My expectation is that this would feasibly be able to be complete in under 2 hours, yet we are at about 14 hours and not yet to 30% complete yet.

[github-actions]

Here are some similar issues that might help you. Please check if they can solve your problem.

• azure storage blob upload-batch needs a --parallelism flag #7616

---

Possible solution (Extracted from existing issue, might be incorrect; please verify carefully)

Solution 1:

Please try to use az storage copy in Azure CLI 2.0.71 or newer version, which already integrates AzCopy and can work in parallel.

Reference:

• azure storage blob upload-batch needs a --parallelism flag #7616 (comment)

Solution 2:

The approach of invoking AzCopy through CLI, which is ongoing in the CLI extension, should improve it. We will work out the plan to bring this option to the main repo when it is ready.

Reference:

• azure storage blob upload-batch needs a --parallelism flag #7616 (comment)

Solution 3:

The functionality is in our storage-preview extension now. Let us know if you are satisfied with the performance. Try that like:

az storage azcopy blob upload \
    -c containerName \
    --account-name accountName \
    -s directory/path \
    -d upload/path \
    --recursive

Reference:

• azure storage blob upload-batch needs a --parallelism flag #7616 (comment)

Powered by issue-sentinel

[yonzhan]

Thank you for opening this issue, we will look into it.

[ryancasburn-KAI]

Thanks to the issue-sentinel bot for pointing me to az storage azcopy blob download. I stopped the download-batch I started 14 hours ago and replaced it with this version. Within 10 minutes we are further than 14 hours got us with download-batch.

It seems like not all the features of download-batch are supported with azcopy (like pattern), but this solves many use cases. Can you please update the docs of download-batch to point to this preview version? Since it is in a different category (azcopy) rather than blob, it doesn't show up on the same page so it hides the fact this is even possible.

Now if I could only get back my wasted 14 hours of NC24ads_A100_v4 time...