MemoryError in ADO build agent when downloading 4+ GB SQL bacpac file from storage account using az storage fs file download

[tbrothers]

Describe the bug

The az storage fs file download command worked fine a few months ago when I was downloading a 1.9GB sql bacpac file from our storage account to the build machine, but lately the bacpac file has grown to over 4 GB and I'm getting a memory error when trying to download the file to the agent machine.

Related command

az storage fs file download -p database.bacpac -d C:\database.bacpac -f container --overwrite True --account-name storageaccount --auth-mode login

Errors

ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR:
Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 664, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 729, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 721, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 430, in new_handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 429, in new_handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 698, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 334, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/operations/fs_file.py", line 43, in download_file
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/filedatalake/v2021_08_06/_download.py", line 60, in readall
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 659, in readall
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 772, in readinto
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 131, in process_chunk
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 154, in _write_to_stream
MemoryError
To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

Issue script & Debug output

az storage fs file download -p database.bacpac -d C:\database.bacpac -f container --overwrite True --account-name storageaccount --auth-mode login --debug

DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 664, in execute
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 729, in _run_jobs_serially
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 721, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 430, in new_handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/init.py", line 429, in new_handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 698, in _run_job
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 334, in call
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/storage/operations/fs_file.py", line 43, in download_file
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/filedatalake/v2021_08_06/_download.py", line 60, in readall
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 659, in readall
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 772, in readinto
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 131, in process_chunk
File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/multiapi/storagev2/blob/v2021_08_06/_download.py", line 154, in _write_to_stream
MemoryError

ERROR: cli.azure.cli.core.azclierror: The command failed with an unexpected error. Here is the traceback:
ERROR: az_command_data_logger: The command failed with an unexpected error. Here is the traceback:
ERROR: cli.azure.cli.core.azclierror:
Traceback (most recent call last):

Expected behavior

The az storage fs download command should download the bacpac file to the agent build machine.

After that is completely, the import to SQL database command is run using SqlPackage.exe Action Import...

Environment Summary

azure-cli 2.57.0

core 2.57.0
telemetry 1.1.0

Extensions:
azure-devops 0.26.0

Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'

Python (Windows) 3.11.7 (tags/v3.11.7:fa7a6f2, Dec 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[calvinhzy]

Hi @tbrothers, wondering how much free memory is left for the machine/vm at the time of downloading? Would the az storage blob download command work for your use case? It might be a bit faster as well. Still trying to replicate this MemoryError.

[arrwhidev]

I am experiencing the same issue as @tbrothers. I can see the python process memory usage spiking before the MemoryError happens. It appears that az storage fs file download is attempting to download the file into memory entirely before writing it to disk. This is definitely going to cause issues with large files (>1GB in my case).

I just switched over to az storage blob download as suggested by @calvinhzy and this is working perfectly.