az vm list-skus is very slow and sometimes fails #31592
Description
Describe the bug
A command that is needed in automation is 'az vm list-skus' as we need to get specifics about a VM SKU before trying to create it.
However, the command, even when given very exacting information, is amazingly slow.
For example:
az vm list-skus --location westus2 --resource-type virtualMachines --size Standard_D8plds_v5
Takes on the order of 2 minutes+ when executed to return that one VM SKU's data. In some environments, this never actually succeeds (secure access workstations running Windows, for example)
The whole json for all sizes is only 77k with gzip compression and expended is 4.5meg. I can do jq queries over that data nearly instantly. (Unfortunately, getting the data is very, very slow) -- Looking at the debug output, it seems that there is even more data provided (122 megs) and then the filtering happens client side but that filtering is really, really slow.
This makes automation commands that provide for VM creation and provisioning really slow or they need to embed the information which is also not very good.
I have tried it with and without the --resource-type option to help the system filter better but it did not change the timing.
For example, I just ran this:
time az vm list-skus --location westus2 --resource-type virtualMachines --size Standard_D8plds_v5 >/dev/null
real 2m22.999s
user 2m3.916s
sys 0m1.653sRelated command
The prior av vm list-sizes was also slow and is deprecated. It did not have a --size filter as users had to use the --query feature of Azure CLI to filter the output (which is just fine but maybe not as discoverable for those who are new to Azure CLI and/or json/jsonpath queries.
Errors
In failure cases, it just sometimes hangs forever.
I have also seen it output this error (sometimes multiple times) and then either hang or eventually crash:
Traceback (most recent call last):
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\asyncio\windows_events.py", line 445, in select
RuntimeError: <_overlapped.Overlapped object at 0x0000017218925FB0> still has pending operation at deallocation, the process may crash
Issue script & Debug output
az vm list-skus --location westus2 --resource-type virtualMachines --size Standard_D8plds_v5 --debug
The output is far too large to show here.
However, here is the output up to the point of the big data dump:
DEBUG: cli.knack.cli: Command arguments: ['vm', 'list-skus', '--location', 'westus2', '--resource-type', 'virtualMachines', '--size', 'Standard_D8plds_v5', '--debug']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f3f78633880>, <function OutputProducer.on_global_arguments at 0x7f3f7837e840>, <function CLIQuery.on_global_arguments at 0x7f3f783c3d80>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'vm': ['azure.cli.command_modules.vm']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name Load Time Groups Commands
DEBUG: cli.azure.cli.core: vm 0.375 59 228
DEBUG: cli.azure.cli.core: Total (1) 0.375 59 228
DEBUG: cli.azure.cli.core: Loaded 59 groups, 228 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command : vm list-skus
DEBUG: cli.azure.cli.core: Command table: vm list
DEBUG: cli.azure.cli.core: remaining : skus
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f3f77608040>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/msinz/.azure/commands/2025-06-03.15-10-08.vm_list-skus.1660582.log'.
INFO: az_command_data_logger: command args: vm list-skus --location {} --resource-type {} --size {} --debug
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f3f7765c540>]
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "/opt/az/lib/python3.12/site-packages/azure/cli/core/profiles/_shared.py", line 471, in _get_attr
op = import_module(full_mod_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.mgmt.compute.v2024_03_03'
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f3f7765ed40>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f3f7765ee80>, <function register_upcoming_breaking_change_info.<locals>.update_breaking_change_info at 0x7f3f7765ef20>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f3f7837e8e0>, <function CLIQuery.handle_query_parameter at 0x7f3f783c3e20>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f3f7765ede0>]
DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
DEBUG: cli.azure.cli.core.auth.msal_credentials: ManagedIdentityCredential.acquire_token: scopes=['https://management.core.windows.net//.default'], kwargs={}
DEBUG: msal.managed_identity: Obtaining token via managed identity on Azure VM
DEBUG: urllib3.connectionpool: Starting new HTTP connection (1): 169.254.169.254:80
DEBUG: urllib3.connectionpool: http://169.254.169.254:80 "GET /metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.core.windows.net%2F&msi_res_id=%2Fsubscriptions%2F******** HTTP/1.1" 200 2376
DEBUG: msal.token_cache: event={
"client_id": "/subscriptions/********",
"data": {},
"params": {},
"response": {
"access_token": "********",
"expires_in": 83894,
"refresh_in": 41947,
"resource": "https://management.core.windows.net/",
"token_type": "Bearer"
},
"scope": [
"https://management.core.windows.net/"
],
"token_endpoint": "https://********"
}
DEBUG: cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/********/providers/Microsoft.Compute/skus?api-version=2019-04-01'
DEBUG: cli.azure.cli.core.sdk.policies: Request method: 'GET'
DEBUG: cli.azure.cli.core.sdk.policies: Request headers:
DEBUG: cli.azure.cli.core.sdk.policies: 'Accept': 'application/json'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': 'd60a353a-408c-11f0-b457-f7c6ba39c517'
DEBUG: cli.azure.cli.core.sdk.policies: 'CommandName': 'vm list-skus'
DEBUG: cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--location --resource-type --size --debug'
DEBUG: cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.73.0 (DEB) azsdk-python-core/1.31.0 Python/3.12.8 (Linux-5.15.0-1074-azure-x86_64-with-glibc2.35)'
DEBUG: cli.azure.cli.core.sdk.policies: 'Authorization': '*****'
DEBUG: cli.azure.cli.core.sdk.policies: Request body:
DEBUG: cli.azure.cli.core.sdk.policies: This request has no body
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/********/providers/Microsoft.Compute/skus?api-version=2019-04-01 HTTP/1.1" 200 128930075
DEBUG: cli.azure.cli.core.sdk.policies: Response status: 200
DEBUG: cli.azure.cli.core.sdk.policies: Response headers:
DEBUG: cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache'
DEBUG: cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache'
DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Length': '128930075'
DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8'
DEBUG: cli.azure.cli.core.sdk.policies: 'Expires': '-1'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-reads': '1099'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-ratelimit-remaining-subscription-global-reads': '16499'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '16dba9dd-8966-460c-988d-ea180de34047'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '16dba9dd-8966-460c-988d-ea180de34047'
DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'EASTUS:20250603T151011Z:16dba9dd-8966-460c-988d-ea180de34047'
DEBUG: cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
DEBUG: cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff'
DEBUG: cli.azure.cli.core.sdk.policies: 'X-Cache': 'CONFIG_NOCACHE'
DEBUG: cli.azure.cli.core.sdk.policies: 'X-MSEdge-Ref': 'Ref A: 756939C06DF345D8AF0C87D4422DB442 Ref B: MWH011020808023 Ref C: 2025-06-03T15:10:08Z'
DEBUG: cli.azure.cli.core.sdk.policies: 'Date': 'Tue, 03 Jun 2025 15:10:16 GMT'
DEBUG: cli.azure.cli.core.sdk.policies: Response content:
... then after the big content dump ...
DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x7f3f7765c2c0>, <function _x509_from_base64_to_hex_transform at 0x7
f3f7765c360>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f3f776082c0>]
INFO: az_command_data_logger: exit code: 0
INFO: cli.__main__: Command ran in 177.010 seconds (init: 0.201, invoke: 176.809)
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3951 in cache file under /home/msinz/.azure/telemetry/20250603151304811
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.12/site-packages/azure/cli/telemetry/__init__.py /home/msinz/.azure /home/msinz
/.azure/telemetry/20250603151304811"
INFO: telemetry.process: Return from creating process 1661287
INFO: telemetry.main: Finish creating telemetry upload process.
... and now the SKU's data (note, not from the same run as I had lost the end bit and had to rerun...)
cli.knack.cli: Event: Cli.SuccessfulExecute []
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fe92b4042c0>]
az_command_data_logger: exit code: 0
cli.__main__: Command ran in 152.182 seconds (init: 0.171, invoke: 152.011)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3951 in cache file under /home/msinz/.azure/telemetry/20250603152451814
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.12/site-packages/azure/cli/telemetry/__init__.py /home/msinz/.azure /home/msinz/.azure/telemetry/20250603152451814"
telemetry.process: Return from creating process 1664358
telemetry.main: Finish creating telemetry upload process.
Expected behavior
A relatively quick return of the data for the VM (with --query operation on it or not)
This would allow for actually validating the SKU selected by the user and the parameters needed to construct it with the correct OS variations (CPU architectures, GPU support, accelerated networking, etc
Environment Summary
I have different installations:
On Linux:
$ az --version
azure-cli 2.73.0 *
core 2.73.0 *
telemetry 1.1.0
Dependencies:
msal 1.32.3
azure-mgmt-resource 23.3.0
Python location '/opt/az/bin/python3'
Config directory '/home/msinz/.azure'
Extensions directory '/home/msinz/.azure/cliextensions'
Python (Linux) 3.12.8 (main, May 13 2025, 11:04:48) [GCC 11.4.0]
On secure access Windows machine:
azure-cli 2.55.0
core 2.55.0
telemetry 1.1.0
Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2
Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\msinz.AME\.azure\cliextensions'
Python (Windows) 3.11.7 (remotes/origin/3.11.7/msft-spython:be2aebf208, Dec 11 2023, 22:59:37) [MSC v.1937 64 bit (AMD64)]
I have seen the crashes happen much more often on the Windows machine than Linux but the performance is equally as bad either place.
Additional context
No response