AzureDeveloperCliCredential.get_token timed out with Python 3.12 on Ubuntu

[hanohrs]

I know that this project only supports Python 3.9, 3.10, or 3.11, but I think it's worth sharing.

When I run scripts/prepdocs.sh with Python 3.12, I get the following error. (USE_LOCAL_PDF_PARSER="true")

[17:13:50] WARNING  AzureDeveloperCliCredential.get_token failed: Timed out waiting for Azure Developer CLI                                                                decorators.py:41
Traceback (most recent call last):
  File "/usr/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/subprocess.py", line 202, in communicate
    await self.wait()
  File "/usr/lib/python3.12/asyncio/subprocess.py", line 137, in wait
    return await self._transport._wait()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_subprocess.py", line 230, in _wait
    return await waiter
           ^^^^^^^^^^^^
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/identity/aio/_credentials/azd_cli.py", line 171, in _run_command
    stdout_b, stderr_b = await asyncio.wait_for(proc.communicate(), timeout)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/tasks.py", line 519, in wait_for
    async with timeouts.timeout(timeout):
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/timeouts.py", line 115, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/hiro/repos/azure-search-openai-demo/./app/backend/prepdocs.py", line 439, in <module>
    loop.run_until_complete(main(ingestion_strategy, setup_index=not args.remove and not args.removeall))
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/./app/backend/prepdocs.py", line 244, in main
    await strategy.run()
  File "/home/hiro/repos/azure-search-openai-demo/app/backend/prepdocslib/filestrategy.py", line 103, in run
    blob_sas_uris = await self.blob_manager.upload_blob(file)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/app/backend/prepdocslib/blobmanager.py", line 52, in upload_blob
    if not await container_client.exists():
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/tracing/decorator_async.py", line 94, in wrapper_use_tracer
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/storage/blob/aio/_container_client_async.py", line 547, in exists
    process_storage_error(error)
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/storage/blob/_shared/response_handlers.py", line 92, in process_storage_error
    raise storage_error
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/storage/blob/aio/_container_client_async.py", line 543, in exists
    await self._client.container.get_properties(**kwargs)
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/tracing/decorator_async.py", line 94, in wrapper_use_tracer
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/storage/blob/_generated/aio/operations/_container_operations.py", line 234, in get_properties
    pipeline_response: PipelineResponse = await self._client._pipeline.run(  # pylint: disable=protected-access
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/pipeline/_base_async.py", line 219, in run
    return await first_node.send(pipeline_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/pipeline/_base_async.py", line 68, in send
    response = await self.next.send(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/pipeline/_base_async.py", line 68, in send
    response = await self.next.send(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/pipeline/_base_async.py", line 68, in send
    response = await self.next.send(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [Previous line repeated 3 more times]
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/pipeline/policies/_authentication_async.py", line 98, in send
    await await_result(self.on_request, request)
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/pipeline/_tools_async.py", line 56, in await_result
    return await result
           ^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/pipeline/policies/_authentication_async.py", line 70, in on_request
    self._token = await await_result(self._credential.get_token, *self._scopes)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/core/pipeline/_tools_async.py", line 56, in await_result
    return await result
           ^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/identity/aio/_internal/decorators.py", line 21, in wrapper
    token = await fn(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/identity/aio/_credentials/azd_cli.py", line 130, in get_token
    output = await _run_command(command, self._process_timeout)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/repos/azure-search-openai-demo/.venv/lib/python3.12/site-packages/azure/identity/aio/_credentials/azd_cli.py", line 176, in _run_command
    raise CredentialUnavailableError(message="Timed out waiting for Azure Developer CLI") from ex
azure.identity._exceptions.CredentialUnavailableError: Timed out waiting for Azure Developer CLI

After I built Python 3.11 from the source and modify scripts/load_python_env.sh to use Python 3.11, it works fine.

Modifying .venv/lib/python3.12/site-packages/azure/identity/aio/_credentials/azd_cli.py to return _SyncAzureDeveloperCliCredential from get_token also fixes this issue.

[pamelafox]

Interesting, I've only experienced this issue when my household was running a DNS server that slowed down the token request. That was resolved, however (Azure/azure-dev#4410).

I've been using Python 3.12 on Mac and the token has been working well. Can you file this in https://github.com/Azure/azure-dev/issues ?

The only thing we can do in this repo is extend the timeout, but we already have process_timeout=60 in most places.

[pvaneck]

Hey @hanohrs, I'm unsure why the asynchronous AzureDeveloperCliCredential doesn't seem to work for you on Python 3.12. I am unable to reproduce on my Ubuntu environment. To help with debugging:

1. Can you share which version of Ubuntu you are using?
2. Since it works fine synchronously, running azd directly likely runs fine, but can you double-check that running the following in the shell yields a token: /bin/sh -c 'azd auth token --output json --scope https://management.azure.com/.default'
3. Can you try the following on your system with Python 3.12:

import asyncio
import logging

from azure.identity import AzureDeveloperCliCredential
from azure.identity.aio import AzureDeveloperCliCredential as AsyncAzureDeveloperCliCredential

logging.basicConfig(level=logging.DEBUG)

def sync_test():
    credential = AzureDeveloperCliCredential()
    token = credential.get_token("https://management.azure.com/.default")
    print(f"Sync Token: {token}\n")

async def async_test():
    credential = AsyncAzureDeveloperCliCredential()
    async with credential:
        token = await credential.get_token("https://management.azure.com/.default")
        print(f"Async Token: {token}")

if __name__ == "__main__":
    sync_test()
    asyncio.run(async_test())

If the above Python script fails the async method, please open an issue here: https://github.com/Azure/azure-sdk-for-python/issues. There could be something strange going on with asyncio in Python 3.12 on your system, but from looking through the Python release notes, I don't see anything obvious that might cause this.

[hanohrs]

Thank you for the replies.

@pvaneck
Thank you for the detailed reply with a script.
It's very interesting that your environment works fine.

1. I tried on several environments. Mostly on my local Ubuntu 24.10 Server on VMware Workstation. Python is 3.12.7.
2. the command works fine
3. the script works fine

@pamelafox
I'm quite unsure of where should I consult to.
Is this a bug of this project, azure-sdk-for-python, azd, or cpython?
I posted here because I couldn't reproduce this issue without the code here.

---

I successfully reproduced this error in a docker environment with Docker Desktop on Windows 11.

docker run -it --name ubuntu-24.04-asod-repro ubuntu:noble-20250127

After the command above, I ran the following commands.

Some of them asked questions and I replied. For azd provision, I used eastus for Document Intelligence, and eastus2 for other purposes.

apt-get update && apt-get install -y curl python3-venv python-is-python3 nodejs npm git vim
curl -fsSL https://aka.ms/install-azd.sh | bash
cd /root
git clone -b 2025-02-13 https://github.com/Azure-Samples/azure-search-openai-demo.git
cd azure-search-openai-demo/
azd auth login --use-device-code
azd env new
sed -i -e 's/host: containerapp/# host: containerapp/' -e 's/# host: appservice/host: appservice/' azure.yaml
azd env set DEPLOYMENT_TARGET appservice
azd env set AZURE_APP_SERVICE_SKU F1
azd env set AZURE_DOCUMENTINTELLIGENCE_SKU F0
azd env set USE_LOCAL_PDF_PARSER true
azd env set USE_LOCAL_HTML_PARSER true
azd provision

After the failure, I ran the following.

find ./data -name "*.md5" -print0 | xargs -0 rm
sed -i '/loop.run_until_complete/i\    asyncio.set_child_watcher(asyncio.ThreadedChildWatcher())' app/backend/prepdocs.py
scripts/prepdocs.sh

This time, everything works fine. ThreadedChildWatcher is the default on Python 3.11.

Changing this line to:

stdout_b, stderr_b = await proc.communicate()

causes the process to hang. The parent process is stuck at subprocess.py#L202 of cpython and the child process is a zombie. I don't understand why this can happen for now.

[hanohrs]

Looks like a cpython issue. The fix made in the first post of the following thread fixes this problem.
python/cpython#130141

@pamelafox
It's not an azd issue. Please remove the azd label.

You may need to work around this problem by using the deprecated asyncio.set_child_watcher(asyncio.ThreadedChildWatcher()) if you want to support the current or older minor releases of Python 3.12 or 3.13 on Linux.

[pamelafox]

Thanks for following up and investigating so thoroughly!

[cforce]

@hanohrs Same issue?
#2586

[hanohrs]

@cforce
I'm not sure if it's the same issue, but if it is, using the ThreadedChildWatcher should fix it.

[cforce]

This time the issue is not in prepdocs.py but in app/backend/app.py - see call stack #2586

[hanohrs]

@cforce
Hypercorn can be used programatically. You can run with the ThreadedChildWatcher.
https://github.com/pgjones/hypercorn?tab=readme-ov-file#quickstart

[cforce]

Oh my, still a bit lost . Is there a less hacking solution. We are at a dead end here, aren’t we. Python 11 is no continuous solution- btw what about 14.x -same issue?

[hanohrs]

Since set_child_watcher will be removed in Python 3.14, fixing the related CPython bug would be a continuous solution.
I'm not currently working on RAGs, so I don't plan to contribute to this for now.