calc-x example fails on next #498
Description
2026-02-27T19:28:35.0128090Z �[36m(TaskRunner pid=7250)�[0m The above exception was the direct cause of the following exception:
2026-02-27T19:28:35.0128599Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0128990Z �[36m(TaskRunner pid=7250)�[0m Traceback (most recent call last):
2026-02-27T19:28:35.0129903Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/openai/_base_client.py", line 1604, in request
2026-02-27T19:28:35.0130773Z �[36m(TaskRunner pid=7250)�[0m response = await self._client.send(
2026-02-27T19:28:35.0131268Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0131882Z �[36m(TaskRunner pid=7250)�[0m ...<3 lines>...
2026-02-27T19:28:35.0132226Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0132529Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0133299Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send
2026-02-27T19:28:35.0134157Z �[36m(TaskRunner pid=7250)�[0m response = await self._send_handling_auth(
2026-02-27T19:28:35.0134874Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0135322Z �[36m(TaskRunner pid=7250)�[0m ...<4 lines>...
2026-02-27T19:28:35.0135660Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0135959Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0136860Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
2026-02-27T19:28:35.0137842Z �[36m(TaskRunner pid=7250)�[0m response = await self._send_handling_redirects(
2026-02-27T19:28:35.0138479Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0139027Z �[36m(TaskRunner pid=7250)�[0m ...<3 lines>...
2026-02-27T19:28:35.0139441Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0140052Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0140974Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
2026-02-27T19:28:35.0142294Z �[36m(TaskRunner pid=7250)�[0m response = await self._send_single_request(request)
2026-02-27T19:28:35.0142913Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0144052Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request
2026-02-27T19:28:35.0145249Z �[36m(TaskRunner pid=7250)�[0m response = await transport.handle_async_request(request)
2026-02-27T19:28:35.0145969Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0147192Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/custom_httpx/aiohttp_transport.py", line 272, in handle_async_request
2026-02-27T19:28:35.0148324Z �[36m(TaskRunner pid=7250)�[0m with map_aiohttp_exceptions():
2026-02-27T19:28:35.0148944Z �[36m(TaskRunner pid=7250)�[0m ~~~~~~~~~~~~~~~~~~~~~~^^
2026-02-27T19:28:35.0149885Z �[36m(TaskRunner pid=7250)�[0m File "/home/cloudtest/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in exit
2026-02-27T19:28:35.0150764Z �[36m(TaskRunner pid=7250)�[0m self.gen.throw(value)
2026-02-27T19:28:35.0151174Z �[36m(TaskRunner pid=7250)�[0m ~~~~~~~~~~~~~~^^^^^^^
2026-02-27T19:28:35.0152980Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/custom_httpx/aiohttp_transport.py", line 74, in map_aiohttp_exceptions
2026-02-27T19:28:35.0154190Z �[36m(TaskRunner pid=7250)�[0m raise mapped_exc(message) from exc
2026-02-27T19:28:35.0155122Z �[36m(TaskRunner pid=7250)�[0m httpx.ConnectError: Cannot connect to host 10.1.0.15:35035 ssl:default [Connect call failed ('10.1.0.15', 35035)]
2026-02-27T19:28:35.0155785Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0156302Z �[36m(TaskRunner pid=7250)�[0m The above exception was the direct cause of the following exception:
2026-02-27T19:28:35.0156815Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0157200Z �[36m(TaskRunner pid=7250)�[0m Traceback (most recent call last):
2026-02-27T19:28:35.0158203Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 840, in acompletion
2026-02-27T19:28:35.0159320Z �[36m(TaskRunner pid=7250)�[0m headers, response = await self.make_openai_chat_completion_request(
2026-02-27T19:28:35.0160172Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0160930Z �[36m(TaskRunner pid=7250)�[0m ...<4 lines>...
2026-02-27T19:28:35.0161601Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0161931Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0163380Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/litellm_core_utils/logging_utils.py", line 190, in async_wrapper
2026-02-27T19:28:35.0164413Z �[36m(TaskRunner pid=7250)�[0m result = await func(*args, **kwargs)
2026-02-27T19:28:35.0164995Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0166070Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 460, in make_openai_chat_completion_request
2026-02-27T19:28:35.0166988Z �[36m(TaskRunner pid=7250)�[0m raise e
2026-02-27T19:28:35.0168116Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 437, in make_openai_chat_completion_request
2026-02-27T19:28:35.0169522Z �[36m(TaskRunner pid=7250)�[0m await openai_aclient.chat.completions.with_raw_response.create(
2026-02-27T19:28:35.0170125Z �[36m(TaskRunner pid=7250)�[0m **data, timeout=timeout
2026-02-27T19:28:35.0170510Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0171531Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/openai/_legacy_response.py", line 384, in wrapped
2026-02-27T19:28:35.0172519Z �[36m(TaskRunner pid=7250)�[0m return cast(LegacyAPIResponse[R], await func(*args, **kwargs))
2026-02-27T19:28:35.0173159Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0174230Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/openai/resources/chat/completions/completions.py", line 2700, in create
2026-02-27T19:28:35.0175163Z �[36m(TaskRunner pid=7250)�[0m return await self._post(
2026-02-27T19:28:35.0175597Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0176000Z �[36m(TaskRunner pid=7250)�[0m ...<49 lines>...
2026-02-27T19:28:35.0176351Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0176702Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0177596Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/openai/_base_client.py", line 1884, in post
2026-02-27T19:28:35.0178830Z �[36m(TaskRunner pid=7250)�[0m return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
2026-02-27T19:28:35.0182034Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0183203Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/openai/_base_client.py", line 1636, in request
2026-02-27T19:28:35.0184866Z �[36m(TaskRunner pid=7250)�[0m raise APIConnectionError(request=request) from err
2026-02-27T19:28:35.0185576Z �[36m(TaskRunner pid=7250)�[0m openai.APIConnectionError: Connection error.
2026-02-27T19:28:35.0186140Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0186739Z �[36m(TaskRunner pid=7250)�[0m During handling of the above exception, another exception occurred:
2026-02-27T19:28:35.0187311Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0187583Z �[36m(TaskRunner pid=7250)�[0m Traceback (most recent call last):
2026-02-27T19:28:35.0188321Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/main.py", line 610, in acompletion
2026-02-27T19:28:35.0188571Z �[36m(TaskRunner pid=7250)�[0m response = await init_response
2026-02-27T19:28:35.0188822Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0189908Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/openai/openai.py", line 887, in acompletion
2026-02-27T19:28:35.0190159Z �[36m(TaskRunner pid=7250)�[0m raise OpenAIError(
2026-02-27T19:28:35.0190369Z �[36m(TaskRunner pid=7250)�[0m ...<4 lines>...
2026-02-27T19:28:35.0190532Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0190954Z �[36m(TaskRunner pid=7250)�[0m litellm.llms.openai.common_utils.OpenAIError: Connection error.
2026-02-27T19:28:35.0191113Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0191738Z �[36m(TaskRunner pid=7250)�[0m During handling of the above exception, another exception occurred:
2026-02-27T19:28:35.0191867Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0192110Z �[36m(TaskRunner pid=7250)�[0m Traceback (most recent call last):
2026-02-27T19:28:35.0192964Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/proxy/proxy_server.py", line 5247, in chat_completion
2026-02-27T19:28:35.0193628Z �[36m(TaskRunner pid=7250)�[0m result = await base_llm_response_processor.base_process_llm_request(
2026-02-27T19:28:35.0194007Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0194211Z �[36m(TaskRunner pid=7250)�[0m ...<16 lines>...
2026-02-27T19:28:35.0194364Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0194513Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0195464Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/proxy/common_request_processing.py", line 680, in base_process_llm_request
2026-02-27T19:28:35.0195716Z �[36m(TaskRunner pid=7250)�[0m responses = await llm_responses
2026-02-27T19:28:35.0195960Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0196704Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/router.py", line 1374, in acompletion
2026-02-27T19:28:35.0196852Z �[36m(TaskRunner pid=7250)�[0m raise e
2026-02-27T19:28:35.0197592Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/router.py", line 1350, in acompletion
2026-02-27T19:28:35.0197980Z �[36m(TaskRunner pid=7250)�[0m response = await self.async_function_with_fallbacks(**kwargs)
2026-02-27T19:28:35.0198273Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0199140Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/router.py", line 4571, in async_function_with_fallbacks
2026-02-27T19:28:35.0199558Z �[36m(TaskRunner pid=7250)�[0m return await self.async_function_with_fallbacks_common_utils(
2026-02-27T19:28:35.0199885Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0200095Z �[36m(TaskRunner pid=7250)�[0m ...<8 lines>...
2026-02-27T19:28:35.0200248Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0200419Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0201566Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/router.py", line 4529, in async_function_with_fallbacks_common_utils
2026-02-27T19:28:35.0201854Z �[36m(TaskRunner pid=7250)�[0m raise original_exception
2026-02-27T19:28:35.0202791Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/router.py", line 4563, in async_function_with_fallbacks
2026-02-27T19:28:35.0203218Z �[36m(TaskRunner pid=7250)�[0m response = await self.async_function_with_retries(*args, **kwargs)
2026-02-27T19:28:35.0203549Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0204567Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/router.py", line 4659, in async_function_with_retries
2026-02-27T19:28:35.0204972Z �[36m(TaskRunner pid=7250)�[0m response = await self.make_call(original_function, *args, **kwargs)
2026-02-27T19:28:35.0205284Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0205955Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/router.py", line 4787, in make_call
2026-02-27T19:28:35.0206200Z �[36m(TaskRunner pid=7250)�[0m response = await response
2026-02-27T19:28:35.0206409Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^
2026-02-27T19:28:35.0207351Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/router.py", line 1660, in _acompletion
2026-02-27T19:28:35.0207510Z �[36m(TaskRunner pid=7250)�[0m raise e
2026-02-27T19:28:35.0208199Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/router.py", line 1606, in _acompletion
2026-02-27T19:28:35.0208426Z �[36m(TaskRunner pid=7250)�[0m response = await _response
2026-02-27T19:28:35.0208690Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0209424Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/utils.py", line 1970, in wrapper_async
2026-02-27T19:28:35.0209571Z �[36m(TaskRunner pid=7250)�[0m raise e
2026-02-27T19:28:35.0210282Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/utils.py", line 1790, in wrapper_async
2026-02-27T19:28:35.0210625Z �[36m(TaskRunner pid=7250)�[0m result = await original_function(*args, **kwargs)
2026-02-27T19:28:35.0210942Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0211886Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/main.py", line 629, in acompletion
2026-02-27T19:28:35.0212125Z �[36m(TaskRunner pid=7250)�[0m raise exception_type(
2026-02-27T19:28:35.0212347Z �[36m(TaskRunner pid=7250)�[0m ~~~~~~~~~~~~~~^
2026-02-27T19:28:35.0212534Z �[36m(TaskRunner pid=7250)�[0m model=model,
2026-02-27T19:28:35.0212751Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^
2026-02-27T19:28:35.0212942Z �[36m(TaskRunner pid=7250)�[0m ...<3 lines>...
2026-02-27T19:28:35.0213175Z �[36m(TaskRunner pid=7250)�[0m extra_kwargs=kwargs,
2026-02-27T19:28:35.0213405Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0213558Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0213688Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0214582Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 2356, in exception_type
2026-02-27T19:28:35.0214728Z �[36m(TaskRunner pid=7250)�[0m raise e
2026-02-27T19:28:35.0215573Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/litellm_core_utils/exception_mapping_utils.py", line 547, in exception_type
2026-02-27T19:28:35.0215799Z �[36m(TaskRunner pid=7250)�[0m raise InternalServerError(
2026-02-27T19:28:35.0215980Z �[36m(TaskRunner pid=7250)�[0m ...<5 lines>...
2026-02-27T19:28:35.0216136Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0217128Z �[36m(TaskRunner pid=7250)�[0m litellm.exceptions.InternalServerError: litellm.InternalServerError: InternalServerError: Hosted_vllmException - Connection error.. Received Model Group=Qwen/Qwen2.5-1.5B-Instruct
2026-02-27T19:28:35.0217544Z �[36m(TaskRunner pid=7250)�[0m Available Model Group Fallbacks=None
2026-02-27T19:28:35.0218758Z �[36m(TaskRunner pid=7250)�[0m ERROR:2026-02-27 19:28:34,983:litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.InternalServerError: InternalServerError: Hosted_vllmException - Connection error.. Received Model Group=Qwen/Qwen2.5-1.5B-Instruct
2026-02-27T19:28:35.0218994Z �[36m(TaskRunner pid=7250)�[0m Available Model Group Fallbacks=None
2026-02-27T19:28:35.0219240Z �[36m(TaskRunner pid=7250)�[0m Traceback (most recent call last):
2026-02-27T19:28:35.0220085Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1298, in _wrap_create_connection
2026-02-27T19:28:35.0220544Z �[36m(TaskRunner pid=7250)�[0m sock = await aiohappyeyeballs.start_connection(
2026-02-27T19:28:35.0220817Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0221003Z �[36m(TaskRunner pid=7250)�[0m ...<6 lines>...
2026-02-27T19:28:35.0221154Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0221460Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0222227Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohappyeyeballs/impl.py", line 122, in start_connection
2026-02-27T19:28:35.0222431Z �[36m(TaskRunner pid=7250)�[0m raise first_exception
2026-02-27T19:28:35.0223158Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohappyeyeballs/impl.py", line 73, in start_connection
2026-02-27T19:28:35.0223373Z �[36m(TaskRunner pid=7250)�[0m sock = await _connect_sock(
2026-02-27T19:28:35.0223588Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0223750Z �[36m(TaskRunner pid=7250)�[0m ...<6 lines>...
2026-02-27T19:28:35.0223889Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0224043Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0224749Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohappyeyeballs/impl.py", line 208, in _connect_sock
2026-02-27T19:28:35.0225067Z �[36m(TaskRunner pid=7250)�[0m await loop.sock_connect(sock, address)
2026-02-27T19:28:35.0225420Z �[36m(TaskRunner pid=7250)�[0m File "uvloop/loop.pyx", line 2633, in sock_connect
2026-02-27T19:28:35.0225601Z �[36m(TaskRunner pid=7250)�[0m await fut
2026-02-27T19:28:35.0225970Z �[36m(TaskRunner pid=7250)�[0m File "uvloop/loop.pyx", line 1108, in uvloop.loop.Loop._sock_connect_cb
2026-02-27T19:28:35.0226295Z �[36m(TaskRunner pid=7250)�[0m raise OSError(err, 'Connect call failed %s' % (address,))
2026-02-27T19:28:35.0226704Z �[36m(TaskRunner pid=7250)�[0m ConnectionRefusedError: [Errno 111] Connect call failed ('10.1.0.15', 35035)
2026-02-27T19:28:35.0226839Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0227208Z �[36m(TaskRunner pid=7250)�[0m The above exception was the direct cause of the following exception:
2026-02-27T19:28:35.0227354Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0227579Z �[36m(TaskRunner pid=7250)�[0m Traceback (most recent call last):
2026-02-27T19:28:35.0228414Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/custom_httpx/aiohttp_transport.py", line 60, in map_aiohttp_exceptions
2026-02-27T19:28:35.0228558Z �[36m(TaskRunner pid=7250)�[0m yield
2026-02-27T19:28:35.0229367Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/custom_httpx/aiohttp_transport.py", line 273, in handle_async_request
2026-02-27T19:28:35.0229717Z �[36m(TaskRunner pid=7250)�[0m response = await self._make_aiohttp_request(
2026-02-27T19:28:35.0230162Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0230369Z �[36m(TaskRunner pid=7250)�[0m ...<5 lines>...
2026-02-27T19:28:35.0230541Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0230684Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0231669Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/custom_httpx/aiohttp_transport.py", line 240, in _make_aiohttp_request
2026-02-27T19:28:35.0231938Z �[36m(TaskRunner pid=7250)�[0m response = await client_session.request(
2026-02-27T19:28:35.0232202Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0232375Z �[36m(TaskRunner pid=7250)�[0m ...<13 lines>...
2026-02-27T19:28:35.0232532Z �[36m(TaskRunner pid=7250)�[0m ).aenter()
2026-02-27T19:28:35.0232858Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^
2026-02-27T19:28:35.0233563Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 1510, in aenter
2026-02-27T19:28:35.0233835Z �[36m(TaskRunner pid=7250)�[0m self._resp: _RetType = await self._coro
2026-02-27T19:28:35.0234122Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0234750Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 779, in _request
2026-02-27T19:28:35.0234974Z �[36m(TaskRunner pid=7250)�[0m resp = await handler(req)
2026-02-27T19:28:35.0235184Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0235918Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohttp/client.py", line 734, in _connect_and_send_request
2026-02-27T19:28:35.0236208Z �[36m(TaskRunner pid=7250)�[0m conn = await self._connector.connect(
2026-02-27T19:28:35.0236432Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0236689Z �[36m(TaskRunner pid=7250)�[0m req, traces=traces, timeout=real_timeout
2026-02-27T19:28:35.0236923Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0237050Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0237199Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0237819Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 672, in connect
2026-02-27T19:28:35.0238159Z �[36m(TaskRunner pid=7250)�[0m proto = await self._create_connection(req, traces, timeout)
2026-02-27T19:28:35.0238442Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0239239Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1239, in _create_connection
2026-02-27T19:28:35.0239653Z �[36m(TaskRunner pid=7250)�[0m _, proto = await self._create_direct_connection(req, traces, timeout)
2026-02-27T19:28:35.0239968Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0240802Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1611, in _create_direct_connection
2026-02-27T19:28:35.0240971Z �[36m(TaskRunner pid=7250)�[0m raise last_exc
2026-02-27T19:28:35.0241909Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1580, in _create_direct_connection
2026-02-27T19:28:35.0242233Z �[36m(TaskRunner pid=7250)�[0m transp, proto = await self._wrap_create_connection(
2026-02-27T19:28:35.0242475Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0242801Z �[36m(TaskRunner pid=7250)�[0m ...<7 lines>...
2026-02-27T19:28:35.0242940Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0243066Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0243810Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/aiohttp/connector.py", line 1321, in _wrap_create_connection
2026-02-27T19:28:35.0244130Z �[36m(TaskRunner pid=7250)�[0m raise client_error(req.connection_key, exc) from exc
2026-02-27T19:28:35.0244870Z �[36m(TaskRunner pid=7250)�[0m aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 10.1.0.15:35035 ssl:default [Connect call failed ('10.1.0.15', 35035)]
2026-02-27T19:28:35.0245021Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0245550Z �[36m(TaskRunner pid=7250)�[0m The above exception was the direct cause of the following exception:
2026-02-27T19:28:35.0245699Z �[36m(TaskRunner pid=7250)�[0m
2026-02-27T19:28:35.0245986Z �[36m(TaskRunner pid=7250)�[0m Traceback (most recent call last):
2026-02-27T19:28:35.0246764Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/openai/_base_client.py", line 1604, in request
2026-02-27T19:28:35.0247040Z �[36m(TaskRunner pid=7250)�[0m response = await self._client.send(
2026-02-27T19:28:35.0247297Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0247479Z �[36m(TaskRunner pid=7250)�[0m ...<3 lines>...
2026-02-27T19:28:35.0247608Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0247738Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0248353Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1629, in send
2026-02-27T19:28:35.0248636Z �[36m(TaskRunner pid=7250)�[0m response = await self._send_handling_auth(
2026-02-27T19:28:35.0248895Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0249089Z �[36m(TaskRunner pid=7250)�[0m ...<4 lines>...
2026-02-27T19:28:35.0249239Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0249386Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0250133Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
2026-02-27T19:28:35.0250452Z �[36m(TaskRunner pid=7250)�[0m response = await self._send_handling_redirects(
2026-02-27T19:28:35.0250705Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0250911Z �[36m(TaskRunner pid=7250)�[0m ...<3 lines>...
2026-02-27T19:28:35.0251069Z �[36m(TaskRunner pid=7250)�[0m )
2026-02-27T19:28:35.0251234Z �[36m(TaskRunner pid=7250)�[0m ^
2026-02-27T19:28:35.0252199Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
2026-02-27T19:28:35.0252534Z �[36m(TaskRunner pid=7250)�[0m response = await self._send_single_request(request)
2026-02-27T19:28:35.0252802Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0253553Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/httpx/_client.py", line 1730, in _send_single_request
2026-02-27T19:28:35.0253917Z �[36m(TaskRunner pid=7250)�[0m response = await transport.handle_async_request(request)
2026-02-27T19:28:35.0254210Z �[36m(TaskRunner pid=7250)�[0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-27T19:28:35.0255211Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/custom_httpx/aiohttp_transport.py", line 272, in handle_async_request
2026-02-27T19:28:35.0255644Z �[36m(TaskRunner pid=7250)�[0m with map_aiohttp_exceptions():
2026-02-27T19:28:35.0255905Z �[36m(TaskRunner pid=7250)�[0m ~~~~~~~~~~~~~~~~~~~~~~^^
2026-02-27T19:28:35.0256638Z �[36m(TaskRunner pid=7250)�[0m File "/home/cloudtest/.local/share/uv/python/cpython-3.13.12-linux-x86_64-gnu/lib/python3.13/contextlib.py", line 162, in exit
2026-02-27T19:28:35.0256895Z �[36m(TaskRunner pid=7250)�[0m self.gen.throw(value)
2026-02-27T19:28:35.0257156Z �[36m(TaskRunner pid=7250)�[0m ~~~~~~~~~~~~~~^^^^^^^
2026-02-27T19:28:35.0258258Z �[36m(TaskRunner pid=7250)�[0m File "/mnt/vss/_work/agent-lightning/agent-lightning/.venv/lib/python3.13/site-packages/litellm/llms/custom_httpx/aiohttp_transport.py", line 74, in map_aiohttp_exceptions
2026-02-27T19:28:35.0258532Z �[36m(TaskRunner pid=7250)�[0m raise mapped_exc(message) from exc
2026-02-27T19:28:35.0259385Z �[36m(TaskRunner pid=7250)�[0m httpx.ConnectError: Cannot connect to host 10.1.0.15:35035 ssl:default [Connect call failed ('10.1.0.15', 35035)]
https://github.com/microsoft/agent-lightning/actions/runs/22500287144/job/65185310731
When fixing, try to find the latest run that succeeded and identify which dependencies have upgraded. then find the root cause: whether to update or pin dependency.