New behavior in kopf 1.37.2 causes kopf to crash with "Ran out of valid credentials" #1158
Description
Long story short
Regarding the new behavior of kopf/_cogs/clients/auth.py as it relates to kopf/_cogs/structs/credentials.py expire: the service account token expiration causes kopf to crash. Our application runs with 1.37.1, but crashes after an hour when running 1.37.2.
Existing observation/queueing/watching/api objects stack after an hour with "Ran out of valid credentials" see logs below.
I assume this is related to the code changes in auth.py which is explicitly closing all responses (aiohttp.ClientResponse) prior to re-athentication occurring, or being called. This is really the only code change between 1.37.1 and 1.37.2
Kopf version
1.37.2
Kubernetes version
1.30 AKS
Python version
3.12.8
Code
Logs
[2025-01-29 12:03:23,751] kopf.activities.star [INFO ] Activity 'startup_tasks' succeeded.
[2025-01-29 12:03:23,751] kopf._core.engines.a [INFO ] Initial authentication has been initiated.
[2025-01-29 12:03:23,752] kopf.activities.auth [INFO ] Activity 'login_via_client' succeeded.
[2025-01-29 12:03:23,752] kopf._core.engines.a [INFO ] Initial authentication has finished.
...
[2025-01-29 13:06:34,041] kopf._core.engines.a [INFO ] Re-authentication has been initiated.
[2025-01-29 13:06:34,043] kopf.activities.auth [INFO ] Activity 'login_via_client' succeeded.
[2025-01-29 13:06:34,043] kopf._core.engines.a [INFO ] Re-authentication has finished.
[2025-01-29 13:06:34,044] kopf._core.reactor.r [ERROR ] Namespace observer has failed: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
await coro
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/observation.py", line 77, in namespace_observer
await queueing.watcher(
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
async for raw_event in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 86, in infinite_watch
async for raw_event in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 201, in continuous_watch
async for raw_input in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 266, in watch_objs
async for raw_input in api.stream(
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/api.py", line 201, in stream
response = await request(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/auth.py", line 48, in wrapper
async for key, info, context in vault.extended(APIContext, 'contexts'):
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 158, in extended
async for key, item in self._items():
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 195, in _items
yielded_key, yielded_item = self.select()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 214, in select
raise LoginError("Ran out of valid credentials. Consider installing "
kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
[2025-01-29 13:06:34,045] kopf._core.reactor.r [ERROR ] Resource observer has failed: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
await coro
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/observation.py", line 127, in resource_observer
await queueing.watcher(
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
async for raw_event in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 86, in infinite_watch
async for raw_event in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 201, in continuous_watch
async for raw_input in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 266, in watch_objs
async for raw_input in api.stream(
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/api.py", line 201, in stream
response = await request(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/auth.py", line 48, in wrapper
async for key, info, context in vault.extended(APIContext, 'contexts'):
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 158, in extended
async for key, item in self._items():
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 195, in _items
yielded_key, yielded_item = self.select()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 214, in select raise LoginError("Ran out of valid credentials. Consider installing " kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/ [2025-01-29 13:06:34,046] kopf._core.reactor.o [ERROR ] Watcher for <class>@<namespace> has failed: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
await coro
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
async for raw_event in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 86, in infinite_watch
async for raw_event in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 201, in continuous_watch
async for raw_input in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 266, in watch_objs
async for raw_input in api.stream(
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/api.py", line 201, in stream
response = await request(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/auth.py", line 48, in wrapper
async for key, info, context in vault.extended(APIContext, 'contexts'):
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 158, in extended
async for key, item in self._items():
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 195, in _items
yielded_key, yielded_item = self.select()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 214, in select
raise LoginError("Ran out of valid credentials. Consider installing "
kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
Traceback (most recent call last):
File "/usr/local/bin/kopf", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/cli.py", line 60, in wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/decorators.py", line 92, in new_func
return ctx.invoke(f, obj, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/cli.py", line 109, in run
return running.run(
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 81, in run
asyncio.run(coro)
File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main) ^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 138, in operator
await run_tasks(operator_tasks, ignored=existing_tasks)
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/running.py", line 419, in run_tasks
await aiotasks.reraise(root_done | root_cancelled | hung_done | hung_cancelled)
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 226, in reraise
task.result() # can raise the regular (non-cancellation) exceptions.
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 96, in guard
await coro
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/observation.py", line 77, in namespace_observer
await queueing.watcher(
File "/usr/local/lib/python3.12/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
async for raw_event in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 86, in infinite_watch
async for raw_event in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 201, in continuous_watch
async for raw_input in stream:
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/watching.py", line 266, in watch_objs
async for raw_input in api.stream(
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/api.py", line 201, in stream
response = await request(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/clients/auth.py", line 48, in wrapper
async for key, info, context in vault.extended(APIContext, 'contexts'):
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 158, in extended
async for key, item in self._items():
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 195, in _items
yielded_key, yielded_item = self.select()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/kopf/_cogs/structs/credentials.py", line 214, in select
raise LoginError("Ran out of valid credentials. Consider installing "
kopf._cogs.structs.credentials.LoginError: Ran out of valid credentials. Consider installing an API client library or adding a login handler. See more: https://kopf.readthedocs.io/en/stable/authentication/
Additional information
No response