-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Description
What happened (please include outputs or screenshots):
When the Kubernetes API server returns a 503 Service Unavailable error with a text/plain content type (instead of application/json), the dynamic client fails with an unhandled JSONDecodeError exception while trying to parse the response as JSON.
This issue was discovered in the async version (kubernetes_asyncio) where it manifests as a ContentTypeError, but the same underlying issue exists in the non-async client as well.
Error stack trace from the async version (similar error occurs in non-async):
File "/home/app/cluster.py", line 518, in get_all_cluster_objects
api_groups = await dynamic_client.resources.parse_api_groups(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/app/.cache/pypoetry/virtualenvs/a-_geWzZZ1-py3.12/lib/python3.12/site-packages/kubernetes_asyncio/dynamic/discovery.py", line 147, in parse_api_groups
resources = await self.get_resources_for_api_version(DISCOVERY_PREFIX, group['name'], version,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/app/.cache/pypoetry/virtualenvs/a-_geWzZZ1-py3.12/lib/python3.12/site-packages/kubernetes_asyncio/dynamic/discovery.py", line 184, in get_resources_for_api_version
response = await self.client.request('GET', path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/app/.cache/pypoetry/virtualenvs/a-_geWzZZ1-py3.12/lib/python3.12/site-packages/kubernetes_asyncio/dynamic/client.py", line 59, in inner
data = await resp.json()
^^^^^^^^^^^^^^^^^
File "/home/app/.cache/pypoetry/virtualenvs/a-_geWzZZ1-py3.12/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1281, in json
raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 503, message='Attempt to decode JSON with unexpected mimetyp
What you expected to happen:
The client should gracefully handle non-JSON responses from the API server during service unavailability, similar to how it already handles ServiceUnavailableError. It should continue execution with an empty resources list rather than crashing with an unhandled exception
How to reproduce it (as minimally and precisely as possible):
-
Use the kubernetes dynamic client to interact with a Kubernetes cluster
-
Have an API endpoint that returns a 503 Service Unavailable with text/plain content type
-
Attempt to discover API resources using the dynamic client
from kubernetes import client, config, dynamic
# Load config
config.load_kube_config()
# Create dynamic client
dynamic_client = dynamic.DynamicClient(
client.ApiClient()
)
# This will fail if an API endpoint returns 503 with text/plain
api_groups = dynamic_client.resources.parse_api_groups()
Anything else we need to know?:
The fix is relatively simple - modify the get_resources_for_api_version method in kubernetes/dynamic/discovery.py to catch JSONDecodeError in addition to ServiceUnavailableError:
# Add import
from json.decoder import JSONDecodeError
# Modify exception handling in get_resources_for_api_version
try:
resources_response = self.client.request('GET', path).resources or []
except (ServiceUnavailableError, JSONDecodeError):
# Handle both service unavailable errors and JSON decode errors
resources_response = []
I have submitted similar fix for async version (kubernetes_asyncio) where the equivalent error is ContentTypeError from aiohttp. Ref: tomplus/kubernetes_asyncio#368
Environment:
-
Kubernetes version (
kubectl version
): v1.32.0 -
OS (e.g., MacOS 10.13.6): Linux 4.18
-
Python version (
python --version
): Python 3.12 -
Python client version (
pip list | grep kubernetes
): kubernetes-33.1.0