Skip to content

Handle JSONDecodeError in dynamic client discovery when API server returns 503 with text/plain  #2429

@tamilhce

Description

@tamilhce

What happened (please include outputs or screenshots):
When the Kubernetes API server returns a 503 Service Unavailable error with a text/plain content type (instead of application/json), the dynamic client fails with an unhandled JSONDecodeError exception while trying to parse the response as JSON.

This issue was discovered in the async version (kubernetes_asyncio) where it manifests as a ContentTypeError, but the same underlying issue exists in the non-async client as well.

Error stack trace from the async version (similar error occurs in non-async):

File "/home/app/cluster.py", line 518, in get_all_cluster_objects
  api_groups = await dynamic_client.resources.parse_api_groups(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/app/.cache/pypoetry/virtualenvs/a-_geWzZZ1-py3.12/lib/python3.12/site-packages/kubernetes_asyncio/dynamic/discovery.py", line 147, in parse_api_groups
  resources = await self.get_resources_for_api_version(DISCOVERY_PREFIX, group['name'], version,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/app/.cache/pypoetry/virtualenvs/a-_geWzZZ1-py3.12/lib/python3.12/site-packages/kubernetes_asyncio/dynamic/discovery.py", line 184, in get_resources_for_api_version
  response = await self.client.request('GET', path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/app/.cache/pypoetry/virtualenvs/a-_geWzZZ1-py3.12/lib/python3.12/site-packages/kubernetes_asyncio/dynamic/client.py", line 59, in inner
  data = await resp.json()
         ^^^^^^^^^^^^^^^^^
File "/home/app/.cache/pypoetry/virtualenvs/a-_geWzZZ1-py3.12/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1281, in json
  raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 503, message='Attempt to decode JSON with unexpected mimetyp

What you expected to happen:
The client should gracefully handle non-JSON responses from the API server during service unavailability, similar to how it already handles ServiceUnavailableError. It should continue execution with an empty resources list rather than crashing with an unhandled exception

How to reproduce it (as minimally and precisely as possible):

  • Use the kubernetes dynamic client to interact with a Kubernetes cluster

  • Have an API endpoint that returns a 503 Service Unavailable with text/plain content type

  • Attempt to discover API resources using the dynamic client

from kubernetes import client, config, dynamic

# Load config
config.load_kube_config()

# Create dynamic client
dynamic_client = dynamic.DynamicClient(
    client.ApiClient()
)

# This will fail if an API endpoint returns 503 with text/plain
api_groups = dynamic_client.resources.parse_api_groups()

Anything else we need to know?:
The fix is relatively simple - modify the get_resources_for_api_version method in kubernetes/dynamic/discovery.py to catch JSONDecodeError in addition to ServiceUnavailableError:

# Add import
from json.decoder import JSONDecodeError

# Modify exception handling in get_resources_for_api_version
try:
    resources_response = self.client.request('GET', path).resources or []
except (ServiceUnavailableError, JSONDecodeError):
    # Handle both service unavailable errors and JSON decode errors
    resources_response = []

I have submitted similar fix for async version (kubernetes_asyncio) where the equivalent error is ContentTypeError from aiohttp. Ref: tomplus/kubernetes_asyncio#368
Environment:

  • Kubernetes version (kubectl version): v1.32.0

  • OS (e.g., MacOS 10.13.6): Linux 4.18

  • Python version (python --version): Python 3.12

  • Python client version (pip list | grep kubernetes): kubernetes-33.1.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions