Kubernetes ApiException (409/Conflict) creating ConfigMaps

[alanbchristie]

In one deployment (kubernetes 1.30.5) using kopf 1.37.5 and kubernetes 31.0.0 we regularly see an already exists error in the operator for the initial objects. Even though each has a unique name it is as though the objects do already exist...

[2025-04-01 14:20:12,960] root                 [INFO    ] Creating ConfigMaps instance-591b35ef-5a8e-4ad9-88bc-bb3873de553d...
[2025-04-01 14:20:12,975] kubernetes.client.re [DEBUG   ] response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"bp-instance-591b35ef-5a8e-4ad9-88bc-bb3873de553d\" already exists","reason":"AlreadyExists","details":{"name":"bp-instance-591b35ef-5a8e-4ad9-88bc-bb3873de553d","kind":"configmaps"},"code":409}

[2025-04-01 14:20:12,975] kopf.objects         [ERROR   ] [data-manager-api-dev/instance-591b35ef-5a8e-4ad9-88bc-bb3873de553d] Handler 'jupyter' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 276, in execute_handler_once
    result = await invoke_handler(
  File "/usr/local/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 371, in invoke_handler
    result = await invocation.invoke(
  File "/usr/local/lib/python3.10/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
    await asyncio.shield(future)  # slightly expensive: creates tasks
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/src/./handlers.py", line 176, in create
    core_api.create_namespaced_config_map(namespace, bp_cm_body)
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 6641, in create_namespaced_config_map
    return self.create_namespaced_config_map_with_http_info(namespace, body, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 6740, in create_namespaced_config_map_with_http_info
    return self.api_client.call_api(
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 391, in request
    return self.rest_client.POST(url,
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 279, in POST
    return self.request("POST", url,
  File "/usr/local/lib/python3.10/site-packages/kubernetes/client/rest.py", line 238, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict({'Audit-Id': '51d628ee-758d-40a5-8cff-005e019d625e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '0fc062ff-faf7-4dcc-bd8d-b616275a18a1', 'X-Kubernetes-Pf-Prioritylevel-Uid': '8cf5978a-d646-4817-b84a-02b5cfc6ee75', 'Date': 'Tue, 01 Apr 2025 14:20:12 GMT', 'Content-Length': '286'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"bp-instance-591b35ef-5a8e-4ad9-88bc-bb3873de553d\" already exists","reason":"AlreadyExists","details":{"name":"bp-instance-591b35ef-5a8e-4ad9-88bc-bb3873de553d","kind":"configmaps"},"code":409}

[alanbchristie]

It's not clear what the fault is but it's clear it's outside our control. So, as a work-around for now we trap the first exception and assume the other objects exist and return the appropriate response: -

This is not a fix it's a WORK-AROUND

    # We currently see a number of 409 exceptions with the objects we create
    # (with the reason 'Conflict'). As each one has a unique name
    # we have to assume there is a serious underlying problem
    # in kopf of kubernetes. For now, if the first object we create
    # already exists let us assume they all do?
    #
    # Added as a work-wround for sc-
    try:
        core_api.create_namespaced_config_map(namespace, bp_cm_body)
    except kubernetes.client.exceptions.ApiException as ex:
        if ex.status != 409 or ex.reason != "Conflict":
            raise ex
        # Warn, but ignore and return a valid 'create' response now.
        logging.warning("Got ApiException [409/Conflict] creating BP ConfigMap. Ignoring [#10]")
        return create_response