CA self-signed certs

[scr112]

Hi. We're using the Alauda container platform. The operator currently can't start because the root certificate is located elsewhere and is self-signed. Logs are included.

2026-02-04 09:16:50,782] urllib3.connectionpo [WARNING ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)'))': /apis/roundtable.lsst.codes/v1beta1/namespaces/ecpk-core-kafka-server/strimzischemaregistries?timeoutSeconds=60
[2026-02-04 09:16:50,790] urllib3.connectionpo [WARNING ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)'))': /apis/roundtable.lsst.codes/v1beta1/namespaces/ecpk-core-kafka-server/strimzischemaregistries?timeoutSeconds=60
[2026-02-04 09:16:50,798] urllib3.connectionpo [WARNING ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)'))': /apis/roundtable.lsst.codes/v1beta1/namespaces/ecpk-core-kafka-server/strimzischemaregistries?timeoutSeconds=60
Traceback (most recent call last):
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 464, in _make_request
    self._validate_conn(conn)
    ~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 1093, in _validate_conn
    conn.connect()
    ~~~~~~~~~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connection.py", line 790, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
        sock=sock,
    ...<14 lines>...
        assert_fingerprint=self.assert_fingerprint,
    )
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connection.py", line 969, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
        sock=sock,
    ...<8 lines>...
        tls_in_tls=tls_in_tls,
    )
  File "/app/.venv/lib/python3.13/site-packages/urllib3/util/ssl_.py", line 480, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/app/.venv/lib/python3.13/site-packages/urllib3/util/ssl_.py", line 524, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sock=sock,
        ^^^^^^^^^^
    ...<5 lines>...
        session=session
        ^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/ssl.py", line 1076, in _create
    self.do_handshake()
    ~~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/ssl.py", line 1372, in do_handshake
    self._sslobj.do_handshake()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
        conn,
    ...<10 lines>...
        **response_kw,
    )
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 488, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/.venv/bin/kopf", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "/app/.venv/lib/python3.13/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "/app/.venv/lib/python3.13/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/app/.venv/lib/python3.13/site-packages/kopf/cli.py", line 64, in wrapper
    return fn(*args, **kwargs)
  File "/app/.venv/lib/python3.13/site-packages/click/decorators.py", line 93, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/app/.venv/lib/python3.13/site-packages/kopf/cli.py", line 108, in run
    loaders.preload(
    ~~~~~~~~~~~~~~~^
        paths=paths,
        ^^^^^^^^^^^^
        modules=modules,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/kopf/_cogs/helpers/loaders.py", line 47, in preload
    importlib.import_module(name)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/local/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/app/.venv/lib/python3.13/site-packages/strimziregistryoperator/handlers/__init__.py", line 12, in <module>
    start_operator(logger=None)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/strimziregistryoperator/startup.py", line 22, in start_operator
    response = api.list_namespaced_custom_object(
        "roundtable.lsst.codes",
    ...<3 lines>...
        timeout_seconds=60,
    )
  File "/app/.venv/lib/python3.13/site-packages/kubernetes/client/api/custom_objects_api.py", line 2455, in list_namespaced_custom_object
    return self.list_namespaced_custom_object_with_http_info(group, version, namespace, plural, **kwargs)  # noqa: E501
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/kubernetes/client/api/custom_objects_api.py", line 2594, in list_namespaced_custom_object_with_http_info
    return self.api_client.call_api(
           ~~~~~~~~~~~~~~~~~~~~~~~~^
        '/apis/{group}/{version}/namespaces/{namespace}/{plural}', 'GET',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<11 lines>...
        _request_timeout=local_var_params.get('_request_timeout'),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        collection_formats=collection_formats)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
                           path_params, query_params, header_params,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
                           _return_http_data_only, collection_formats,
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                           _preload_content, _request_timeout, _host)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
        method, url, query_params=query_params, headers=header_params,
        post_params=post_params, body=body,
        _preload_content=_preload_content,
        _request_timeout=_request_timeout)
  File "/app/.venv/lib/python3.13/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
           ~~~~~~~~~~~~~~~~~~~~^^^^^
                                query_params=query_params,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^
                                _preload_content=_preload_content,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                _request_timeout=_request_timeout,
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                headers=headers)
                                ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/kubernetes/client/rest.py", line 244, in GET
    return self.request("GET", url,
           ~~~~~~~~~~~~^^^^^^^^^^^^
                        headers=headers,
                        ^^^^^^^^^^^^^^^^
                        _preload_content=_preload_content,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        _request_timeout=_request_timeout,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                        query_params=query_params)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/kubernetes/client/rest.py", line 217, in request
    r = self.pool_manager.request(method, url,
                                  fields=query_params,
                                  preload_content=_preload_content,
                                  timeout=timeout,
                                  headers=headers)
  File "/app/.venv/lib/python3.13/site-packages/urllib3/_request_methods.py", line 135, in request
    return self.request_encode_url(
           ~~~~~~~~~~~~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<3 lines>...
        **urlopen_kw,
        ^^^^^^^^^^^^^
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/urllib3/_request_methods.py", line 182, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/urllib3/poolmanager.py", line 459, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
           ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<13 lines>...
        **response_kw,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
           ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<13 lines>...
        **response_kw,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 871, in urlopen
    return self.urlopen(
           ~~~~~~~~~~~~^
        method,
        ^^^^^^^
    ...<13 lines>...
        **response_kw,
        ^^^^^^^^^^^^^^
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
        method, url, error=new_e, _pool=self, _stacktrace=sys.exc_info()[2]
    )
  File "/app/.venv/lib/python3.13/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.4.0.1', port=443): Max retries exceeded with url: /apis/roundtable.lsst.codes/v1beta1/namespaces/ecpk-core-kafka-server/strimzischemaregistries?timeoutSeconds=60 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)')))

[afausti]

I don't think the problem is just a self-signed certificate, the warning message
'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Missing Authority Key Identifier (_ssl.c:1032)')
suggests that the Kubernetes API server certificate is missing the authority key identifier extension which is now required by Python 3.13 when validating the certificate.
I'm not familiar with Alauda but maybe check the documentation or look for issues related to that.

[scr112]

Well, I was able to get it running here, but I had to write a bunch of patches using DeepSeek. I don't think everything is correct, so I'm not sending you a pull request. It creates a separate pull request, understands the SSL verification level, and patches aiohttp.
We've created a ticket for the Alauda vendor. + Added an import file for our root certificate to the Docker file.
If you're worried, I'll create a pull request for analysis, since we'll be using your solution. Maintaining it and constantly patching it wouldn't be the right solution.

I checked, yes, our subsidiary certificates do not have AKI.

[scr112]

[scr112]

python/cpython#138193