root CA certificate does not include key usage extension

[mfmarche]

Summary

The root CA certificate that is generated does not have the keyUsage defined (see https://github.com/canonical/microk8s/blob/master/microk8s-resources/actions/common/utils.sh#L689)

an example CA cert extension has:

    X509v3 extensions:
        X509v3 Subject Key Identifier: 
            0E:EF:10:1C:40:F6:85:87:76:23:A4:40:C7:D7:73:41:AB:F4:9E:A8
        X509v3 Authority Key Identifier: 
            0E:EF:10:1C:40:F6:85:87:76:23:A4:40:C7:D7:73:41:AB:F4:9E:A8
        X509v3 Basic Constraints: critical
            CA:TRUE

Python 3.13 now has:

Changed in version 3.13: The context now uses VERIFY_X509_PARTIAL_CHAIN and VERIFY_X509_STRICT in its default verify flags.

An example error, when running kopf via python 3.13, sees:

[2025-02-11 21:16:16,609] kopf._core.reactor.o [ERROR ] Request attempt #9/9 failed; escalating: GET https://10.152.183.1:443/api -> ClientConnectorCertificateError(ConnectionKey(host='10.152.183.1', port=443, is_ssl=True, ssl=True, proxy=None, proxy_auth=None, proxy_headers_hash=None), SSLCertVerificationError(1,
'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: CA cert does not include key usage extension (_ssl.c:1028)'))

In order to workaround this issue, the strict setting must be removed in order to establish an SSL connection to microk8s with the CA certificate, which is not ideal (there could be multiple packages, for example, kopf, kubernetes client, etc), and less secure.

What Should Happen Instead?

Generate a CA certificate with keyUsage defined.

Is there another way to update the openssl.cnf file and its defaults? I see in:

https://github.com/canonical/microk8s/blob/master/microk8s-resources/wrappers/openssl.wrapper#L9C1-L10C1

and it refers to
export OPENSSL_CONF="${SNAP}/etc/ssl/openssl.cnf"

however, I don't believe this is a file that users can update/modify.

[mfmarche]

In case anyone else hits this, I resolved it by creating and installing a CA:

mkdir cadir
openssl genrsa -out cadir/ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 360 -out cadir/ca.crt -addext "keyUsage=critical,digitalSignature,keyCertSign"
microk8s.refresh-certs cadir

[eaudetcobello]

Hi @mfmarche,

I've notified the team and we'll schedule work to add the -addext "keyUsage=critical,digitalSignature,keyCertSign" as suggested.

Thanks!

[brentgroves]

If you make your own root certificate, install it in your system trust store.

[duesee]

I've notified the team and we'll schedule work to add the -addext "keyUsage=critical,digitalSignature,keyCertSign" as suggested.

I'm not exactly sure what the certificate is used for but if it's mainly for a TLS PKI, I feel keyCertSign and cRLSign -- maybe it's a good idea to mimic Let's Encrypt -- could be the KeyUsages you are looking for.

From https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1.3:

If the subject public key is only to be used for verifying signatures on certificates and/or CRLs, then the digitalSignature and nonRepudiation bits SHOULD NOT be set. However, the digitalSignature and/or nonRepudiation bits MAY be set in addition to the keyCertSign and/or cRLSign bits if the subject public key is to be used to verify signatures on certificates and/or CRLs as well as other objects.

[ZephirDev]

I write a small Ansible playbook to apply this fix based upon what you said. I have to add some step because the certificate was not trust by kubectl and the mysql-operator (my use-case). I don't know if that can help some people.

Ansible will require that collection:

ansible-galaxy collection install community.crypto

#
# This file is there to fix a bug from microk8s certificate
#
- hosts: all
  vars:

    ca_dir: "{{ microk8s.ca_dir | default('/usr/local/share/ca-certificates/microk8s') }}"

  tasks:

    - name: "Create CA directory"
      become: yes
      file:
        dest: "{{ ca_dir }}"
        state: directory
        recurse: yes

    - name: "Create a CA private key"
      become: yes
      community.crypto.openssl_privatekey:
        path: "{{ ca_dir }}/ca.key"
        size: 2048

    - name: "Create a CA certificate signing request"
      become: yes
      community.crypto.openssl_csr:
        path: "{{ ca_dir }}/ca.csr"
        privatekey_path: "{{ ca_dir }}/ca.key"
        basic_constraints_critical: true
        basic_constraints:
          - "CA:TRUE"
        commonName: 10.152.183.1
        key_usage_critical: true
        key_usage:
          - digitalSignature
          - keyCertSign

    - name: "Create a CA certificate"
      become: yes
      community.crypto.x509_certificate:
        path: "{{ ca_dir }}/ca.crt"
        privatekey_path: "{{ ca_dir }}/ca.key"
        csr_path: "{{ ca_dir }}/ca.csr"
        provider: selfsigned

    - name: "Apply ca certificate"
      become: yes
      shell: "/snap/bin/microk8s.refresh-certs {{ ca_dir }}"

    - name: "Trust ca certificate"
      become: yes
      shell: "update-ca-certificates"

.

[MurrayData]

Hi @mfmarche,

I've notified the team and we'll schedule work to add the -addext "keyUsage=critical,digitalSignature,keyCertSign" as suggested.

Thanks!

@eaudetcobello Please can you advise if there's an expected date for this change? It is causing us issues in newer containers and we're having to implement workarounds.

Many thanks.

[carlosmarind]

same issue here,

[GeoffreyStemys]

This bug is really annoying, the grafana sidecar won't work because of the missing "Key Usage" in the CA ...

[SylteA]

+1

[AgentK20]

Also running into this in my cluster with kube-prometheus-stack's grafana sidecars.

[sabpamdev1]

is downgrading to a lower microk8s version worked for anyone, if tried? Thanks

[sabpamdev1]

Hi @mfmarche,

I've notified the team and we'll schedule work to add the -addext "keyUsage=critical,digitalSignature,keyCertSign" as suggested.

Thanks!

Hi @eaudetcobello ,
I have attempted to change the /var/snap/microk8s/current/certs/csr.conf.template file by updating keyUsage attribute of [ v3_ext ] block and tried to refresh-certs. No luck.

As a random attempt, I have added a [ v3_ca ] block to set as below:

[ v3_ca ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=critical,CA:TRUE
keyUsage=critical,digitalSignature,keyCertSign

Tried the refreshing the certs again but still same error.

@mfmarche's solution works but its persistent to the current session and had to be applied again when we restart microk8s as microk8s is potentially detecting a 'change' in the current certs and refreshing them, which brings back the original error, making the workaround unsustainable in the long term.

My setup is WSL2 (Ubuntu 24.04) on a Windows11 machine. It very unfortunate and ironic that Microk8s is having an issue with one of its sibling products from Canonical.

[sabpamdev1]

Ok I have attempted to raise a PR but couldn't due to me not meeting certain criteria immediately. It looks like a single line change, from the draft PR I was going to raise (@mfmarche spotted it long time ago)

Would be really nice to have this fixed soon 🙏

[davepgreene]

Is there a process for fixing this in a live cluster, especially with multiple nodes? I'm having to pin back Grafana sidecars in multiple places now that they've updated the version of kiwigrid/k8s-sidecar they use everywhere and I'd love to be able to roll forward.