Redpanda does not create a client certificate when mTLS is enabled.

[Ichthyony]

1)install operator
helm upgrade --install redpanda-controller redpanda/operator --namespace redpanda-stage --create-namespace --version v25.1.3 --set crds.enabled=true

2)Install a Redpanda custom resource

apiVersion: cluster.redpanda.com/v1alpha2
kind: Redpanda
metadata:
  name: redpanda-stage
  namespace: redpanda-stage
spec:
  clusterSpec:
    image:
      tag: v25.2.4

    tls:
      enabled: true
      certs:
        default:
          caEnabled: true
        external:
          caEnabled: true
    
    listeners:
      admin:
        port: 9644
        tls:
          cert: default
          requireClientAuth: true

      kafka:
        port: 9093
        authenticationMethod:
        tls:
          cert: default
          requireClientAuth: true

      rpc:
        port: 33145
        tls:
          cert: default
          requireClientAuth: true

      schemaRegistry:
        enabled: true
        port: 8081
        authenticationMethod:
        tls:
          cert: default
          requireClientAuth: true

      http:
        enabled: true
        port: 8082
        authenticationMethod:
        tls:
          cert: default
          requireClientAuth: true

    config:
      cluster:
        admin_api_require_auth: true

3)Logs in redpanda-operator
could not find the requested resource (patch certificates.cert-manager.io redpanda-stage-client)"
And cant create redpanda-pods

redpanda-cluster-stage-client certificate not create
other root and selfsigned issuers, certs, secret created

[chrisseto]

Strange, I though we had a test for this!

If you still have this cluster around, would you be able to copy the full log output? There should be a stacktrace in there which would be very helpful.

[Ichthyony]

-- stack trace:\n  | github.com/redpanda-data/redpanda-operator/operator/internal/controller/redpanda.(*RedpandaReconciler).syncStatusErr\n  | \t/work/operator/internal/controller/redpanda/redpanda_controller.go:700\n  | github.com/redpanda-data/redpanda-operator/operator/internal/controller/redpanda.(*RedpandaReconciler).Reconcile\n  | \t/work/operator/internal/controller/redpanda/redpanda_controller.go:221\n  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n  | \t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119\n  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n  | \t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334\n  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n  | \t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\n  | sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n  | \t/root/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255\n  | runtime.goexit\n  | \t/nix/store/5xvi25nqmbrg58aixp4zgczilfnp7pwg-go-1.24.3/share/go/src/runtime/asm_amd64.s:1700\nWraps: (2) the server could not find the requested resource (patch certificates.cert-manager.io redpanda-stage-client)\n└─ Wraps: (3) the server could not find the requested resource (patch certificates.cert-manager.io redpanda-stage-client)\n  └─ Wraps: (4) the server could not find the requested resource (patch certificates.cert-manager.io redpanda-stage-client)\nError types: (1) *withstack.withStack (2) *join.joinError (3) *errors.joinError (4) *errors.StatusError"}

kubectl get certificates -n redpanda-stage

NAME                                       READY   SECRET                                     AGE
redpanda-stage-client                      True    redpanda-stage-client                      7m28s
redpanda-stage-default-cert                True    redpanda-stage-default-cert                5h54m
redpanda-stage-default-root-certificate    True    redpanda-stage-default-root-certificate    5h54m
redpanda-stage-external-cert               True    redpanda-stage-external-cert               5h54m
redpanda-stage-external-root-certificate   True    redpanda-stage-external-root-certificate   5h54m

The redpanda-stage-client was added later myself
And he still can't find it.

kubectl auth can-i patch certificates/redpanda-stage-client -n redpanda-stage --as=system:serviceaccount:redpanda-stage:redpanda-controller-operator

yes

kubectl patch certificate redpanda-stage-client -n redpanda-stage --type=merge -p '{\"metadata\":{\"annotations\":{\"test\":\"value\"}}}' --as=system:serviceaccount:redpanda-stage:redpanda-controller-operator

certificate.cert-manager.io/redpanda-stage-client patched

by rbac itself, which was configured (there were also problems with it from the beginning until it was created by myself)
rbac.enabled.true and rbac.createRPKBundleCRs=true by default

"error":"certificates.cert-manager.io \"redpanda-stage-client\" is forbidden: User \"system:serviceaccount:redpanda-stage:redpanda-controller-operator\" cannot patch resource \"certificates\" in API group \"cert-manager.io\" at the cluster scope"}

this fix problem with access

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: redpanda-cert-manager-access
rules:
- apiGroups: ["cert-manager.io"]
  resources: ["certificates", "issuers", "clusterissuers"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

--- 

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: redpanda-cert-manager-binding
subjects:
- kind: ServiceAccount
  name: redpanda-controller-operator
  namespace: redpanda-stage
roleRef:
  kind: ClusterRole
  name: redpanda-cert-manager-access
  apiGroup: rbac.authorization.k8s.io

Old installation with helm was success
redpanda-operator 0.4.25 \ v2.1.26-24.1.9
redpanda 5.8.13 \ v24.1.11

New cluster with operator and cluster.redpanda.com/v1alpha2