custom resources not found, please enable the respective API group

[yuqun20230302]

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

Version

2.24.0

Storage Type

Kubernetes

Installation Type

Official container image

Expected Behavior

According to the 2.24.0 kubernetes.md document, the dex service is created normally

Actual Behavior

According to the 2.24.0 kubernetes.md document，An error occurred when the dex pod was started。
The error information is as follows：
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource authrequests.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource oauth2clients.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource signingkeies.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"

Steps To Reproduce

1.Configuring the OpenID Connect plugin

vim /usr/lib/systemd/system/kube-apiserver.service

ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --allow-privileged=true  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --advertise-address=172.16.58.101 \
      --service-cluster-ip-range=192.168.0.0/16  \
      --service-node-port-range=30000-32767  \
      --etcd-servers=https://172.16.58.101:2379,https://172.16.58.102:2379,https://172.16.58.103:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
      --requestheader-allowed-names=front-proxy-client  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User  \
      --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
      # --token-auth-file=/etc/kubernetes/token.csv
      --oidc-issuer-url=https://dex.example.com:32000 \
      --oidc-client-id=example-app \
      --oidc-ca-file=/root/yq/dex/ssl/ca.pem \
      --oidc-username-claim=email \
      --oidc-groups-claim=groups

systemctl daemon-reload && systemctl restart --now kube-apiserver

2.Deploying dex on Kubernetes
1).Generate TLS assets

./gencert.sh

#!/bin/bash

mkdir -p ssl

cat << EOF > ssl/req.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = dex.example.com
EOF

openssl genrsa -out ssl/ca-key.pem 2048
openssl req -x509 -new -nodes -key ssl/ca-key.pem -days 10 -out ssl/ca.pem -subj "/CN=kube-ca"

openssl genrsa -out ssl/key.pem 2048
openssl req -new -key ssl/key.pem -out ssl/csr.pem -subj "/CN=kube-ca" -config ssl/req.cnf
openssl x509 -req -in ssl/csr.pem -CA ssl/ca.pem -CAkey ssl/ca-key.pem -CAcreateserial -out ssl/cert.pem -days 10 -extensions v3_req -extfile ssl/req.cnf

pwd
/root/yq/dex/ssl

2).Configure the API server

export GITHUB_CLIENT_ID=Ov23limiiqmMQVFcVUlR
export GITHUB_CLIENT_SECRET=f5ad53f6735b5a457607f5f13bf59fee7214751c

kubectl create secret tls dex.example.com.tls --cert=ssl/cert.pem --key=ssl/key.pem
kubectl create secret \
    generic github-client \
    --from-literal=client-id=$GITHUB_CLIENT_ID \
    --from-literal=client-secret=$GITHUB_CLIENT_SECRET

3). crds

kubectl apply -f dex-crd.yaml

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: authcodes.dex.coreos.com
spec:
  group: dex.coreos.com
  names:
    kind: AuthCode
    listKind: AuthCodeList
    plural: authcodes
    singular: authcode
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object

4).Deploy the Dex server

kubectl apply -f dex.yaml -n auth

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: dex
  name: dex
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dex
  template:
    metadata:
      labels:
        app: dex
    spec:
      serviceAccountName: dex # This is created below
      containers:
      - image: 172.16.58.104:28081/kubeflow/dex:v2.24.0
        name: dex
        command: ["/usr/local/bin/dex", "serve", "/etc/dex/cfg/config.yaml"]

        ports:
        - name: https
          containerPort: 5556

        volumeMounts:
        - name: config
          mountPath: /etc/dex/cfg
        - name: tls
          mountPath: /etc/dex/tls

        env:
        - name: KUBERNETES_POD_NAMESPACE
          value: auth
        - name: GITHUB_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: github-client
              key: client-id
        - name: GITHUB_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: github-client
              key: client-secret
      volumes:
      - name: config
        configMap:
          name: dex
          items:
          - key: config.yaml
            path: config.yaml
      - name: tls
        secret:
          secretName: dex.example.com.tls
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: dex
data:
  config.yaml: |
    issuer: https://dex.example.com:32000
    storage:
      type: kubernetes
      config:
        inCluster: true
    web:
      https: 0.0.0.0:5556
      tlsCert: /etc/dex/tls/tls.crt
      tlsKey: /etc/dex/tls/tls.key
    connectors:
    - type: github
      id: github
      name: GitHub
      config:
        clientID: $GITHUB_CLIENT_ID
        clientSecret: $GITHUB_CLIENT_SECRET
        redirectURI: https://dex.example.com:32000/callback
        org: kubernetes
    oauth2:
      skipApprovalScreen: true

    staticClients:
    - id: example-app
      redirectURIs:
      - 'http://127.0.0.1:5555/callback'
      name: 'Example App'
      secret: ZXhhbXBsZS1hcHAtc2VjcmV0

    enablePasswordDB: true
    staticPasswords:
    - email: "admin@example.com"
      # bcrypt hash of the string "password"
      hash: "$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"
      username: "admin"
      userID: "08a8684b-db88-4b73-90a9-3cd1661f5466"
---
apiVersion: v1
kind: Service
metadata:
  name: dex
spec:
  type: NodePort
  ports:
  - name: dex
    port: 5556
    protocol: TCP
    targetPort: 5556
    nodePort: 32000
  selector:
    app: dex
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: dex
  name: dex
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: dex
rules:
- apiGroups: ["dex.coreos.com"] # API group created by dex
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["create"] # To manage its own resources, dex must be able to create customresourcedefinitions
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: dex
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: dex
subjects:
- kind: ServiceAccount
  name: dex           # Service account assigned to the dex pod, created above
  namespace: auth  # The namespace dex is running in

Additional Information

kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:21:34Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.6", GitCommit:"741c8db18a52787d734cbe4795f0b4ad860906d6", GitTreeState:"clean", BuildDate:"2023-09-13T09:14:09Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}

Linux xxxx 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
````bash

````bash
dex images: quay.io/dexidp/dex:v2.24.0

Configuration

No response

Logs

time="2024-07-09T07:08:08Z" level=info msg="config issuer: https://dex.example.com:32000"
time="2024-07-09T07:08:08Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1"
time="2024-07-09T07:08:08Z" level=info msg="creating custom Kubernetes resources"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource authrequests.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource oauth2clients.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource signingkeies.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource refreshtokens.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource refreshtokens.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource passwords.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource passwords.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource offlinesessionses.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource offlinesessionses.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource connectors.dex.coreos.com has been created already..."
time="2024-07-09T07:08:08Z" level=info msg="failed to list custom resource connectors.dex.coreos.com, attempting to create: not found"
time="2024-07-09T07:08:08Z" level=error msg="custom resources not found, please enable the respective API group"
time="2024-07-09T07:08:08Z" level=error msg="failed creating custom resources: <nil>"
time="2024-07-09T07:08:08Z" level=info msg="config storage: kubernetes"
time="2024-07-09T07:08:08Z" level=info msg="config static client: Example App"
time="2024-07-09T07:08:08Z" level=info msg="config connector: github"
time="2024-07-09T07:08:08Z" level=info msg="config connector: local passwords enabled"
time="2024-07-09T07:08:08Z" level=info msg="config skipping approval screen"
time="2024-07-09T07:08:08Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has been created already..."
failed to initialize server: server: failed to list connector objects from storage: failed to list connectors: not found

[nabokihms]

The Dex version needs to be updated for your Kubernetes, I'm afraid.
Dex 2.24 uses the apiextensions.k8s.io/v1beta1 API to create custom resources, which does not exist in newer Kubernetes.