GITHUB_TOKEN env variable from SecretKeyRef not working

[extravio]

Controller Version

0.20.2

Helm Chart Version

0.25.2

CertManager Version

No response

Deployment Method

Other

cert-manager installation

Installed as per the documentation.
Tried with both
certManagerEnabled: false
and
certManagerEnabled: true

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue

Resource Definitions

# values.yml

labels: {}

replicaCount: 1

webhookPort: 9443
syncPeriod: 1m
defaultScaleDownDelay: 10m

enableLeaderElection: true
# Specifies the controller id for leader election.
# Must be unique if more than one controller installed onto the same namespace.
#leaderElectionId: "actions-runner-controller"

# The URL of your GitHub Enterprise server, if you're using one.
#githubEnterpriseServerURL: https://github.example.com

# Override GitHub URLs in case of using proxy APIs
#githubURL: ""
#githubUploadURL: ""
#runnerGithubURL: ""

# Only 1 authentication method can be deployed at a time
# Uncomment the configuration you are applying and fill in the details
#
# If authSecret.enabled=true these values are inherited to actions-runner-controller's controller-manager container's env.
#
# Do set authSecret.enabled=false and set env if you want full control over
# the GitHub authn related envvars of the container.
# See https://github.com/actions-runner-controller/actions-runner-controller/pull/937 for more details.
authSecret:
  enabled: true
  create: false
  name: "controller-manager"
  annotations: {}
  ### GitHub Apps Configuration
  ## NOTE: IDs MUST be strings, use quotes
  #github_app_id: ""
  #github_app_installation_id: ""
  #github_app_private_key: |
  ### GitHub PAT Configuration
  #github_token: ""
  ### Basic auth for github API proxy
  #github_basicauth_username: ""
  #github_basicauth_password: ""

dockerRegistryMirror: ""
image:
  repository: "summerwind/actions-runner-controller"
  actionsRunnerRepositoryAndTag: "summerwind/actions-runner:latest"
  dindSidecarRepositoryAndTag: "docker:dind"
  pullPolicy: IfNotPresent
  # The default image-pull secrets name for self-hosted runner container.
  # It's added to spec.ImagePullSecrets of self-hosted runner pods.
  actionsRunnerImagePullSecrets: []

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

runner:
  statusUpdateHook:
    enabled: false

rbac:
  {}
  # # This allows ARC to dynamically create a ServiceAccount and a Role for each Runner pod that uses "kubernetes" container mode,
  # # by extending ARC's manager role to have the same permissions required by the pod runs the runner agent in "kubernetes" container mode.
  # # Without this, Kubernetes blocks ARC to create the role to prevent a priviledge escalation.
  # # See https://github.com/actions-runner-controller/actions-runner-controller/pull/1268/files#r917327010
  # allowGrantingKubernetesContainerModePermissions: true

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

podAnnotations: {}

podLabels: {}

podSecurityContext:
  {}
  # fsGroup: 2000

securityContext:
  {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

# Webhook service resource
service:
  type: ClusterIP
  port: 443
  annotations: {}

# Metrics service resource
metrics:
  serviceAnnotations: {}
  serviceMonitor: false
  serviceMonitorLabels: {}
  port: 8443
  proxy:
    enabled: true
    image:
      repository: quay.io/brancz/kube-rbac-proxy
      tag: v0.13.0

resources:
  {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #   cpu: 100m
  #   memory: 128Mi
  # requests:
  #   cpu: 100m
  #   memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

# Only one of minAvailable or maxUnavailable can be set
podDisruptionBudget:
  enabled: false
  # minAvailable: 1
  # maxUnavailable: 3

# Leverage a PriorityClass to ensure your pods survive resource shortages
# ref: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
# PriorityClass: system-cluster-critical
priorityClassName: ""

env:
  - name: GITHUB_TOKEN
    valueFrom:
      secretKeyRef:
        key: github-token
        name: github-token
        optional: true

## specify additional volumes to mount in the manager container, this can be used
## to specify additional storage of material or to inject files from ConfigMaps
## into the running container
additionalVolumes:
- name: secret-store-inline
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: ${spc_name}

## specify where the additional volumes are mounted in the manager container
additionalVolumeMounts:
- name: secret-store-inline
  mountPath: /etc/ssl/certs
  readOnly: true

scope:
  # If true, the controller will only watch custom resources in a single namespace
  singleNamespace: ${scope_singleNamespace}
  # If `scope.singleNamespace=true`, the controller will only watch custom resources in this namespace
  # The default value is "", which means the namespace of the controller
  watchNamespace: ""

certManagerEnabled: false

admissionWebHooks:
  {}
  #caBundle: "Ci0tLS0tQk...<base64-encoded PEM bundle containing the CA that signed the webhook's serving certificate>...tLS0K"

# There may be alternatives to setting `hostNetwork: true`, see
# https://github.com/actions-runner-controller/actions-runner-controller/issues/1005#issuecomment-993097155
#hostNetwork: true

githubWebhookServer:
  enabled: ${githubWebhookServer_enabled}
  replicaCount: 1
  useRunnerGroupsVisibility: false
  secret:
    enabled: false
    create: false
    name: "github-webhook-server"
    ### GitHub Webhook Configuration
    github_webhook_secret_token: ""
    ### GitHub Apps Configuration
    ## NOTE: IDs MUST be strings, use quotes
    #github_app_id: ""
    #github_app_installation_id: ""
    #github_app_private_key: |
    ### GitHub PAT Configuration
    #github_token: ""
  imagePullSecrets: []
  nameOverride: ""
  fullnameOverride: ""
  serviceAccount:
    # Specifies whether a service account should be created
    create: true
    # Annotations to add to the service account
    annotations: {}
    # The name of the service account to use.
    # If not set and create is true, a name is generated using the fullname template
    name: ""
  podAnnotations: {}
  podLabels: {}
  podSecurityContext: {}
  # fsGroup: 2000
  securityContext: {}
  resources: {}
  nodeSelector: {}
  tolerations: []
  affinity: {}
  priorityClassName: ""
  service:
    type: ClusterIP
    annotations: {}
    ports:
      - port: 80
        targetPort: http
        protocol: TCP
        name: http
        #nodePort: someFixedPortForUseWithTerraformCdkCfnEtc
  ingress:
    enabled: true
    ingressClassName: ""
    annotations: {
      kubernetes.io/ingress.class: azure/application-gateway
    }
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
    hosts:
      - host: "${host}"
        paths:
         - path: /actions-runner-controller-github-webhook-server
           pathType: Exact
        # Extra paths that are not automatically connected to the server. This is useful when working with annotation based services.
        extraPaths: []
        # - path: /*
        #   backend:
        #     serviceName: ssl-redirect
        #     servicePort: use-annotation
        ## for Kubernetes >=1.19 (when "networking.k8s.io/v1" is used)
        # - path: /*
        #   pathType: Prefix
        #   backend:
        #     service:
        #       name: ssl-redirect
        #       port:
        #         name: use-annotation
    tls:
     - secretName:  tls-csi
       hosts:
         - "${host}"

  # Only one of minAvailable or maxUnavailable can be set
  podDisruptionBudget:
    enabled: false
    # minAvailable: 1
    # maxUnavailable: 3

To Reproduce

Create a SecretProviderClass to define the github-token secret
Deploy the controller using the Helm Chart attached

Describe the bug

I'm using a CSI-provider to store the Github token and I defined an environment variable GITHUB_TOKEN in the helm chart, following the pattern in this Pull Request:
#1565

env:
  - name: GITHUB_TOKEN
    valueFrom:
      secretKeyRef:
        key: github-token
        name: github-token
        optional: true

However, my controller pod is stuck in "ContainerCreating" status , with FailedMount Errors

MountVolume.SetUp failed for volume "secret" : secret "controller-manager" not found
MountVolume.SetUp failed for volume "cert" : secret "actions-runner-controller-serving-cert" not found
(see controller logs attached)

My certs are also on the same volumes, and when I pass the github token as a variable, the pods start correctly and the certs are implemented correctly. So I'm quite confident the "volumes" configuration is correct.

Is this a bug or am I missing something in the configuration?
Could you please document this feature?

Describe the expected behavior

The controller pod starts correctly

Controller Logs

Warning  FailedMount             6s (x10 over 4m16s)   kubelet                       MountVolume.SetUp failed for volume "secret" : secret "controller-manager" not found
  Warning  FailedMount             6s (x10 over 4m16s)   kubelet                       MountVolume.SetUp failed for volume "cert" : secret "actions-runner-controller-serving-cert" not found

Runner Pod Logs

-

Additional Context

No response

[mumoshu]

@extravio Hey! It seems like you somehow misconfigured your secret CSI driver, which resulted in the missing secret. Could you double-check the config of your CSI driver/provider and also make sure it works in a regular pod?
To be clear, ARC doesn't do anything special to support/un-support secrets store CSI driver. It should just work, as ARC has nothing to do with the secrets store CSI driver.
Please let me move this to Discussions so that this can turned into a community support case.

[cwong-archy]

@extravio any update? were you able to get this working in your environment? i am trying to do the same but unclear exactly what needs to be done.

i have a SecretProviderClass yaml file and applied via kubectl -

---
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: actions-runner-controller-secret-provider-class
spec:
  provider: aws
  secretObjects:
  - secretName: test01-eks-actions-runner-controller-secret
    type: Opaque
    data:
    - objectName: github_token
      key: github_token
  parameters:
    objects: |
      - objectName: "test01-eks-actions-runner-controller-secret"
        objectType: "secretsmanager"
        jmesPath:
          - path: github_token
            objectAlias: github_token

my values.yaml file for helm looks like this -

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxx:role/test01-eks-actions-runner-controller-role

authSecret:
  enabled: true
  create: true
  name: "controller-manager"

additionalVolumes:
- name: secret-store-inline
  csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
      secretProviderClass: "actions-runner-controller-secret-provider-class"

but its not working for me. am i missing the env: block? am i naming things correctly? the doc doesnt really explain it clearly for someone whos not a kubernetes expect.