The arc-runner-set-754b578d-listener pod remains to be in pending state even if the resource limits and requests are specified.

[AkshaySuryawanshi8]

Hello everyone,

I'm seeking assistance in resolving an issue I'm facing. I've successfully installed the "arc controller," but I'm encountering a pending error when trying to install "arc-scale-set." I believe the problem might be related to missing resource configurations in my values.yml file. I've already added resource limit and request settings for the controller in the values.yml file.

I'm sharing the outputs of the describe commands for both the runner-set-listener and the controller below. If anyone could provide guidance on where exactly to include the resources block for limits and requests in the values.yml file, I would greatly appreciate it.

Listerner Pod described

➜   kubectl describe pods arc-runner-set-754b578d-listener -n arc-systems
Name:             arc-runner-set-754b578d-listener
Namespace:        arc-systems
Priority:         0
Service Account:  arc-runner-set-754b578d-listener
Node:             <none>
Labels:           actions.github.com/enterprise=
                  actions.github.com/organization=GithubRunnerTest
                  actions.github.com/repository=
                  actions.github.com/scale-set-name=arc-runner-set
                  actions.github.com/scale-set-namespace=arc-runners
                  app.kubernetes.io/component=runner-scale-set-listener
                  app.kubernetes.io/part-of=gha-runner-scale-set
                  app.kubernetes.io/version=0.4.0
                  runner-spec-hash=bb9f5c946
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Controlled By:    AutoscalingListener/arc-runner-set-754b578d-listener
Containers:
  autoscaler:
    Image:      ghcr.io/actions/gha-runner-scale-set-controller:0.4.0
    Port:       <none>
    Host Port:  <none>
    Command:
      /github-runnerscaleset-listener
    Environment:
      GITHUB_CONFIGURE_URL:                   https://github-qa.siemens.cloud/GithubRunnerTest
      GITHUB_EPHEMERAL_RUNNER_SET_NAMESPACE:  arc-runners
      GITHUB_EPHEMERAL_RUNNER_SET_NAME:       arc-runner-set-m8lst
      GITHUB_MAX_RUNNERS:                     4
      GITHUB_MIN_RUNNERS:                     2
      GITHUB_RUNNER_SCALE_SET_ID:             8
      http_proxy:                             <set to the key 'http_proxy' in secret 'arc-runner-set-754b578d-listener-proxy'>   Optional: false
      https_proxy:                            <set to the key 'https_proxy' in secret 'arc-runner-set-754b578d-listener-proxy'>  Optional: false
      no_proxy:                               <set to the key 'no_proxy' in secret 'arc-runner-set-754b578d-listener-proxy'>     Optional: false
      GITHUB_TOKEN:                           <set to the key 'github_token' in secret 'arc-runner-set-754b578d-listener'>       Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nw6g8 (ro)
Volumes:
  kube-api-access-nw6g8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From        Message
  ----     ------     ----                  ----        -------
  Warning  SyncError  4m22s (x21 over 42m)  pod-syncer  Error syncing to physical cluster: pods "arc-runner-set-754b578d-listener-x-arc-systems-x-vcluster" is forbidden: failed quota: namespace-limits: must specify limits.cpu for: autoscaler; limits.memory for: autoscaler; requests.cpu for: autoscaler; requests.memory for: autoscaler

Controler Pod described

➜  kubectl describe pod arc-gha-runner-scale-set-controller-84d858ddc-8gm6j -n arc-systems
Name:             arc-gha-runner-scale-set-controller-84d858ddc-8gm6j
Namespace:        arc-systems
Priority:         0
Service Account:  arc-gha-runner-scale-set-controller
Node:             ip-test-.eu-central-1.compute.internal/test
Start Time:       Mon, 14 Aug 2023 23:33:03 +0530
Labels:           app.kubernetes.io/component=controller-manager
                  app.kubernetes.io/instance=arc
                  app.kubernetes.io/name=gha-runner-scale-set-controller
                  app.kubernetes.io/part-of=gha-runner-scale-set-controller
                  app.kubernetes.io/version=0.4.0
                  pod-template-hash=84d858ddc
Annotations:      kubectl.kubernetes.io/default-container: manager
Status:           Running
IP:               192.168.251.221
IPs:
  IP:           192.168.251.221
Controlled By:  ReplicaSet/arc-gha-runner-scale-set-controller-84d858ddc
Containers:
  manager:
    Container ID:  containerd://7f3660d01c36fe372ad6126c318faef4fb9add83f0d2f600567153c5125b36a9
    Image:         ghcr.io/actions/gha-runner-scale-set-controller:0.4.0
    Image ID:      ghcr.io/actions/gha-runner-scale-set-controller@sha256:7dda34d5a99842769060dceaf51e4761c1000f6e8bfc489032e424d9d7a4e1b4
    Port:          <none>
    Host Port:     <none>
    Command:
      /manager
    Args:
      --auto-scaling-runner-set-only
      --log-level=debug
    State:          Running
      Started:      Mon, 14 Aug 2023 23:33:08 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  128Mi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      CONTROLLER_MANAGER_CONTAINER_IMAGE:             ghcr.io/actions/gha-runner-scale-set-controller:0.4.0
      CONTROLLER_MANAGER_POD_NAMESPACE:               arc-systems (v1:metadata.namespace)
      CONTROLLER_MANAGER_LISTENER_IMAGE_PULL_POLICY:  IfNotPresent
    Mounts:
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tzds5 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-tzds5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Guaranteed
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  45m   default-scheduler  Successfully assigned arc-systems/arc-gha-runner-scale-set-controller-84d858ddc-8gm6j to ip-100-1-0-226.eu-central-1.compute.internal
  Normal  Pulled     45m   kubelet            Container image "ghcr.io/actions/gha-runner-scale-set-controller:0.4.0" already present on machine
  Normal  Created    45m   kubelet            Created container manager
  Normal  Started    45m   kubelet            Started container manager

[matteovivona]

I've had the same problem on some repos in my org for the past few days. If I trigger other workflows in another repo, 2 pods are created: one for the triggered workflow and the other for the pending one.

Also, I thought I understood that the GITHUB_MIN_RUNNERS value referred to the always-on pod replicas