ActionsMetricsServer not publishing new metrics on GHES

[Samsanga]

Controller Version

0.27.4

Helm Chart Version

0.23.3

I am unable to get new metrics published from actions-runner-controller-actions-metrics-server on GHES 3.8.2 using the following configuration. I am using webhook-based autoscaling in the EKS cluster with ARC.

• Has anyone managed to get actions-metrics-server working on GHES?
• Using which endpoint can we see the workflow metrics exposed by new actions-metrics-server?

authSecret:
  github_app_id: <hidden>
  github_app_installation_id: <hidden>
  github_app_private_key: |
    <hidden>
  github_token: <hidden>
githubEnterpriseServerURL: https://github.xyz.com
certManagerEnabled: true
scope:
  singleNamespace: false
githubWebhookServer:
  enabled: true
  secret:
    enabled: true
    create: true
    name: github-webhook-server
    github_app_id: <hidden>
    github_app_installation_id: <hidden>
    github_app_private_key: |
      <hidden>
    github_webhook_secret_token: <hidden>
  useRunnerGroupsVisibility: true
  service:
    type: LoadBalancer
    ports:
      - port: 443
        targetPort: http
        protocol: TCP
        name: http
        nodePort: 31000
actionsMetricsServer:
  enabled: true
  secret:
    enabled: true
    name: github-webhook-server
    github_webhook_secret_token: <hidden>
  serviceAccount:
    create: true
  service:
    type: LoadBalancer
    ports:
      - port: 443
        targetPort: http
        protocol: TCP
        name: http
        nodePort: 32001
actionsMetrics:
  serviceMonitor: true
  proxy:
    enabled: false
metrics:
  serviceMonitor: true
  port: 8002
  serviceMonitorLabels: {release: prometheus-stack}
  proxy:
    enabled: false

Runners are deployed using RunnerDeployment and HorizontalRunnerAutoscaler.

kind: HorizontalRunnerAutoscaler
spec:
  scaleTargetRef:
    kind: RunnerDeployment
    name: RunnerDeployment-Name
  scaleUpTriggers:
    - githubEvent:
        workflowJob: {}

NAMESPACE               NAME                                                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                        AGE
actions-runner-system   actions-runner-controller-actions-metrics-server     LoadBalancer   172.20.187.183   arc-actions-metrics-server.elb.eu-west-1.amazonaws.com                          443:32001/TCP,8002:32586/TCP   7m55s
actions-runner-system   actions-runner-controller-github-webhook-server      LoadBalancer   172.20.202.213   arc-github-webhook-server.elb.eu-west-1.amazonaws.com                           443:31000/TCP,8002:31363/TCP   7m55s
actions-runner-system   actions-runner-controller-metrics-service            LoadBalancer   172.20.211.176   arc-metrics-service.elb.eu-west-1.amazonaws.com                                 443:31348/TCP,8002:30012/TCP   7m55s
actions-runner-system   actions-runner-controller-webhook                    ClusterIP      172.20.27.113    <none>                                                                          443/TCP                        7m55s

For actions-metrics-server service endpoint, I have setup an organization level hook for actions-metrics-server as suggested in Configuring GitHub for sending webhooks for our newly created webhook server section. I have also tried to setup a seperate GitHub App for actions-metrics-server, but no luck.

$ curl -k https://arc-actions-metrics-server.elb.eu-west-1.amazonaws.com/metrics
actions-metrics-server is running

$ curl -k https://arc-github-webhook-server.elb.eu-west-1.amazonaws.com
webhook server is running

Though actions-runner-controller-actions-metrics-server is running normally and receiving webhooks. But I am not seeing any additional or new metrics made available by actions-runner-controller-metrics-service.

$ kubectl logs -l app.kubernetes.io/instance=actions-runner-controller-actions-metrics-server -n actions-runner-system -f
2023-05-15T16:35:40Z	INFO	workflowjobmetrics-webhookserver	ok	{"event": "workflow_job", "hookID": "147", "delivery": "870c3700-f33e-11ed-835a-623e26cfab14"}
2023-05-15T16:35:41Z	INFO	workflowjobmetrics-eventreader	reading workflow_job logs	{"job_id": "17486", "job_name": "Sleep for 720 Seconds", "repository": "test-wokflows", "repository_full_name": "Sangam/test-wokflows", "owner": "Sangam", "job_id": "17486", "job_name": "Sleep for 720 Seconds", "repository": "test-wokflows", "repository_full_name": "Sangam/test-wokflows", "owner": "Sangam", "exit_code": "null"}
2023-05-15T16:36:02Z	INFO	workflowjobmetrics-webhookserver	ok	{"event": "workflow_job", "hookID": "147", "delivery": "93f43350-f33e-11ed-8d3f-39849e3fec1c"}
2023-05-15T16:36:16Z	INFO	workflowjobmetrics-webhookserver	ok	{"event": "workflow_job", "hookID": "147", "delivery": "9cb01220-f33e-11ed-9344-32756df1ab41"}
2023-05-15T16:36:17Z	INFO	workflowjobmetrics-eventreader	reading workflow_job logs	{"job_id": "17489", "job_name": "Sleep for 300 Seconds", "repository": "test-wokflows", "repository_full_name": "Sangam/test-wokflows", "owner": "Sangam"}

$ curl -k https://arc-metrics-service.elb.eu-west-1.amazonaws.com/metrics
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 1
...
...
workqueue_work_duration_seconds_bucket{name="runnerset-controller",le="+Inf"} 0
workqueue_work_duration_seconds_sum{name="runnerset-controller"} 0
workqueue_work_duration_seconds_count{name="runnerset-controller"} 0

$ curl -ks https://arc-metrics-service.elb.eu-west-1.amazonaws.com/metrics | grep -i horizontalrunnerautoscaler_
# HELP horizontalrunnerautoscaler_spec_max_replicas maxReplicas of HorizontalRunnerAutoscaler
# TYPE horizontalrunnerautoscaler_spec_max_replicas gauge
horizontalrunnerautoscaler_spec_max_replicas{horizontalrunnerautoscaler="sangam-runners",namespace="sangam-cloud-gurus"} 20
# HELP horizontalrunnerautoscaler_spec_min_replicas minReplicas of HorizontalRunnerAutoscaler
# TYPE horizontalrunnerautoscaler_spec_min_replicas gauge
horizontalrunnerautoscaler_spec_min_replicas{horizontalrunnerautoscaler="sangam-runners",namespace="sangam-cloud-gurus"} 1
# HELP horizontalrunnerautoscaler_status_desired_replicas desiredReplicas of HorizontalRunnerAutoscaler
# TYPE horizontalrunnerautoscaler_status_desired_replicas gauge
horizontalrunnerautoscaler_status_desired_replicas{horizontalrunnerautoscaler="sangam-runners",namespace="sangam-cloud-gurus"} 1

# No additional metrics related to github_workflow_job* 😞 

$ curl -ks https://arc-metrics-service.elb.eu-west-1.amazonaws.com/metrics | grep -i workflow
$

[ColinHeathman]

@sangameshajm Did you set up webhook events from Github to both the webhook-server and the metrics-server? There isn't any data sharing so Github needs to be configured to send webhooks to both.

I have also tried to setup a seperate GitHub App for actions-metrics-server, but no luck.

I think if you want to use seperate credentials for actions-metrics-server they need to be also configured in the Helm chart.