x509: certificate signed by unknown authority #2658

Edarjak · 2023-06-09T09:22:03Z

Edarjak
Jun 9, 2023

Hi community,

I have deployed ARC last month successfully and it was forking fine for some time. Suddenly, after I tried to run CodeQL actions on the runners with the label "small" all runners from that namespace stopped working.

Here are the controller logs:

2023-06-09T09:06:48Z	ERROR	Reconciler error	{"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "Runner": {"name":"github-actions-runners-small-8dq7z-xmcqq","namespace":"github-actions-runners-small"}, "namespace": "github-actions-runners-small", "name": "github-actions-runners-small-8dq7z-xmcqq", "reconcileID": "402f2220-6360-4d21-b152-8df14096f41f", "error": "Internal error occurred: failed calling webhook \"mutate.runner.actions.summerwind.dev\": failed to call webhook: Post \"https://actions-runner-controller-webhook.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runner?timeout=10s\": x509: certificate signed by unknown authority"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

I've deleted other deployments as well (runners with different labels), got the same output:

2023-06-09T09:06:48Z	ERROR	runner	Unable to remove finalizer	{"runner": "github-actions-runners-micro/github-actions-runners-micro-bmcq8-2n7fn", "error": "Internal error occurred: failed calling webhook \"mutate.runner.actions.summerwind.dev\": failed to call webhook: Post \"https://actions-runner-controller-webhook.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runner?timeout=10s\": x509: certificate signed by unknown authority"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).processRunnerDeletion
	github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:271
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReconciler).Reconcile
	github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_controller.go:123
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-06-09T09:06:48Z	ERROR	Reconciler error	{"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "Runner": {"name":"github-actions-runners-micro-bmcq8-2n7fn","namespace":"github-actions-runners-micro"}, "namespace": "github-actions-runners-micro", "name": "github-actions-runners-micro-bmcq8-2n7fn", "reconcileID": "e731af85-18c3-4862-948d-28fb5d86303f", "error": "Internal error occurred: failed calling webhook \"mutate.runner.actions.summerwind.dev\": failed to call webhook: Post \"https://actions-runner-controller-webhook.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runner?timeout=10s\": x509: certificate signed by unknown authority"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023/06/09 09:06:48 http: TLS handshake error from 10.42.0.0:48898: remote error: tls: bad certificate

Here is one of my deployments:

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-actions-runners-small
  namespace: github-actions-runners-small
spec:
  template:
    spec:
      organization: org
      env:
        - name: GIT_SSL_CAINFO
          value: /etc/ssl/certs/root-ca.pem
      labels:
        - x-small
        - small
      group: Default
      resources:
        limits:
          cpu: "0.8"
          memory: "1.5Gi"
          ephemeral-storage: "60Gi"
        requests:
          cpu: "0.4"
          memory: "1.5Gi"
          ephemeral-storage: "60Gi"
      volumes:
        - name: ca-cert
          configMap:
            name: ca-cert
      volumeMounts:
        - name: ca-cert
          mountPath: /etc/ssl/certs/root-ca.pem
          subPath: root-ca.pem
          readOnly: true
      containers:
        - name: docker
          image: images.artifactory.myorgnet.com/local/docker:dind
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: github-actions-runners-small-autoscaler
  namespace: github-actions-runners-small
spec:
  # Runners in the targeted RunnerDeployment won't be scaled down
  # for 5 minutes instead of the default 10 minutes now
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    kind: RunnerDeployment
    # # In case the scale target is RunnerSet:
    # kind: RunnerSet
    name: github-actions-runners-small
  minReplicas: 4
  maxReplicas: 10
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'
    scaleDownThreshold: '0.25'
    scaleUpFactor: '2'
    scaleDownFactor: '0.5'

When I apply a deployment:

k3s kubectl apply -f github-actions-runners-small-autoscaling.yml
horizontalrunnerautoscaler.actions.summerwind.dev/github-actions-runners-small-autoscaler created
Error from server (InternalError): error when creating "github-actions-runners-small-autoscaling.yml": Internal error occurred: failed calling webhook "mutate.runnerdeployment.actions.summerwind.dev": failed to call webhook: Post "https://actions-runner-controller-webhook.actions-runner-system.svc:443/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment?timeout=10s": x509: certificate signed by unknown authority

Could someone help on this, please?

Thanks in advance,

Answered by Edarjak

Jun 13, 2023

SOLVED:

kubectl get validatingwebhookconfiguration -A
kubectl get mutatingwebhookconfiguration -A

kubectl delete mutatingwebhookconfiguration actions-runner-controller-mutating-webhook-configuration
kubectl delete validatingwebhookconfiguration actions-runner-controller-validating-webhook-configuration

View full answer

Edarjak · 2023-06-12T08:58:52Z

Edarjak
Jun 12, 2023
Author

UPDATE:

I have added additional pod, here are logs:

Defaulted container "manager" out of: manager, kube-rbac-proxy
2023-06-12T08:50:02Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2023-06-12T08:50:02Z	INFO	Initializing actions-runner-controller	{"version": "v0.27.1", "default-scale-down-delay": "10m0s", "sync-period": "1m0s", "default-runner-image": "repo.artifactory.myorg.com/summerwind/actions-runner:latest", "default-docker-image": "docker-images.artifactory.myorg.com/repo/docker:dind", "common-runnner-labels": null, "leader-election-enabled": true, "leader-election-id": "actions-runner-controller", "watch-namespace": ""}
2023-06-12T08:50:02Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2023-06-12T08:50:02Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2023-06-12T08:50:03Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2023-06-12T08:50:03Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2023-06-12T08:50:03Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-06-12T08:50:03Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-06-12T08:50:03Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-06-12T08:50:03Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2023-06-12T08:50:03Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-06-12T08:50:03Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-06-12T08:50:03Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-06-12T08:50:03Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2023-06-12T08:50:03Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-runner-set-pod"}
2023-06-12T08:50:03Z	INFO	starting manager
2023-06-12T08:50:03Z	INFO	controller-runtime.webhook.webhooks	Starting webhook server
2023-06-12T08:50:03Z	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
2023-06-12T08:50:03Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2023-06-12T08:50:03Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
2023-06-12T08:50:03Z	INFO	controller-runtime.webhook	Serving webhook server	{"host": "", "port": 9443}
I0612 08:50:03.102108       1 leaderelection.go:248] attempting to acquire leader lease actions-runner-system/actions-runner-controller...
2023/06/12 08:50:16 http: TLS handshake error from 10.42.0.0:58142: remote error: tls: bad certificate
2023/06/12 08:50:16 http: TLS handshake error from 10.42.0.0:58154: remote error: tls: bad certificate
2023/06/12 08:50:16 http: TLS handshake error from 10.42.0.0:58166: remote error: tls: bad certificate
2023/06/12 08:50:16 http: TLS handshake error from 10.42.0.0:58182: remote error: tls: bad certificate
2023/06/12 08:50:16 http: TLS handshake error from 10.42.0.0:58196: remote error: tls: bad certificate
2023/06/12 08:51:12 http: TLS handshake error from 10.42.0.0:62978: remote error: tls: bad certificate
2023/06/12 08:51:12 http: TLS handshake error from 10.42.0.0:62986: remote error: tls: bad certificate
2023/06/12 08:51:12 http: TLS handshake error from 10.42.0.0:62992: remote error: tls: bad certificate

@mumoshu maybe you could help on this, please?

0 replies

Edarjak · 2023-06-13T13:21:42Z

Edarjak
Jun 13, 2023
Author

SOLVED:

kubectl get validatingwebhookconfiguration -A
kubectl get mutatingwebhookconfiguration -A

kubectl delete mutatingwebhookconfiguration actions-runner-controller-mutating-webhook-configuration
kubectl delete validatingwebhookconfiguration actions-runner-controller-validating-webhook-configuration

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

x509: certificate signed by unknown authority #2658

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

x509: certificate signed by unknown authority #2658

Uh oh!

Edarjak Jun 9, 2023

Replies: 2 comments

Uh oh!

Edarjak Jun 12, 2023 Author

Uh oh!

Edarjak Jun 13, 2023 Author

Edarjak
Jun 9, 2023

Edarjak
Jun 12, 2023
Author

Edarjak
Jun 13, 2023
Author