Deployment of action runner controller through Argo CD gives Updating registration token failed issue when we resync it again

[rmalakar8x8]

Checks

• I've already read https://github.com/actions-runner-controller/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.26.0

Helm Chart Version

0.21.1

CertManager Version

No response

Deployment Method

ArgoCD

cert-manager installation

yes followed the steps mentioned in https://github.com/actions-runner-controller/actions-runner-controller

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

actions-runner-controller:
  env: 
    - name: GITHUB_TOKEN
      valueFrom:
        secretKeyRef:
          key: github_token
          name: my-secret-name
      
    
  authSecret:
    enabled: false
    create: false
    name: "controller-manager"
    annotations: {}

To Reproduce

Once this is worked but now its not working when i again deploy helm chart using argo cd

Describe the bug

2022-11-15T13:50:51Z DEBUG actions-runner-controller.horizontalrunnerautoscaler Suggested desired replicas of 0 by TotalNumberOfQueuedAndInProgressWorkflowRuns {"workflow_runs_completed": 0, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0, "namespace": "actions-runner-controller", "kind": "runnerdeployment", "name": "self-hosted-runner", "horizontal_runner_autoscaler": "runner-deployment-autoscaler"}
2022-11-15T13:50:51Z DEBUG actions-runner-controller.horizontalrunnerautoscaler Calculated desired replicas of 1 {"horizontalrunnerautoscaler": "actions-runner-controller/runner-deployment-autoscaler", "suggested": 1, "reserved": 0, "min": 1, "max": 5}
2022-11-15T13:51:09Z DEBUG actions-runner-controller.runnerreplicaset Skipped reconcilation because owner is not synced yet {"runnerreplicaset": "actions-runner-controller/self-hosted-runner-nq7g9", "owner": "actions-runner-controller/self-hosted-runner-nq7g9-62x72", "pods": null}
2022-11-15T13:51:09Z ERROR actions-runner-controller.runner Failed to get new registration token {"runner": "self-hosted-runner-nq7g9-62x72", "error": "failed to create registration token: POST https://api.github.com/repos/8x8/auto_gitops_infra8-workflow-test-tree/actions/runners/registration-token: 404 Not Found []"}
github.com/actions-runner-controller/actions-runner-controller/controllers.(*RunnerReconciler).processRunnerCreation
/workspace/controllers/runner_controller.go:255
github.com/actions-runner-controller/actions-runner-controller/controllers.(*RunnerReconciler).Reconcile
/workspace/controllers/runner_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234

Describe the expected behavior

It should work with github token , i have check that token permission its all correct and work with manual helm deployment.
When using the same with argo cd deployment i am facing above issue

Whole Controller Logs

/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
2022-11-15T13:54:47Z	DEBUG	events	Warning	{"object": {"kind":"Runner","namespace":"actions-runner-controller","name":"self-hosted-runner-nq7g9-62x72","uid":"e4c81364-aa10-4ab7-a6da-f23c88f015f1","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"10558531"}, "reason": "FailedUpdateRegistrationToken", "message": "Updating registration token failed"}
2022-11-15T13:55:10Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Suggested desired replicas of 0 by TotalNumberOfQueuedAndInProgressWorkflowRuns	{"workflow_runs_completed": 0, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0, "namespace": "actions-runner-controller", "kind": "runnerdeployment", "name": "self-hosted-runner", "horizontal_runner_autoscaler": "runner-deployment-autoscaler"}
2022-11-15T13:55:10Z	DEBUG	actions-runner-controller.horizontalrunnerautoscaler	Calculated desired replicas of 1	{"horizontalrunnerautoscaler": "actions-runner-controller/runner-deployment-autoscaler", "suggested": 1, "reserved": 0, "min": 1, "max": 5}
2022-11-15T13:55:42Z	DEBUG	actions-runner-controller.runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "actions-runner-controller/self-hosted-runner-nq7g9", "owner": "actions-runner-controller/self-hosted-runner-nq7g9-62x72", "pods": null}
2022-11-15T13:55:42Z	ERROR	actions-runner-controller.runner	Failed to get new registration token	{"runner": "self-hosted-runner-nq7g9-62x72", "error": "failed to create registration token: POST https://api.github.com/repos/8x8/auto_gitops_infra8-workflow-test-tree/actions/runners/registration-token: 404 Not Found []"}
github.com/actions-runner-controller/actions-runner-controller/controllers.(*RunnerReconciler).processRunnerCreation
	/workspace/controllers/runner_controller.go:255
github.com/actions-runner-controller/actions-runner-controller/controllers.(*RunnerReconciler).Reconcile
	/workspace/controllers/runner_controller.go:136
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234

Whole Runner Pod Logs

There is not runner pod is getting created due to above issue

Additional Context

No response

[github-actions[bot]]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[mumoshu]

@rmalakar8x8 It's more likely that you specified an incorrect token or the token has already expired. Please try manually calling any GitHub API or even better the create-regstration-token API using curl or some other tool, with the token you specified, to verify that the token is actually working.
That said, this doesn't look like a valid bug report atm. Can I convert this to a Discussion so that we can clearly communicate that Q&A should be done in Discussions and you should open a bug report only after it turned out to be a bug?

[rmalakar8x8]

I have checked the token it has all the required permission as per documentation even it works sometimes and when i resync the argo app with some automation changes I do with helm I start getting Updating registration token failed

Note: sync argo project with replace option checked as per forum discussions.

Let me know is there any cache or something maintained at action-runner-controller which gets disturbed after a resync at argo ?
Since the manual deployment of action-runner-controller helm using upgrade command works fine and its stable,
the deployment with argo is giving this problem

Let me know the right way of deployment and resync through argo cd =

[ross-at-finix]

I have this issue deploying through helm with terraform and can definitely get a registration token through the organization API. Failed registration token just about every time.

[rmalakar8x8]

this issue is resolved when we make the user (the token belongs to )as admin at the repo.