ERROR: Reconciler error the object has been modified; please apply your changes to the latest version and try again

[semihural-tomtom]

What might be the reason of this error?

 helm ls -n actions-runner-system
NAME                     	NAMESPACE            	REVISION	UPDATED                                	STATUS  	CHART                           	APP VERSION
actions-runner-controller	actions-runner-system	8       	2023-04-12 07:49:56.653364389 +0000 UTC	deployed	actions-runner-controller-0.23.1	0.27.2

• We replace the CRD
• We upgraded helm with 0.23.1
• Our controller version 0.27.2

2023-04-12T13:29:22Z	ERROR	Reconciler error	{"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "RunnerReplicaSet": {"name":"gha-staging-8c-32gb-n8vkw","namespace":"staging-8c-32gb"}, "namespace": "staging-8c-32gb", "name": "gha-staging-8c-32gb-n8vkw", "reconcileID": "77e7e91c-56f0-4b16-a5d4-850d768805aa", "error": "Operation cannot be fulfilled on runners.actions.summerwind.dev \"gha-staging-8c-32gb-n8vkw-rxftt\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

[schniedergers]

We're seeing the same - starting 50 small identical jobs in parallel, a small number fails and we're seeing this in the log.
Controller version 0.27.3.

2023-05-10T20:19:21Z    ERROR   runnerdeployment        Failed to update runnerreplicaset resource      {"runnerdeployment": "actions-runner-system/alltrails-runner-deploy-small", "error": "Operation cannot be
fulfilled on runnerreplicasets.actions.summerwind.dev \"alltrails-runner-deploy-small-5mwth\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerDeploymentReconciler).Reconcile
        github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runnerdeployment_controller.go:199
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-05-10T20:19:21Z    ERROR   Reconciler error        {"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "RunnerDeployment": {"nam
e":"alltrails-runner-deploy-small","namespace":"actions-runner-system"}, "namespace": "actions-runner-system", "name": "alltrails-runner-deploy-small", "reconcileID": "d6b8d2f6-935b-4693-aea9-a5dbbb3ad9f1", "er
ror": "Operation cannot be fulfilled on runnerreplicasets.actions.summerwind.dev \"alltrails-runner-deploy-small-5mwth\": the object has been modified; please apply your changes to the latest version and try ag
ain"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

[shaoqin2]

This is a very common error we see not just in ARC but k8s clients in general, any call to modify state should ideally retry on conflict. Busy cluster seems to suffer more and short bursts of modification

https://pkg.go.dev/k8s.io/client-go/util/retry#RetryOnConflict

[semihural-tomtom]

@shaoqin2 any workaround or shall we ignore?