action runner controller restart to reconcile terminating runner pods

[gbadola1]

Hello Team,

We implemented ARC on OpenShift 4.10 using helm. Today we upgraded OpenShift to 4.12. Some the of the runners are in terminating state will not be deleted no matter what. These runner are causing ARC to not able to reconcile because of there state. I have tried reinstalling ARC but has not helped. Any idea/suggestions will be greatly appreciated.
Here is the ARC log
2023-09-01T22:27:24Z ERROR Reconciler error {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"runner-java11-ent-7mjml-n67xk","namespace":"enterprise-arc-runners"}, "namespace": "enterprise-arc-runners", "name": "runner-java11-ent-7mjml-n67xk", "reconcileID": "30c819a5-b84e-42d0-8f93-2d3495447af3", "error": "strconv.ParseInt: parsing "": invalid syntax"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-09-01T22:27:24Z ERROR Reconciler error {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"runner-java17-ent-cdcfh-6bs6s","namespace":"enterprise-arc-runners"}, "namespace": "enterprise-arc-runners", "name": "runner-java17-ent-cdcfh-6bs6s", "reconcileID": "364495ff-1cfe-4998-8e1c-9927335bbd65", "error": "strconv.ParseInt: parsing "": invalid syntax"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-09-01T22:27:24Z ERROR Reconciler error {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"runner-java8-ent-x4ts7-cjqb6","namespace":"enterprise-arc-runners"}, "namespace": "enterprise-arc-runners", "name": "runner-java8-ent-x4ts7-cjqb6", "reconcileID": "ed24ffae-a36f-422a-a171-944b8566a320", "error": "strconv.ParseInt: parsing "": invalid syntax"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-09-01T22:27:25Z ERROR Reconciler error {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"runner-java17-ent-cdcfh-f2gzt","namespace":"enterprise-arc-runners"}, "namespace": "enterprise-arc-runners", "name": "runner-java17-ent-cdcfh-f2gzt", "reconcileID": "bd54f35b-0e9a-4a18-a390-d0d1e50242fd", "error": "strconv.ParseInt: parsing "": invalid syntax"}