Controller is taking several minutes of time to reconcile an increase in desired runners in RunnerDeployments

[cniessigma]

Hello all. I am having this issue where the Controller and Horizontal Runner Autoscaler will both work to set the correct number of desired runners for a RunnerDeployment, but actually reconciling that state takes several minutes, at which point it happens all at once

{"severity":"debug","ts":"2023-07-21T22:04:06Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 18","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":18,"reserved":0,"min":0,"max":50}
{"severity":"debug","ts":"2023-07-21T22:04:06Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 18 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":7,"workflow_runs_queued":11,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:04:07Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 18","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":18,"reserved":0,"min":0,"max":50,"last_scale_up_time":"2023-07-21 22:04:06 +0000 UTC","scale_down_delay_until":"2023-07-21T22:24:06Z"}
{"severity":"debug","ts":"2023-07-21T22:04:07Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 18 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":7,"workflow_runs_queued":11,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:07:16Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 18","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":18,"reserved":0,"min":0,"max":50,"last_scale_up_time":"2023-07-21 22:04:06 +0000 UTC","scale_down_delay_until":"2023-07-21T22:24:06Z"}
{"severity":"debug","ts":"2023-07-21T22:07:16Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 18 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":5,"workflow_runs_queued":13,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:08:19Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-7gm4c","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:19Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-pfvn9","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:20Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-gjg9d","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:21Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-5tmzm","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:22Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-dcf22","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:22Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-j557l","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:22Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-qzwv5","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-2p8kz","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-58xwb","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-5bmxs","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-k2nbz","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-rvtjd","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:08:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-t6w2n","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:10:31Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 22","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":22,"reserved":0,"min":0,"max":50}
{"severity":"debug","ts":"2023-07-21T22:10:31Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 22 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":17,"workflow_runs_queued":5,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:14:19Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 28","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":28,"reserved":0,"min":0,"max":50}
{"severity":"debug","ts":"2023-07-21T22:14:19Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 28 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":13,"workflow_runs_queued":15,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:18:37Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-8nh9z","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:37Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-qbkkn","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:38Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-56rw4","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:38Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-86dhv","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:38Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-k8nw4","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:38Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-pmtl2","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:38Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-tdxhv","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:38Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-zn4jx","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-5hts9","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-7n6dl","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-clsgf","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-d5qqf","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-f78dj","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-kbrmv","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-km89k","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-mj2wb","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-vgtbz","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:18:39Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-xrc6r","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:19:33Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 28","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":19,"reserved":1,"min":0,"max":50,"last_scale_up_time":"2023-07-21 22:14:19 +0000 UTC","scale_down_delay_until":"2023-07-21T22:34:19Z"}
{"severity":"debug","ts":"2023-07-21T22:19:33Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 19 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":14,"workflow_runs_queued":5,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:19:33Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-nf5qf","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:19:33Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-vnvhz","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:08Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 36","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":30,"reserved":6,"min":0,"max":50}
{"severity":"debug","ts":"2023-07-21T22:23:08Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 30 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":27,"workflow_runs_queued":3,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:23:08Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-8dz55","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:08Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-gntgg","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:08Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-p9kh8","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:09Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-4pjtp","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:09Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-559sz","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:09Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-dg6lw","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:09Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-k88cg","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:09Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-kl6cf","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:09Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-lrx68","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:09Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-pwfxj","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:09Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-qpwhw","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:23:09Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-td8b2","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:26:30Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 36","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":28,"reserved":0,"min":0,"max":50,"last_scale_up_time":"2023-07-21 22:23:08 +0000 UTC","scale_down_delay_until":"2023-07-21T22:43:08Z"}
{"severity":"debug","ts":"2023-07-21T22:26:30Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 28 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":27,"workflow_runs_queued":1,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:31:50Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 36","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":24,"reserved":0,"min":0,"max":50,"last_scale_up_time":"2023-07-21 22:23:08 +0000 UTC","scale_down_delay_until":"2023-07-21T22:43:08Z"}
{"severity":"debug","ts":"2023-07-21T22:31:50Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 24 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":23,"workflow_runs_queued":1,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:33:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-9r5gk","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-g5zzb","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-ghjhb","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-gjvpd","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-p6dpx","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-sc7nt","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:23Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-wwthf","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-2gfxj","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-4v8qc","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-b6bn9","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-fpwc5","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-jkgs8","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-n8gb5","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-r6cxn","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:33:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-wjv79","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:37:34Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 36","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":13,"reserved":0,"min":0,"max":50,"last_scale_up_time":"2023-07-21 22:23:08 +0000 UTC","scale_down_delay_until":"2023-07-21T22:43:08Z"}
{"severity":"debug","ts":"2023-07-21T22:37:34Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 13 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":13,"workflow_runs_queued":0,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:42:17Z","logger":"horizontalrunnerautoscaler","message":"Calculated desired replicas of 36","horizontalrunnerautoscaler":"gh-namespace/modular-runners-intel-normal-deployment-autoscaler","suggested":4,"reserved":0,"min":0,"max":50,"last_scale_up_time":"2023-07-21 22:23:08 +0000 UTC","scale_down_delay_until":"2023-07-21T22:43:08Z"}
{"severity":"debug","ts":"2023-07-21T22:42:17Z","logger":"horizontalrunnerautoscaler","message":"Suggested desired replicas of 4 by TotalNumberOfQueuedAndInProgressWorkflowRuns","workflow_runs_completed":0,"workflow_runs_in_progress":4,"workflow_runs_queued":0,"workflow_runs_unknown":0,"namespace":"gh-namespace","kind":"runnerdeployment","name":"modular-runners-intel-normal-deployment","horizontal_runner_autoscaler":"modular-runners-intel-normal-deployment-autoscaler"}
{"severity":"debug","ts":"2023-07-21T22:43:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-crb2h","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-fsmv6","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:24Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-x9lk9","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:25Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-74lf4","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:25Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-8886b","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:25Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-dgqk6","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:25Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-j2wp9","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:25Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-sqmzv","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:25Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-tk7dx","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:25Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-v8zh7","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-6ndvk","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-bwckp","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-bxtmt","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-drdb9","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-fw9rj","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-gcghr","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-kdzrp","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-pv5dg","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-s5l76","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-vhwkq","repository":"modularml/modular"}
{"severity":"debug","ts":"2023-07-21T22:43:26Z","logger":"runner","message":"Created runner pod","runner":"gh-namespace/modular-runners-intel-normal-deployment-lkk8w-z6gwn","repository":"modularml/modular"}

Some context is we have a lot of runner deployments (>50) and a lot of webhook activity, the logs above are a small sample. Ideally I'd be able to horizontally scale the controller but the default of having there need to be one leader is making this difficult. Any ideas on how to proceed or debug further?

To me this screams that the controller itself is overwhelmed when trying to reconcile state of so many resources