Horizontal autoscaling for sharded controllers

[timebertt]

What would you like to be added:

Let's try setting up horizontal autoscaling for a sharded controller.

Probably the controller SLIs (queue and reconciliation latency) are good candidate metrics for driving autoscaling.

A corresponding experiment load test scenario should evaluate the proposed setup.

Why is this needed:

Controller setups with highly dynamic loads could benefit from horizontal autoscaling.
Setting this up, evaluating it, and documenting the results would be great for (potential) adopters of the project.