feat: Add option to enabled warmup when leader election is used

[GnatorX]

Fixes #N/A

Description

• Add option to enable controller runtime warm up when leader election is used. Warm up allows non-leader pods to warm up k8s cache and watch for events hydrating the informer cache even when it isn't the leader. This allows faster fail over (don't have to hydrate informer after leader changes).

How was this change tested?

• Ran with the change. Follower memory didn't increase dramatically (note that this is simply the informer cache, any subsequent caches that is built after pod is created isn't ran until it becomes leader)
• Attempted to force a leader election by killing the leader and making sure follower comes up normally

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coveralls]

Pull Request Test Coverage Report for Build 21188536214

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 2 of 2 (100.0%) changed or added relevant lines in 2 files are covered.
• 25 unchanged lines in 5 files lost coverage.
• Overall coverage increased (+0.2%) to 80.742%

---

Files with Coverage Reduction	New Missed Lines	%
pkg/controllers/disruption/consolidation.go	4	88.0%
pkg/controllers/nodeoverlay/controller.go	4	75.42%
pkg/controllers/state/informer/nodeclaim.go	4	72.73%
pkg/controllers/metrics/nodepool/controller.go	6	87.27%
pkg/controllers/controllers.go	7	0.0%

Totals
Change from base Build 21011765273:	0.2%
Covered Lines:	11756
Relevant Lines:	14560

---

💛 - Coveralls

[ellistarn]

Any reason to not have this on by default?

There's a network tradeoff, but it seems very worthy given the risk of coordinated lists across controllers during zonal fail over.

[GnatorX]

Same question was asked during working group. I think it make sense to turn it on by default but I think its likely safer to go first as not default then maybe flip it on as default after a release or two?

[ellistarn]

Sounds good. If our intended goal state is on by default, wdyt about flipping the flag to "disable" and making it true by default (for now).

The general idea being that the natural state of a Boolean is false. We haven't been perfect with this, but I think it generally helps with readability (short of something more sophisticated like feature gates).

@jmdeal I'm noticing that feature gates has a comment to deprecate the field. What's the history there?

[GnatorX]

That make sense to me, I can update it to do that. I also needed to test one more code path then will ask for another review

[ellistarn]

/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ellistarn, GnatorX

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ellistarn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment