feat: Add Pod Disruption Schedule

[jukie]

Fixes #1719
Related to #2744 (RFC)

Description
This is a renewal of #1720 and adds the ability to define a pod disruption schedule at the pod-level which behaves somewhat akin to nodepool disruption budgets.

Note: This was mostly AI generated but I have reviewed and understand the implementation. I've built a custom controller that runs alongside Karpenter with similar logic and have previously added this functionality in #1720.

How was this change tested?
Add tests and will run inside a real cluster and update this comment with validation

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jukie
Once this PR has been reviewed and has the lgtm label, please assign njtran for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @jukie. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coveralls]

Pull Request Test Coverage Report for Build 20545058840

Details

• 88 of 89 (98.88%) changed or added relevant lines in 13 files are covered.
• 2 unchanged lines in 1 file lost coverage.
• Overall coverage increased (+0.05%) to 80.518%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controllers/state/statenode.go	13	14	92.86%

Files with Coverage Reduction	New Missed Lines	%
pkg/controllers/static/provisioning/controller.go	2	60.0%

Totals
Change from base Build 20489292152:	0.05%
Covered Lines:	12052
Relevant Lines:	14968

---

💛 - Coveralls

[ellistarn]

I think we want to think holistically about this space of pod oriented disruption controls, including whether or not these features should land in upstream's PDB.

Happy to discuss more in working group.

[jukie]

Having something in upstream PDBs would be great and I can start a KEP or design spec for that as well but in the best case scenario I suspect it'd be over a year before that becomes available. Would you be open to solving this within Karpenter first to figure out the best model which can then be added upstream? Perhaps you'd prefer the CRD approach described in my RFC vs this example?

This issues has been open since 2024 along with similar issues describing the desire for more pod-level disruption controls. I agree that we should think holistically about a solution but would also like to get some sense of progress in the area. My proposal would solve a real problem that exists in Karpenter where it's currently unsafe to use do-not-disrupt for anything besides ephemeral workloads that eventually exit on their own and doing so risks violating PDBs upon Nodes reaching max expiration + grace period. As a result of hitting that in the real world I currently maintain a custom controller as a workaround.

[ellistarn]

I wonder if we might want to consider an alpha API in Karpenters group that is our proposal for an upstream KEP and we target eventually landing it upstream. It might not be able to solve everything (I.e. surge eviction), but could be a good middle ground to get things moving.