diff --git a/keps/sig-apps/5682-pdbs-for-multipod-replicas/README.md b/keps/sig-apps/5682-pdbs-for-multipod-replicas/README.md new file mode 100644 index 00000000000..71b7bb68be3 --- /dev/null +++ b/keps/sig-apps/5682-pdbs-for-multipod-replicas/README.md @@ -0,0 +1,1226 @@ + +# KEP-5682: PDB for Multi-Pod Replicas + + + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) within one minor version of promotion to GA +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +Voluntary disruptions (node drains) will evict pods from a node, potentially causing issues in an application that relies on having a certain one or more replicas running. To specify that a certain number (or percentage) of pods must remain available, users may create a `PodDisruptionBudget` (PBD) object and declare a `minAvailable` or `maxUnavailable` in its spec. Then, if a pod eviction would violate the availability threshold given by the PDB, the disruption controller will block the eviction and protect the availability of the application. + +However, some applications will use `PodGroups` as defined in the new [Workload API](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/4671-gang-scheduling), in which a group of pods acts as a single "superpod" entity (i.e., each replica is composed of multiple pods). These require more complex eviction logic to protect from disruptions. For example, in a [LeaderWorkerSet](https://lws.sigs.k8s.io/docs/overview/) running a distributed ML training job, one pod in a group being evicted would cause the entire group to fail. + +This KEP will allow the Eviction API to treat each pod group as if it were a single replica when calculating availability for a PDB. To enable this new behavior, the PDB spec will have optional boolean `usePodGroups`, and if `true`, the PDB will enforce a number of *pod group replicas* that must remain available, rather than a number of *individual pod replicas* as it is now. + +**Note: as of this draft, the Workload API is still in progress, for this KEP we assume it is fully implemented** + +## Motivation + + + +The goal of this KEP is to improve the experience of using PDBs and the Eviction API for applications with multi-pod replicas. Most importantly, eviction of a small number of pods spread across multiple multi-pod replicas could disrupt each replica. This will be prevented by new functionality for calculating avaiability for eviction based on disrupted pod groups, rather than individual pods. + +### Goals + + + +- **Introduce a field to enable group-based PDBs:** Add a new boolean field `usePodGroups` to the `PodDisruptionBudget.spec`. +- **Define availability for pod groups:** Allow application owners to define PDBs for multi-pod replicas (as defined by the `Workload` API) rather than individual pods. +- **Update eviction logic:** When enabled, the eviction logic will interpret the disruption budget given in the PDB (`minAvailable` or `maxUnavailable`) as a count of pod group replicas, rather than individual pod replicas. +- **Integrate with Workload API:** Use the pod spec's `workload.name` and `workload.podGroup` to retrieve `Workload` objects and their `PodGroup` groupings. +- **Maintain compatibility:** Ensure that common cluster operations that respect PDBs, such as `kubectl drain` and node drains initiated by `cluster-autoscaler`, follow group-based disruption budgets when enabled. +- **Preserve existing functionality:** For backward compatibility, the behavior of PDBs without `usePodGroups: true` will be unchanged. + +### Non-Goals + + + +This change will only affect the Eviction API. Involuntary disruptions do not use the Eviction API include: manual pod deletion, pods being deleted by their owner (e.g. during Deployment rollout), node failure, pod cleanup due to a node being removed from the cluster, evictions by the Kubelet due to node pressure (e.g. memory shortage), taint manager deleting NoExecute tainted pods + +The only change to k8s resource definitions will be the optional field in the `PodDisruptionBudget` spec to enable the changes. + +This change will not affect the behavior of controllers for `Deployment`, `StatefulSet`, `Workload`, or `LeaderWorkerSet`, which will be responsible for setting the `workload.name` and `workload.podGroup` on their managed pods. The lifecycle and recovery of a disrupted replica is the responsibility of their owning controller. + +There will be no additions or changes to scheduling, including gang scheduling. This only handles eviction of already-scheduled pods. + +This KEP follows the definition of multi-pod replica health using `minCount` from the `Workload` API. A replica is considered healthy if it meets `minCount` of healthy pods, and unhealthy otherwise. We are not introducing any other definition of partial health (e.g. a percentage). + +If a PDB has multi-pod replicas enabled, individual pods without an assigned workload will be treated as single-pod groups. We will log a warning as mixing types is not recommended, but the user is responsible for correct setup of replicas. + +## Proposal + + + +We will add a new optional boolean `usePodGroups` to the `PodDisruptionBudget.spec`. If this field is unset or `false` (default), the Eviction API will evaluate the PDB based on individual pod counts, preserving all existing behavior. If `true`, the Eviction API will find the `Workload` object and its `PodGroup` as specified by the Pod spec. This `PodGroup` defines the minimum number of pods required for a replica of that group to be healthy, and how many replicas are expected. Using this information, the PDB's `minAvailable` or `maxUnavailable` will be interpreted in terms of these `PodGroup` replicas, rather than individual pods. + +### User Stories (Optional) + + +*If the user is not using the `Workload` API, their process will be unaffected.* + +#### Story 1: Distributed Workload + +An ML engineer is running distributed training jobs using Workload API. The `Workload` defines a `PodGroup` named `worker` with `replicas: 10` and `policy.gang.minCount: 8`. This means the job has 10 replicas, each consisting of at least 8 pods. + +To protect this long-running job from voluntary disruptions, the user wants to ensure at least 9 of the 10 worker groups remain available. + +This user would create a PDB targeting the worker pods: + +```yaml +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: my-training-job-workers-pdb +spec: + minAvailable: 9 + usePodGroups: true # <-- New field to enable + selector: + matchLabels: + # Assuming pods are labeled + workload: my-training-job + pod-group: worker +``` + +Upon node drain, the Eviction API will: +1. See the PDB `my-training-job-workers-pdb` with `spec.usePodGroups: true`. +2. Select all pods matching the selector. +3. Detect that these pods have `spec.workload.name: my-training-job` and `spec.workload.podGroup: worker`. +4. Fetch the `Workload` object `my-training-job`. +5. Find `worker` `PodGroup` in the `Workload`, which has 10 `replicas` and 8 `minSize`. +6. Interpreting `minAvailable: 9` as pod groups, a group is considered disrupted if evicting a pod would cause its healthy pod count to drop below 8. +7. The drain will proceed only if it does not cause the number of available worker groups to drop below 9. + +This way, the job is protected to run with sufficient replicas during cluster maintenance. + +#### Story 2: Cluster Maintenance + +A cluster administrator frequently drains nodes for upgrades. The cluster has various workloads, including multi-pod applications defined by the `Workload` API. + +The admin would like to upgrade a node which is running the job from Story 1. To perform node drains safely, they rely on application owners' PDBs. When they issue `kubectl drain `, the Eviction API sees the PDB and uses the process above, interpreting the disruption in terms of `PodGroup` replicas and ensuring that the drain does not violate the application's group-based availability requirements. + +This allows safe maintenance without causing outages, as the drain will pause if it cannot evict pods without violating a group-based PDB. It will wait for better replica health, more availability, lower PDB requirements, or the admin may contact the application owner to resolve the block. + +#### Simplified Setup Example + +```mermaid +graph TD + %% Define Styles + classDef node_box fill:#ececff,stroke:#9696ff,stroke-width:2px,color:#1a1a1a + classDef replica_box fill:#f9f9f9,stroke:#aaa,stroke-width:1px,color:#1a1a1a + classDef pod_box fill:#fff,stroke:#ccc,color:#1a1a1a + + subgraph NodeToDrain ["Node (Being Drained)"] + direction LR %% Arrange replicas side-by-side + + subgraph Replica0 ["Replica 0"] + P0A("Pod 0A") + P0B("Pod 0B") + end + class Replica0 replica_box + + subgraph Replica1 ["Replica 1"] + P1A("Pod 1A") + P1B("Pod 1B") + end + class Replica1 replica_box + + end + + class NodeToDrain node_box + class P0A,P0B,P1A,P1B pod_box +``` + +In this setup, the node being drained contains two replicas, each with two pods (there may be more nodes and replicas which we can ignore). The PDB wants at most one replica unavailable. Currently, the user might try `minUnavailable: 2` (one two-pod replica unavailable). The node drain would start, and could evict a pod from replica 0 and a pod from replica 1 before pausing (as there are only 2 pods left). This would disrupt both replicas. With the new changes, a PDB with `usePodGroups: true` and `minUnavailable: 1` (one replica unavailable) would pause before evicting a pod from the second replica, protecting one of the replicas as intended. + +In a real cluster, there may be additional nodes or replicas, pods from other jobs sharing those nodes, etc. + + +```mermaid +graph TD + %% Define Styles for Flowchart Diagram + classDef action fill:#e6f3ff,stroke:#66b3ff,stroke-width:2px,color:#111 + classDef decision fill:#fff0e6,stroke:#ff9933,stroke-width:2px,color:#111 + classDef pdb_spec fill:#ffccff,stroke:#cc00cc,stroke-width:2px,color:#111 + classDef outcome_bad fill:#fff0f0,stroke:#ffaaaa,stroke-width:2px,color:#111 + classDef outcome_good fill:#f0fff0,stroke:#aaffaa,stroke-width:2px,color:#111 + classDef process fill:#f0f0f0,stroke:#ccc,color:#111 + + %% --- Start --- + StartDrain("kubectl drain node") + class StartDrain action + + StartDrain --> PDB_Type{"PDB"} + class PDB_Type decision + + %% --- Path 1: Traditional PDB --- + PDB_Type -- "Traditional PDB" --> PDB_Old(PDB Spec:
maxUnavailable: 2 pods
usePodGroups: false) + class PDB_Old pdb_spec + + PDB_Old --> TryEvictP0A("Try to evict Pod 0A
(from Replica 0)") + class TryEvictP0A action + + TryEvictP0A --> CheckPods1{"Unavailable pods (1) <= 2?"} + class CheckPods1 decision + + CheckPods1 -- "Yes (1 <= 2)" --> EvictP0A("Eviction Allowed") + class EvictP0A process + + EvictP0A --> TryEvictP1A("Try to evict Pod 1A
(from Replica 1)") + class TryEvictP1A action + + TryEvictP1A --> CheckPods2{"Unavailable pods (2) <= 2?"} + class CheckPods2 decision + + CheckPods2 -- "Yes (2 <= 2)" --> EvictP1A("Eviction Allowed") + class EvictP1A process + + EvictP1A --> DrainStops("Drain Pauses
(PDB limit reached)") + class DrainStops action + + DrainStops --> AppDown("Application State:
Both replicas are broken
(One pod lost from each)") + class AppDown outcome_bad + + %% --- Path 2: Group-Aware PDB (KEP) --- + PDB_Type -- "Group-Aware PDB (KEP)" --> PDB_New(PDB Spec:
maxUnavailable: 1 group
usePodGroups: true) + class PDB_New pdb_spec + + PDB_New --> TryEvictP0A_New("Try to evict Pod 0A
(from Replica 0)") + class TryEvictP0A_New action + + TryEvictP0A_New --> CheckGroups1{"Eviction breaks Replica 0.
Unavailable groups (1) <= 1?"} + class CheckGroups1 decision + + CheckGroups1 -- "Yes (1 <= 1)" --> EvictR0("Eviction Allowed") + class EvictR0 process + + EvictR0 --> TryEvictP1A_New("Try to evict Pod 1A
(from Replica 1)") + class TryEvictP1A_New action + + TryEvictP1A_New --> CheckGroups2{"Eviction breaks Replica 1.
Total unavailable groups (2) <= 1?"} + class CheckGroups2 decision + + CheckGroups2 -- "No (2 > 1)" --> EvictP1A_Denied("Eviction Denied
Drain Pauses") + class EvictP1A_Denied action + + EvictP1A_Denied --> AppHealthy("Application State:
Replica 1 is protected
(Only Replica 0 is disrupted)") + class AppHealthy outcome_good +``` + +### Notes/Constraints/Caveats (Optional) + + + +#### Background on the `Workload` API + +This KEP assumes that a pod controller (like the one managing `Workload` objects) will create pods and set `pod.spec.workload.name` and `pod.spec.workload.podGroup` on each pod it creates, linking it back to the `Workload` definition. The eviction logic uses this link to read the group's requirements. + +In this KEP, the `Workload` object from the gang scheduling API is the source of truth for pod grouping. + +A `Workload` object contains a list of `PodGroup`s. Each `PodGroup` defines: +* `name`: A unique identifier for the group within the `Workload`. +* `replicas`: The number of instances (replicas) of this group. +* `policy`: The scheduling policy, such as `Gang`. +* `policy.gang.minCount`: The minimum number of pods required for one replica of that group. + +#### Background on multi-pod replicas (LeaderWorkerSet) + +[LeaderWorkerSet](https://lws.sigs.k8s.io/docs/overview/) (LWS) is the primary implementation of a multi-pod replica. The LWS API allows users to manage a group of pods together as if they were a single pod, by specifying a template for a "leader" pod and for the "worker" pods. This is useful in cases where a leader process coordinates multiple worker processes, particularly in AI/ML distributed workloads for model training and inference. All worker pods are treated the same: they are created from the same template, scheduled in parallel, and if any workers fail the group is considered failing. A LeaderWorkerSet object will specify `replicas` for the number of leader+workers groups and `size` for the number of pods per group. + +LWS is planned to be integrated with the Workload API ([KEP](https://docs.google.com/document/d/1QlcIBtR2KyOKYRUTGubhhxuy7NfjHs1fXMJlvdUCyhM/edit?tab=t.0#heading=h.dxr6zknxhiui)). Each LWS replica is would correspond to a PodGroup replica, with its `size` being `minCount`. + +### Risks and Mitigations + + + +- This feature relies on the pod's `spec.workload.name` and `spec.workload.podGroup` fields being correctly set by its managing controller. If a user sets `usePodGroups: true` but the pods are not correctly linked to a `Workload` object, the eviction logic will fall back to per-pod counting to prevent a drain from being blocked by misconfiguration, which may violate the application's intended availability requirements. +- One failing pod in a large group will make that group unhealthy if it drops below its `minCount`. In this way, a small number of failing pods spread across many replicas could prevent all evictions and block node drains. This is intended behavior (as the application is unhealthy), but may be surprising to operators. +- A PDB `selector` that matches pods from multiple different `PodGroup`s (or a mix of grouped and individual pods) may have complex or unintended behavior. Users should be advised to create separate PDBs for each distinct `PodGroup` they wish to protect. + +## Design Details + + + +### API Definition + +We will add a new field to `PodDisruptionBudgetSpec` in `pkg/apis/policy/v1/types.go`. + +```go +// PodDisruptionBudgetSpec defines the desired state of PodDisruptionBudget +type PodDisruptionBudgetSpec struct { + // An eviction is allowed if at least "minAvailable" pods selected by + // ... + MinAvailable *intstr.IntOrString `json:"minAvailable,omitempty" protobuf:"bytes,1,opt,name=minAvailable"` + + // Label query over pods whose evictions are managed by the disruption + // ... + Selector *metav1.LabelSelector `json:"selector,omitempty" protobuf:"bytes,2,opt,name=selector"` + + // An eviction is allowed if at most "maxUnavailable" pods selected by + // ... + MaxUnavailable *intstr.IntOrString `json:"maxUnavailable,omitempty" protobuf:"bytes,3,opt,name=maxUnavailable"` + + // usePodGroups indicates that availability should be calculated based on + // pod groups defined by the Workload API (pod.spec.workloadReference). + // If set to true, the eviction logic will interpret minAvailable/maxUnavailable + // as a count of PodGroup replicas, not individual pods. + // If a pod matched by the selector does not have a workloadReference, + // it will be treated as an individual pod for availability calculations, + // and a warning will be logged. + // Defaults to false. + // +optional + usePodGroups bool `json:"usePodGroups,omitempty" protobuf:"varint,4,opt,name=usePodGroups"` +} +``` + +#### Eviction Logic Flow + +If `pdb.spec.usePodGroups: false` or unset, follow the existing per-pod availability behavior. +If `true`: +1. Get all pods matching the PDB's `selector`. +2. Check if all pods have `spec.workloadReference.Name` set. +3. If no pods have `spec.workloadReference.Name`, log a warning (misconfiguration) and fall back to existing per-pod availability. If some pods have `spec.workloadReference.Name` unset, log a warning, as mixing pod types in one PDB is discouraged. Individual pods will be counted as their own group. +4. Find the `Workload` object for each `spec.workloadReference.Name` +5. Find the `PodGroup` in the `Workload` for each `spec.workloadReference.PodGroup` +6. Get `PodGroup.replicas` (total replicas) and `PodGroup.policy.gang.minCount` (pods in each replica). +7. Count the number of available replicas: a replica is available if its count of existing, healthy, non-evicting pods `>= minCount`. +8. Count the total desired replicas, the sum of `replicas` for all `PodGroup`s. +9. Compare this available group count and total against the PDB's `minAvailable` or `maxUnavailable` to decide if an eviction is allowed. + +```mermaid +graph TD + %% Define Styles + classDef decision fill:#fff0e6,stroke:#ff9933,stroke-width:2px,color:#111 + classDef process fill:#e6f3ff,stroke:#66b3ff,stroke-width:2px,color:#111 + classDef startEnd fill:#f0fff0,stroke:#aaffaa,stroke-width:2px,color:#111 + classDef error fill:#fff0f0,stroke:#ffaaaa,stroke-width:2px,color:#111 + classDef warning fill:#fff9e6,stroke:#ffd666,stroke-width:2px,color:#111 + + subgraph "Group-Aware Eviction Logic Flow" + direction TB + + Start(Eviction API Triggered
for a PDB) --> CheckFlag{"usePodGroups: true?"} + + %% Branch 1: Legacy Path (Flag False/Unset) + CheckFlag -- "No (default)" --> LegacyLogic[Use existing
per-pod availability logic] + LegacyLogic --> DecisionLegacy{"Pods meet
PDB spec?"} + DecisionLegacy -- "Yes" --> Allow[✅ Allow Eviction] + DecisionLegacy -- "No" --> Deny[❌ Deny Eviction] + + %% Branch 2: New Path (Flag True) + CheckFlag -- "Yes" --> GetPods[1. Get all pods matching
PDB selector] + + GetPods --> CheckWorkloadRefs{"2. Pods with
workloadReference?"} + + %% Path 2a: No workloadReference (Misconfiguration) + CheckWorkloadRefs -- "None" --> WarnMisconfig[3. Log Warning:
'usePodGroups: true'
but no pods have
workloadReference] + WarnMisconfig --> LegacyLogic[Fall back to
per-pod logic] + + %% Path 2b: Mixed pod types + CheckWorkloadRefs -- "Some (Mixed)" --> WarnMixed[3. Log Warning:
Mixed pod types found.
Treat individual pods
as their own group.] + WarnMixed --> FindWorkloads + + %% Path 2c: All pods have workloadReference + CheckWorkloadRefs -- "All" --> FindWorkloads[4. Find Workload object
for each pod] + + %% Continue Group Logic Flow + FindWorkloads --> FindPodGroups[5. Find PodGroup in
Workload for each pod] + FindPodGroups --> GetGroupInfo[6. Get PodGroup
replicas & minCount] + + %% --- CORRECTED LINE --- + GetGroupInfo --> CountAvailable["7. Count available replicas
(healthy pods >= minCount)"] + + CountAvailable --> SumTotal[8. Sum total desired replicas
from all PodGroups] + SumTotal --> DecisionNew{"9. Compare available/total
group counts
against PDB spec"} + + DecisionNew -- "Yes" --> Allow + DecisionNew -- "No" --> Deny + end + + %% Styling + class Start,Allow,Deny startEnd + class Deny error + class WarnMisconfig,WarnMixed warning + class GetPods,LegacyLogic,DecisionLegacy,FindWorkloads,FindPodGroups,GetGroupInfo,CountAvailable,SumTotal process + class CheckWorkloadRefs,DecisionNew,CheckFlag decision +``` + +#### Group Health +A `PodGroup` replica is considered healthy if its number of existing, healthy, non-evicting pods is greater than or equal to its `policy.gang.minCount`. + +For example, if a replica is intended to have 10 pods and has `minCount: 8` but only has 9 healthy pods (1 is missing or unhealthy), the replica is still considered healthy. If 3 pods were missing or unhealthy so only 7 healthy pods were found, the replica would be unhealthy. If any pod in a healthy group is targeted for eviction, it would be unhealthy post-eviction and is also counted as unhealthy for the PDB calculation. + + +### Pods missing fields + +If a PDB's `selector` matches a pod that is missing the `spec.workloadReference` field (or its `Name` is empty), it will be treated as an individual pod. If `usePodGroups: true` is set, this will be logged as a warning. If the PDB matches *only* individual pods, this will be equivalent to the standard per-pod logic. If a selected pod has `spec.workload.name` but no `spec.workload.podGroup`, this is a misconfiguration and it will be treated as unhealthy. + +### Test Plan + + + +[x] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +- `k8s.io/kubernetes/pkg/controller/disruption`: `` - `` (tests for new eviction logic). + +##### Integration tests + + + + + +- An integration test will be added to `test/integration/disruption` to simulate the eviction process. +- **Test 1:** PDB with `usePodGroups: false` (default) and `Workload`-managed pods. Verify eviction uses per-pod counting. +- **Test 2:** PDB with `usePodGroups: true` and `Workload`-managed pods. Verify eviction uses per-group counting and blocks when `minAvailable` groups would be violated. +- **Test 3:** PDB with `usePodGroups: true` but with non-`Workload` pods. Verify eviction falls back to per-pod counting and logs a warning. + +##### e2e tests + + + +An e2e test will be added. +1. Create a `Workload` with 2 `PodGroup` replicas, each with `minCount: 3`. +2. Create a PDB with `minAvailable: 1` and `usePodGroups: true` selecting these pods. +3. Manually schedule pods such that one node drain would disrupt both groups (as in the example given earlier). +4. Attempt to drain the node. +5. Verify the drain is blocked by the PDB. +6. Update PDB to `minAvailable: 0`. +7. Verify the drain proceeds. + +### Graduation Criteria + + + +### Upgrade / Downgrade Strategy + + + +Upgrade: +- No changes are required. The new field `usePodGroups` defaults to `false`, so all existing PDBs will continue to function with per-pod logic. +- To use the feature, users must edit their PDBs to set `usePodGroups: true`. + +Downgrade: +- If a PDB was created with `usePodGroups: true`, this field will be dropped when the API server is downgraded (as it's an unknown field). +- The PDB will revert to per-pod logic. This is a behavior change that could violate the application's intended availability (as shown in the user story). +- Operators should remove `usePodGroups` on all PDBs before a downgrade. + + +### Version Skew Strategy + + + +This feature is entirely contained within the disruption controller in `kube-controller-manager` and the API server. By defaulting to false, a conflict generally reverts to the existing behavior. +- **New API server, old KCM:** The API server will accept the `usePodGroups` field, but the old KCM will not know about it and will ignore it, always using per-pod logic. This matches the downgrade scenario. +- **Old API server, new KCM:** The new KCM will attempt to read the `usePodGroups` field, but it won't exist on PDB objects. The KCM will default to `false` and use per-pod logic. + +The feature will only be active when both the API server and `kube-controller-manager` are at the new version and the user has set the field to `true`. + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [ ] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: + - Components depending on the feature gate: +- [x] Other + - Describe the mechanism: The feature is enabled on a per-PDB basis with `spec.usePodGroups: true`. It is disabled by default. + - Will enabling / disabling the feature require downtime of the control plane? No + - Will enabling / disabling the feature require downtime or reprovisioning of a node? No + +###### Does enabling the feature change any default behavior? + + + +No. The default behavior (field unset or `false`) uses existing per-pod availability. The new behavior is opt-in per-PDB. + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +Yes, update the PDB to remove the field or set to `false`. + +###### What happens if we reenable the feature if it was previously rolled back? + +The group-based logic will be re-enabled on the next eviction which uses the PDB. + +###### Are there any tests for feature enablement/disablement? + + + +Testing will cover both states of the boolean field. + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +If an operator downgrades the control plane, PDBs with `usePodGroups: true` will have that field dropped by the older API server. The PDB will silently revert to per-pod logic, which could lead to an application outage during a node drain if the operator was relying on group-based protection. + +###### What specific metrics should inform a rollback? + + + +- An unusually low eviction count (`evictions_total`) might indicate the new logic is too restrictive, or a large number of PDBs are blocking drains. +- An increase in metrics related to unhealthy workloads could indicate the group-based logic is not sufficientlly protecting pod groups. + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +No + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +`kubectl get pdb -A -o jsonpath='{..spec.usePodGroups}'` will show PDBs which have the field set. + +If needed, add metric `disruption_controller_pdbs_using_pod_grouping` for the number of PDBs with `usePodGroups: true`. + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [x] Other (treat as last resort) + - Details: A `kubectl drain` command will stop and report that it is blocked by the PDB, when before it would not have been. + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +- PDB reconciliation latency should not increase significantly. +- Eviction API latency should not increase significantly. The new logic involves additional API calls to get the `Workload` objects, which should be negligible. + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [x] Metrics + - Metric name: `apiserver_request_duration_seconds` (for eviction requests) + - Components exposing the metric: `kube-apiserver` + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +Metrics related to the disruption controller, e.g. a `disruption_controller_reconciliations_total` labeled with the replica mode (individual or pod groups). +For catching issues, `disruption_controller_pdb_grouping_misconfig_total` for when `usePodGroups: true` but no `workloadReference` is found on pods, triggering a fallback. + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +- `Workload` API (CRD) + - Usage description: The disruption controller must be able to GET `Workload` objects by name from a pod's `workloadReference`. + - Impact of its outage on the feature: If the API server is down, evictions won't happen anyway. If the `Workload` CRD is somehow unavailable or the object is missing, the controller will fail to find the group definition. In this case for safety we would deny eviction, as availability cannot be guaranteed. + - Impact of its degraded performance: High latency on GET requests for `Workload` objects would increase the latency of eviction requests. + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +`GET` on `workload.k8s.io/v1alpha1.Workload` objects from `kube-controller-manager` (disruption controller) during an eviction request and controller reconciliation. This should be low-volume, as evictions are not typically frequent. The controller will may use a cache to reduce API calls, for example an informer could prevent some new API calls, but add a `WATCH` from the controller on `Workload`s. + +###### Will enabling / using this feature result in introducing new API types? + + + +No + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +No + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +Yes. +- API type(s): `policy/v1.PodDisruptionBudget` +- Estimated increase in size: One boolean field `usePodGroups`. +- Estimated amount of new objects: 0. + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +Not significantly. The eviction check may now potentially perform an additional API call for `Workload` objects and perform the group-based counting logic. + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +If an informer/cache for `Workload` objects is added to the `kube-controller-manager`, this will increase its RAM usage by a small amount for each `Workload` object in the cluster. + +###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)? + + + +No + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +No different behavior. Eviction requests will fail regardless if the API server is down. + +###### What are other known failure modes? + + + +None that are not already part of the Eviction API + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +## Drawbacks + + + +## Alternatives + + + +Initially there was a plan to integrate directly with multi-pod replica systems (LWS). This would add optional field `replicaKey` to the PDB spec, so the user may provide a label which would identify pods in the same group. For LWS, all pods in a leader+workers group will share the same value for label key `leaderworkerset.sigs.k8s.io/group-key`. This would also require keys to fetch the expected replica count (otherwise we could not detect a missing replica for `maxUnavailable` or a percentage `minAvailable`) and replica size (otherwise we could not detect a missing pod making a replica unhealthy). This would also require some changes to make the LWS [labels/annotations](https://lws.sigs.k8s.io/docs/reference/labels-annotations-and-environment-variables/) more easily avaiable. With the `Workload` API approved and implementaiton in progress, it is better to have both PDBs and LWS integrate with this new core component. + +In the case given in the simplified example above, there may be a way to change the eviction logic to such that the order of pod eviction preserves replicas when possible (e.g. prioritize evicting pods from the replica with the most pods in the node). However, it is simpler to understand and easier ensure intended behavior by just extending the existing PDB budget pattern. It is also unclear if this would work fully when gang scheduling is not used or the number of pods is greater than `minCount`. + +Rather than using a field in the PDB spec, it would be possible to detect if any selected pods have the Workload API enabled by checking their spec for `workload.name`. However, we want this new behavior to be something explicitly enabled. Silently changing the behavior of existing PDB fields (`minAvaiable`/`maxUnavailable`), based on context from other objects, could cause confusion and possibly unintended disruptions. + +## Infrastructure Needed (Optional) + + diff --git a/keps/sig-apps/5682-pdbs-for-multipod-replicas/kep.yaml b/keps/sig-apps/5682-pdbs-for-multipod-replicas/kep.yaml new file mode 100644 index 00000000000..ab64e3e70c3 --- /dev/null +++ b/keps/sig-apps/5682-pdbs-for-multipod-replicas/kep.yaml @@ -0,0 +1,43 @@ +title: Multipod PDB +kep-number: TBD +authors: + - "@LogicalShark" +owning-sig: sig-apps +participating-sigs: + - sig-scheduling + - sig-node +status: provisional #provisional|implementable|implemented|deferred|rejected|withdrawn|replaced +creation-date: 2025-10-28 +reviewers: + - TBD +approvers: + - TBD + +# The target maturity stage in the current dev cycle for this KEP. +# If the purpose of this KEP is to deprecate a user-visible feature +# and a Deprecated feature gates are added, they should be deprecated|disabled|removed. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.36" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.36" + beta: "v1.37" + stable: "v1.38" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +# feature-gates: +# - name: MyFeature +# components: +# - kube-apiserver +# - kube-controller-manager +# disable-supported: true + +# The following PRR answers are required at beta release +# metrics: +# - my_feature_metric