Misconfigured PDB blocks draining node indefinitely

[felipecruz91]

Description

Observed Behavior: Karpenter cannot disrupt a node if there's a pod with PDB (minAvailable: 100%) despite having a NodeClaim with both expireAfter (720h) and terminationGracePeriod (1h).

Expected Behavior: Once the terminationGracePeriod elapses, remaining pods should be forcibly deleted and the underlying instance will be terminated. Source: https://karpenter.sh/docs/concepts/disruption/#terminationgraceperiod

Reproduction Steps (Please include YAML):

1. Deploy a Pod with a PDB like the following:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-test-pdb-pod
  labels:
    app: nginx-test-pdb-pod
spec:
  nodeSelector:
    karpenter.sh/nodepool: test-huge-crowd-general
  tolerations:
  - key: taint.nr-ops.net/general-pool
    operator: Exists
    effect: NoSchedule
  containers:
  - name: nginx
    image: nginx:latest
    ports:
    - containerPort: 80
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-test-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: nginx-test-pdb-pod

2. Manually drain the node with kubectl drain node <node> ...

3. Observe that the node gets in unschedulable state and never gets deleted (even after the grace period of 1h has expired), causing a costs increase. (FWIW if the pod is deleted manually then Karpenter removes the node eventually).

│ Events:                                                                                                                                                                                                                       │
│   Type    Reason              Age                 From       Message                                                                                                                                                          │
│   ----    ------              ----                ----       -------                                                                                                                                                          │
│   Normal  Unconsolidatable    13m (x86 over 23h)  karpenter  Can't replace with a cheaper node                                                                                                                                │
│   Normal  DisruptionBlocked   26s                 karpenter  Pdb "default/nginx-test-pdb" prevents pod evictions

Versions:

• Chart Version: 1.3.2
• Kubernetes Version (kubectl version): v1.32.9-0

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[moko-poi]

Hi @felipecruz91,

Thank you for reporting this issue. I've investigated the codebase and documentation, and I believe this is a legitimate bug.

According to the TerminationGracePeriod design document and the API specification, when a node enters deletion and the terminationGracePeriod elapses, remaining pods should be forcibly deleted, bypassing PDBs:

karpenter/pkg/apis/v1/nodeclaim.go

Lines 58 to 61 in 2731eb9

	// Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.
	//
	// This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period.
	// When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.

However, as you observed, the PDB continues to block pod eviction indefinitely, even after the 1-hour grace period has elapsed. This indicates the forceful termination logic is not triggering.

The root cause appears to be in the implementation at pkg/controllers/node/termination/controller.go:

karpenter/pkg/controllers/node/termination/controller.go

Lines 378 to 392 in 2731eb9

	func (c *Controller) nodeTerminationTime(node *corev1.Node, nodeClaim *v1.NodeClaim) (*time.Time, error) {
	if nodeClaim == nil {
	return nil, nil
	}
	expirationTimeString, exists := nodeClaim.Annotations[v1.NodeClaimTerminationTimestampAnnotationKey]
	if !exists {
	return nil, nil
	}
	c.recorder.Publish(terminatorevents.NodeTerminationGracePeriodExpiring(node, expirationTimeString))
	expirationTime, err := time.Parse(time.RFC3339, expirationTimeString)
	if err != nil {
	return nil, serrors.Wrap(fmt.Errorf("parsing annotation, %w", err), "annotation", v1.NodeClaimTerminationTimestampAnnotationKey)
	}
	return &expirationTime, nil
	}

The termination grace period enforcement depends on a specific annotation being set by the NodeClaim lifecycle controller. If this annotation is not set properly or timing issues occur during the deletion process, the forceful eviction logic never triggers, leaving the node blocked by the PDB indefinitely.

This contradicts the documented behavior where terminationGracePeriod should bypass blocked PDBs after the grace period elapses.

[jmdeal]

However, as you observed, the PDB continues to block pod eviction indefinitely, even after the 1-hour grace period has elapsed. This indicates the forceful termination logic is not triggering.

I don't think this is quite accurate. In the repro, you are draining the node via kubectl drain but that won't trigger Karpenter's disruption flow. There's a good reason for that too - there are use-cases where you have an unhealthy node and you want to drain pods off of it, but you don't want it to be terminated by Karpenter so you can investigate. To trigger Karpenter's disruption flow you have to delete either the node or the nodeclaim. Karpenter uses finalizers to retain the resources until they are drained and terminated.

You can also see where the annotation is added here:

karpenter/pkg/controllers/nodeclaim/lifecycle/controller.go

Lines 188 to 194 in bc64ee6

	if err := c.ensureTerminationGracePeriodTerminationTimeAnnotation(ctx, nodeClaim); err != nil {
	if errors.IsConflict(err) {
	return reconcile.Result{Requeue: true}, nil
	}
	return reconcile.Result{}, fmt.Errorf("adding nodeclaim terminationGracePeriod annotation, %w", err)
	}

Karpenter adds this annotation to the NodeClaim as soon as it detects the delete operation. If there's a reason it's not being added, I'd prefer to dive into that rather than add a fallback mechanism (at least at first). That being said, the use of the annotation vs the deletion timestamp is vestigial. Changing it to use the deletion timestamp directly is a welcome change. Nevermind, I forgot the reason we still use the annotation is the node health controller took a dependency on it.

---

edit: adding a direct response to the OP here since it's buried in this message. To drain and terminate a node with Karpenter, you should delete the node or the nodeclaim (e.g. kubectl delete node <node-name>). Draining a node with kubectl is a separate operation which Karpenter doesn't natively integrate with and generally serves a different purpose.

[jmdeal]

/triage solved
/kind support

[jmdeal]

/remove-kind bug

[felipecruz91]

@jmdeal Thank you for your answer. The kubectl drain operation is what I (mistakenly) thought would trigger the Karpenter disruption flow just to reproduce the issue I came across in several nodes of the cluster.

I constantly see the following kind of message in several nodes of the cluster:

│ Events:                                                                                                                                                                                                                       │
│   Type    Reason              Age                 From       Message                                                                                                                                                          │
│   ----    ------              ----                ----       -------                                                                                                                                                          │
│   Normal  Unconsolidatable    13m (x86 over 23h)  karpenter  Can't replace with a cheaper node                                                                                                                                │
│   Normal  DisruptionBlocked   26s                 karpenter  Pdb "default/nginx-test-pdb" prevents pod evictions

When these messages appear, I did not run a kubectl drain operation - so I assume they are the result of Karpenter's disruption flow kicking in.

So my question is: is there a way to configure Karpenter to forcefully delete pods with PDBs after a period of time? And thus allow the node to be disrupted.

[jmdeal]

The events you're seeing indicate Karpenter is choosing to not take any action against the node. In this case Karpenter's excluding nodes from consolidation due to blocking PDBs.

So my question is: is there a way to configure Karpenter to forcefully delete pods with PDBs after a period of time? And thus allow the node to be disrupted.

In this case since Karpenter isn't attempting to drain the node, there isn't any point in time where PDBs would be bypassed. It's a different story when Karpenter does start to drain a node for any number of reasons (drift, expiration, etc). That's when termination grace period comes into play. The timer for termination grace period starts with the drain and pods will be deleted once it elapses.