Skip to content

[Bug] Deployment analyzer reports in-progress rollouts as failures #1742

Description

@junnhwan

Expected Behavior

A Deployment that is actively progressing (rollout or scaling in progress) should not be reported as a problem. status.readyReplicas trailing spec.replicas is a normal, healthy intermediate state while the Deployment controller is rolling out a new ReplicaSet or scaling up — image pulls and readiness probes take time, and maxSurge/maxUnavailable deliberately allow temporarily fewer ready pods.

Current Behavior

pkg/analyzer/deployment.go reports a failure whenever spec.replicas != status.readyReplicas, without inspecting the Deployment's progress state. During a normal rollout (e.g. spec.replicas: 3, status.replicas: 3, status.readyReplicas: 2), running k8sgpt analyze --filter Deployment emits:

Deployment default/web has 3 replicas but 2 are available with status running

This is a false positive: the third pod is still pulling the image / waiting on its readiness probe, and the Deployment is healthy.

The check also does not consider status.observedGeneration vs metadata.generation. When the controller has not yet reconciled the latest spec (observedGeneration < generation), status is stale relative to spec, and the replica comparison is misleading. No analyzer in pkg/analyzer/ currently consults observedGeneration.

Steps to Reproduce

Reproduces purely at the status level with the fake client already used in pkg/analyzer/deployment_test.go (no live cluster required):

  1. Create a Deployment with spec.replicas: 3 and a pod template.
  2. Set status.replicas: 3, status.readyReplicas: 2, status.availableReplicas: 2 — the mid-rollout shape.
  3. Run the Deployment analyzer (k8sgpt analyze --filter Deployment, or the unit test harness).
  4. Observe the failure text above, even though nothing is wrong.

To reproduce on a live cluster: deploy a Deployment with a readiness probe, update its image, and run k8sgpt analyze during the rollout window (seconds to minutes depending on image size and probe time).

Environment

  • k8sgpt version: main
  • Kubernetes version: n/a (analyzer logic; reproduced with fake client). Rollout semantics are identical across versions.
  • AI Backend/Provider: n/a (analyzer path, --explain not required)
  • OS/Platform: n/a

Additional Context

Current check:

if deployment.Spec.Replicas != nil && *deployment.Spec.Replicas != deployment.Status.ReadyReplicas {
    // report replica mismatch only
}

Suggested direction for the fix (open to maintainer preference; deliberately does not overlap with the Progressing=False / ProgressDeadlineExceeded reporting from #1739 / #1740, which covers the stuck rollout case):

  1. ObservedGeneration guard: skip the replica comparison while status.observedGeneration < metadata.generation (controller hasn't caught up to the latest spec yet).
  2. Healthy-progress check: when status.conditions show Progressing=True (with Available=True), treat the replica mismatch as in-progress and do not report; only report when the rollout is genuinely stuck or replicas are unavailable.
  3. Add regression tests for: healthy rollout-in-progress, scaling-in-progress, controller-not-yet-synced (observedGeneration < generation), and a genuinely stuck rollout (to keep the existing [Bug] Deployment analyzer misses ProgressDeadlineExceeded when ReadyReplicas still match spec #1739 behavior).

Scope: single file pkg/analyzer/deployment.go + deployment_test.go.

I would like to work on this and can send a PR in that shape if the direction looks right.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    • Status
      Proposed

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions