Refactor replica handling to prevent scale-to-zero scenarios

[adrobisch]

One-line summary

Refactors the replicas values of the EDS and related fallbacks/interfaces, and implements defense-in-depth protection against scale-to-zero scenarios.

Description

Problem

After #516, we identified a regression where the operator would scale the StatefulSet of an existing EDS to zero replicas under specific conditions:

Regression scenario:

• spec.scaling.enabled: true
• spec.excludeSystemIndices: true
• Only system indices exist in the cluster
• Autoscaler.calculateScalingOperation() returns early with noop because len(managedIndices) == 0
• edsReplicas() = Replicas() returns 0 because scaling is enabled and spec.replicas is nil
• operatePods scales to 0, causing complete cluster unavailability

Root Cause

The issue stemmed from ambiguous handling of nil vs. zero replica values:

1. The autoscaler couldn't distinguish between "not yet initialized" (nil) and "explicitly set to zero" (0)
2. When spec.replicas was nil, different code paths made conflicting assumptions
3. No validation prevented minReplicas=0 when autoscaling was enabled
4. Fallback logic could result in zero replicas under edge cases

Solution

Part 1: Pointer-based Replicas API (commits c8daefd, 89e1a58, f1d176d)

Key changes:

• Replicas() and edsReplicas() now return *int32 (pointer) instead of int32
• nil explicitly means "no safe/defined value yet" - don't touch replicas
• Only scaleEDS() initializes spec.replicas with defensive fallback logic
• rescaleStatefulSet() and operatePods() bail out early if desired replicas are nil
• Split rescaleStatefulSet() into smaller functions to reduce cyclomatic complexity

Benefits:

• Clear semantic distinction: nil = uninitialized, non-nil = explicit value
• Prevents accidental writes of zero during initialization
• Makes control flow more explicit and easier to reason about

Part 2: Defense-in-Depth Scale-to-Zero Prevention (commit 008d647)

To ensure scale-to-zero never happens under any circumstance, implemented five layers of protection:

Layer 1: API Validation

File: operator/elasticsearch.go:1046-1095

Added validation to require minReplicas >= 1 when autoscaling is enabled:

if scaling.MinReplicas < 1 {
    return fmt.Errorf(
        "minReplicas must be at least 1 when autoscaling is enabled (got %d)",
        scaling.MinReplicas,
    )
}

Protects against: Configuration errors at resource creation/update time

Layer 2: Autoscaler Bounds Enforcement

File: operator/autoscaler.go:248-258

Enhanced ensureBoundsNodeReplicas() to enforce an absolute minimum of 1:

// Enforce absolute minimum of 1 replica to prevent scale-to-zero
if newDesiredNodeReplicas < 1 {
    as.logger.Warnf("EDS %s/%s: Requested to scale to %d, enforcing minimum of 1 replica.",
        as.eds.Namespace, as.eds.Name, newDesiredNodeReplicas)
    return 1
}

// Enforce minReplicas with absolute floor of 1
effectiveMinReplicas := scalingSpec.MinReplicas
if effectiveMinReplicas < 1 {
    effectiveMinReplicas = 1
}

Protects against: Autoscaler calculations that result in zero (edge cases, bugs)

Layer 3: Fallback Logic Safety

File: operator/elasticsearch.go:938-952

Updated scaleEDS() fallback to never allow 0:

if currentReplicasPtr == nil {
    // Default fallback value (minimum 1)
    desired := int32(1)

    if scaling != nil && scaling.MinReplicas > 0 {
        desired = scaling.MinReplicas
        if eds.Status.Replicas > 0 && eds.Status.Replicas > desired {
            desired = eds.Status.Replicas
        }
    } else if eds.Status.Replicas > 0 {
        desired = eds.Status.Replicas
    }

    // Absolute safety check
    if desired < 1 {
        log.Infof("EDS %s/%s: Fallback calculation resulted in %d, enforcing minimum of 1",
            eds.Namespace, eds.Name, desired)
        desired = 1
    }
}

Protects against: Initialization with zero when status and minReplicas are both zero

Layer 4: Replica Calculation Safety

File: operator/elasticsearch.go:826-845

Updated edsReplicas() to enforce minimum of 1:

// Use max(minReplicas, 1) as base to prevent scale-to-zero
desired := scaling.MinReplicas
if desired < 1 {
    desired = 1
}

Protects against: Calculation logic returning zero in any scenario

Layer 5: Operator Reconciliation Safety

File: operator/operator.go:187-198

Added validation before creating/updating StatefulSet:

// Safety check to prevent scale-to-zero
if desiredReplicas != nil && *desiredReplicas < 1 {
    return nil, fmt.Errorf(
        "refusing to scale StatefulSet %s/%s to %d replicas (minimum is 1)",
        sr.Namespace(), sr.Name(), *desiredReplicas,
    )
}

Protects against: Any code path that attempts to write zero to StatefulSet

Observability Enhancements

Added structured logging at all critical decision points:

• Warning when autoscaler enforces minimum of 1 replica
• Info when fallback logic enforces minimum of 1
• Debug when rescaleStatefulSet bails out due to nil replicas

This ensures operators can debug scale-to-zero prevention in production.

Comprehensive Test Coverage

Added three new test functions:

1. TestValidateScalingSettingsRejectsZeroMinReplicas (elasticsearch_test.go)

   • Validates that minReplicas=0 with autoscaling enabled is rejected

2. TestAutoscalerEnforcesMinimumOneReplica (autoscaler_test.go)

   • Tests that ensureBoundsNodeReplicas() never returns 0
   • Tests with inputs: -1, 0, 1, 5 → all return >= 1

3. TestEDSReplicasEnforcesMinimumOne (elasticsearch_test.go)

   • Verifies edsReplicas() enforces minimum of 1
   • Tests edge cases: status=0, status=5, etc.

Updated existing test documentation to clarify that minReplicas=0 should be rejected by validation.

Key API Changes

Before:

type StatefulResource interface {
    Replicas() int32  // Ambiguous: 0 could mean "not set" or "scale to zero"
}

After:

type StatefulResource interface {
    Replicas() *int32  // Explicit: nil = "not set", non-nil = explicit value
}

Edge Cases Addressed

The defense-in-depth approach protects against:

1. Configuration errors: minReplicas=0 rejected by validation
2. Manual kubectl patches: kubectl patch eds --type=merge -p '{"spec":{"replicas":0}}' → caught by Layer 5
3. Autoscaler edge cases: excludeSystemIndices=true + only system indices → caught by Layers 2-4
4. Status initialization: New EDS with status=0 → fallback ensures >= 1
5. Nil replicas during reconciliation: Early bailout prevents writes
6. Concurrent updates: Kubernetes API optimistic concurrency handles conflicts

Testing

Key test results:

• TestValidateScalingSettings/test_minReplicas_=_0_with_autoscaling_enabled_(scale-to-zero_prevention)
• TestAutoscalerEnforcesMinimumOneReplica
• TestEDSReplicasEnforcesMinimumOne
• All existing autoscaler and operator tests

Migration & Backward Compatibility

Existing EDS resources: No migration needed

• Resources with minReplicas >= 1 continue working unchanged
• Resources with minReplicas = 0 will fail validation on next update (intended behavior)

Recommended action for operators:

1. Audit existing EDS resources: kubectl get eds -A -o json | jq '.items[] | select(.spec.scaling.minReplicas < 1)'
2. Update any with minReplicas=0 to minReplicas=1 or higher
3. Deploy updated operator

Related Issues

• Fixes regression after Fix scale-to-zero bug when spec.replicas is nil #515 and autoscaler: enforce bounds when scaling hint is NONE #516
• Addresses scale-to-zero scenarios discussed with @otrosien
• Implements recommendations from code review analysis

[otrosien]

👍

[adrobisch]

👍