Rollout strategy enum

misc/helm-charts/operator/README.md

@@ -287,7 +287,9 @@ helm upgrade my-materialize-operator materialize/misc/helm-charts/operator -f my
 
 To upgrade your Materialize instances, you'll need to update the Materialize custom resource and trigger a rollout.
 
-By default, the operator performs rolling upgrades (`inPlaceRollout: false`) which minimize downtime but require additional Kubernetes cluster resources during the transition. However, keep in mind that rolling upgrades typically take longer to complete due to the sequential rollout process. For environments where downtime is acceptable, you can opt for in-place upgrades (`inPlaceRollout: true`).
+By default, the operator performs rolling upgrades (`rolloutStrategy: WaitUntilReady`) which minimizes downtime but require additional Kubernetes cluster resources during the transition.
+
+For environments without enough capacity to perform the `WaitUntilReady` strategy, and where downtime is acceptable, there is the `ImmediatelyPromoteCausingDowntime` strategy. This strategy will cause downtime and is not recommended. If you think you need this, please reach out to Materialize engineering to discuss your situation.
 
 #### Determining the Version
 
@@ -350,7 +352,6 @@ spec:
   environmentdImageRef: materialize/environmentd:v0.147.0 # Update version as needed
   requestRollout: 22222222-2222-2222-2222-222222222222    # Generate new UUID
   forceRollout: 33333333-3333-3333-3333-333333333333      # Optional: for forced rollouts
-  inPlaceRollout: false                                   # When false, performs a rolling upgrade rather than in-place
   backendSecretName: materialize-backend
 ```
 
@@ -371,10 +372,6 @@ kubectl patch materialize <instance-name> \
   -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\", \"forceRollout\": \"$(uuidgen)\"}}"
 ```
 
-The behavior of a forced rollout follows your `inPlaceRollout` setting:
-- With `inPlaceRollout: false` (default): Creates new instances before terminating the old ones, temporarily requiring twice the resources during the transition
-- With `inPlaceRollout: true`: Directly replaces the instances, causing downtime but without requiring additional resources
-
 ### Verifying the Upgrade
 
 After initiating the rollout, you can monitor the status:
@@ -392,9 +389,6 @@ kubectl logs -l app.kubernetes.io/name=materialize-operator -n materialize
 - `requestRollout` triggers a rollout only if there are actual changes to the instance (like image updates)
 - `forceRollout` triggers a rollout regardless of whether there are changes, which can be useful for debugging or when you need to force a rollout for other reasons
 - Both fields expect UUID values and each rollout requires a new, unique UUID value
-- `inPlaceRollout`:
-  - When `false` (default): Performs a rolling upgrade by spawning new instances before terminating old ones. While this minimizes downtime, there may still be a brief interruption during the transition.
-  - When `true`: Directly replaces existing instances, which will cause downtime.
 
 # Operational Guidelines
 

misc/helm-charts/operator/README.md.gotmpl

@@ -228,7 +228,9 @@ helm upgrade my-materialize-operator materialize/misc/helm-charts/operator -f my
 
 To upgrade your Materialize instances, you'll need to update the Materialize custom resource and trigger a rollout.
 
-By default, the operator performs rolling upgrades (`inPlaceRollout: false`) which minimize downtime but require additional Kubernetes cluster resources during the transition. However, keep in mind that rolling upgrades typically take longer to complete due to the sequential rollout process. For environments where downtime is acceptable, you can opt for in-place upgrades (`inPlaceRollout: true`).
+By default, the operator performs rolling upgrades (`rolloutStrategy: WaitUntilReady`) which minimizes downtime but require additional Kubernetes cluster resources during the transition.
+
+For environments without enough capacity to perform the `WaitUntilReady` strategy, and where downtime is acceptable, there is the `ImmediatelyPromoteCausingDowntime` strategy. This strategy will cause downtime and is not recommended. If you think you need this, please reach out to Materialize engineering to discuss your situation.
 
 #### Determining the Version
 
@@ -291,7 +293,6 @@ spec:
   environmentdImageRef: materialize/environmentd:v0.147.0 # Update version as needed
   requestRollout: 22222222-2222-2222-2222-222222222222    # Generate new UUID
   forceRollout: 33333333-3333-3333-3333-333333333333      # Optional: for forced rollouts
-  inPlaceRollout: false                                   # When false, performs a rolling upgrade rather than in-place
   backendSecretName: materialize-backend
 ```
 
@@ -312,10 +313,6 @@ kubectl patch materialize <instance-name> \
   -p "{\"spec\": {\"requestRollout\": \"$(uuidgen)\", \"forceRollout\": \"$(uuidgen)\"}}"
 ```
 
-The behavior of a forced rollout follows your `inPlaceRollout` setting:
-- With `inPlaceRollout: false` (default): Creates new instances before terminating the old ones, temporarily requiring twice the resources during the transition
-- With `inPlaceRollout: true`: Directly replaces the instances, causing downtime but without requiring additional resources
-
 ### Verifying the Upgrade
 
 After initiating the rollout, you can monitor the status:
@@ -333,9 +330,6 @@ kubectl logs -l app.kubernetes.io/name=materialize-operator -n materialize
 - `requestRollout` triggers a rollout only if there are actual changes to the instance (like image updates)
 - `forceRollout` triggers a rollout regardless of whether there are changes, which can be useful for debugging or when you need to force a rollout for other reasons
 - Both fields expect UUID values and each rollout requires a new, unique UUID value
-- `inPlaceRollout`:
-  - When `false` (default): Performs a rolling upgrade by spawning new instances before terminating old ones. While this minimizes downtime, there may still be a brief interruption during the transition.
-  - When `true`: Directly replaces existing instances, which will cause downtime.
 
 # Operational Guidelines
 

src/cloud-resources/src/crd/materialize.rs

@@ -58,6 +58,30 @@ pub mod v1alpha1 {
         // Additional annotations and labels to include in the Certificate object.
         pub secret_template: Option<CertificateSecretTemplate>,
     }
+    #[derive(Clone, Debug, PartialEq, Deserialize, Serialize, JsonSchema)]
+    pub enum MaterializeRolloutStrategy {
+        // Default. Create a new generation of pods, leaving the old generation around until the
+        // new ones are ready to take over.
+        // This minimizes downtime, and is what almost everyone should use.
+        WaitUntilReady,
+
+        // WARNING!!!
+        // THIS WILL CAUSE YOUR MATERIALIZE INSTANCE TO BE UNAVAILABLE FOR SOME TIME!!!
+        // WARNING!!!
+        //
+        // Tear down the old generation of pods and promote the new generation of pods immediately,
+        // without waiting for the new generation of pods to be ready.
+        //
+        // This strategy should ONLY be used by customers with physical hardware who do not have
+        // enough hardware for the WaitUntilReady strategy. If you think you want this, please
+        // consult with Materialize engineering to discuss your situation.
+        ImmediatelyPromoteCausingDowntime,
+    }
+    impl Default for MaterializeRolloutStrategy {
+        fn default() -> Self {
+            Self::WaitUntilReady
+        }
+    }
 
     #[derive(
         CustomResource, Clone, Debug, Default, PartialEq, Deserialize, Serialize, JsonSchema,
@@ -147,11 +171,12 @@ pub mod v1alpha1 {
         // even without making any meaningful changes.
         #[serde(default)]
         pub force_rollout: Uuid,
-        // If false (the default), orchestratord will use the leader
-        // promotion codepath to minimize downtime during rollouts. If true,
-        // it will just kill the environmentd pod directly.
+        // Deprecated and ignored. Use rollout_strategy instead.
         #[serde(default)]
         pub in_place_rollout: bool,
+        // Rollout strategy to use when upgrading this Materialize instance.
+        #[serde(default)]
+        pub rollout_strategy: MaterializeRolloutStrategy,
         // The name of a secret containing metadata_backend_url and persist_backend_url.
         // It may also contain external_login_password_mz_system, which will be used as
         // the password for the mz_system user if authenticator_kind is Password.
@@ -368,10 +393,6 @@ pub mod v1alpha1 {
             self.spec.request_rollout
         }
 
-        pub fn in_place_rollout(&self) -> bool {
-            self.spec.in_place_rollout
-        }
-
         pub fn rollout_requested(&self) -> bool {
             self.requested_reconciliation_id()
                 != self
@@ -386,6 +407,8 @@ pub mod v1alpha1 {
 
         pub fn should_force_promote(&self) -> bool {
             self.spec.force_promote == self.spec.request_rollout
+                || self.spec.rollout_strategy
+                    == MaterializeRolloutStrategy::ImmediatelyPromoteCausingDowntime
         }
 
         pub fn conditions_need_update(&self) -> bool {

src/orchestratord/src/controller/materialize.rs

@@ -28,7 +28,7 @@ use uuid::Uuid;
 use crate::metrics::Metrics;
 use mz_cloud_provider::CloudProvider;
 use mz_cloud_resources::crd::materialize::v1alpha1::{
-    Materialize, MaterializeCertSpec, MaterializeStatus,
+    Materialize, MaterializeCertSpec, MaterializeRolloutStrategy, MaterializeStatus,
 };
 use mz_license_keys::validate;
 use mz_orchestrator_kubernetes::KubernetesImagePullPolicy;
@@ -431,8 +431,7 @@ impl k8s_controller::Context for Context {
         let has_current_changes = status.resources_hash != active_resources.generate_hash();
         let active_generation = status.active_generation;
         let next_generation = active_generation + 1;
-        let increment_generation = has_current_changes && !mz.in_place_rollout();
-        let desired_generation = if increment_generation {
+        let desired_generation = if has_current_changes {
             next_generation
         } else {
             active_generation
@@ -449,8 +448,9 @@ impl k8s_controller::Context for Context {
         );
         let resources_hash = resources.generate_hash();
 
-        let mut result = if has_current_changes {
-            if mz.rollout_requested() {
+        let mut result = match (has_current_changes, mz.rollout_requested()) {
+            // There are changes pending, and we want to appy them.
+            (true, true) => {
                 // we remove the environment resources hash annotation here
                 // because if we fail halfway through applying the resources,
                 // things will be in an inconsistent state, and we don't want
@@ -492,14 +492,20 @@ impl k8s_controller::Context for Context {
                 let mz = &mz;
                 let status = mz.status();
 
+                if mz.spec.rollout_strategy
+                    == MaterializeRolloutStrategy::ImmediatelyPromoteCausingDowntime
+                {
+                    // The only reason someone would choose this strategy is if they didn't have
+                    // space for the two generations of pods.
+                    // Lets make room for the new ones by deleting the old generation.
+                    resources
+                        .teardown_generation(&client, mz, active_generation)
+                        .await?;
+                }
+
                 trace!("applying environment resources");
                 match resources
-                    .apply(
-                        &client,
-                        increment_generation,
-                        mz.should_force_promote(),
-                        &mz.namespace(),
-                    )
+                    .apply(&client, mz.should_force_promote(), &mz.namespace())
                     .await
                 {
                     Ok(Some(action)) => {
@@ -510,12 +516,12 @@ impl k8s_controller::Context for Context {
                         // do this last, so that we keep traffic pointing at
                         // the previous environmentd until the new one is
                         // fully ready
+                        // TODO add condition saying we're about to promote,
+                        // and check it before aborting anything.
                         resources.promote_services(&client, &mz.namespace()).await?;
-                        if increment_generation {
-                            resources
-                                .teardown_generation(&client, mz, active_generation)
-                                .await?;
-                        }
+                        resources
+                            .teardown_generation(&client, mz, active_generation)
+                            .await?;
                         self.update_status(
                             &mz_api,
                             mz,
@@ -578,7 +584,9 @@ impl k8s_controller::Context for Context {
                         Err(e)
                     }
                 }
-            } else {
+            }
+            // There are changes pending, but we don't want to apply them yet.
+            (true, false) => {
                 let mut needs_update = mz.conditions_need_update();
                 if mz.update_in_progress() {
                     resources
@@ -613,44 +621,46 @@ impl k8s_controller::Context for Context {
                 debug!("changes detected, waiting for approval");
                 Ok(None)
             }
-        } else {
-            // this can happen if we update the environment, but then revert
-            // that update before the update was deployed. in this case, we
-            // don't want the environment to still show up as
-            // WaitingForApproval.
-            let mut needs_update = mz.conditions_need_update() || mz.rollout_requested();
-            if mz.update_in_progress() {
-                resources
-                    .teardown_generation(&client, mz, next_generation)
+            // No changes pending, but we might need to clean up a partially applied rollout.
+            (false, _) => {
+                // this can happen if we update the environment, but then revert
+                // that update before the update was deployed. in this case, we
+                // don't want the environment to still show up as
+                // WaitingForApproval.
+                let mut needs_update = mz.conditions_need_update() || mz.rollout_requested();
+                if mz.update_in_progress() {
+                    resources
+                        .teardown_generation(&client, mz, next_generation)
+                        .await?;
+                    needs_update = true;
+                }
+                if needs_update {
+                    self.update_status(
+                        &mz_api,
+                        mz,
+                        MaterializeStatus {
+                            active_generation,
+                            last_completed_rollout_request: mz.requested_reconciliation_id(),
+                            resource_id: status.resource_id,
+                            resources_hash: status.resources_hash,
+                            conditions: vec![Condition {
+                                type_: "UpToDate".into(),
+                                status: "True".into(),
+                                last_transition_time: Time(chrono::offset::Utc::now()),
+                                message: format!(
+                                    "No changes found from generation {active_generation}"
+                                ),
+                                observed_generation: mz.meta().generation,
+                                reason: "Applied".into(),
+                            }],
+                        },
+                        active_generation != desired_generation,
+                    )
                     .await?;
-                needs_update = true;
-            }
-            if needs_update {
-                self.update_status(
-                    &mz_api,
-                    mz,
-                    MaterializeStatus {
-                        active_generation,
-                        last_completed_rollout_request: mz.requested_reconciliation_id(),
-                        resource_id: status.resource_id,
-                        resources_hash: status.resources_hash,
-                        conditions: vec![Condition {
-                            type_: "UpToDate".into(),
-                            status: "True".into(),
-                            last_transition_time: Time(chrono::offset::Utc::now()),
-                            message: format!(
-                                "No changes found from generation {active_generation}"
-                            ),
-                            observed_generation: mz.meta().generation,
-                            reason: "Applied".into(),
-                        }],
-                    },
-                    active_generation != desired_generation,
-                )
-                .await?;
+                }
+                debug!("no changes");
+                Ok(None)
             }
-            debug!("no changes");
-            Ok(None)
         };
 
         // balancers rely on the environmentd service existing, which is