Skip to content

Rollout strategy enum #33758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Rollout strategy enum #33758

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
08756f8
Rollout strategy as an enum, and always increment generation
alex-hunt-materialize Oct 3, 2025
ddabc47
match statement to more clearly show which rollout situation we're in
alex-hunt-materialize Oct 3, 2025
6ef47e9
update helm docs and terraform tests
alex-hunt-materialize Oct 3, 2025
ee55e06
Tear down old gen immediately when ImmediatelyPromoteCausingDowntime
alex-hunt-materialize Oct 14, 2025
aa2d6b5
remove some unused code
alex-hunt-materialize Oct 14, 2025
1942eb4
rename WaitForReady to WaitUntilReady
alex-hunt-materialize Oct 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 30 additions & 7 deletions src/cloud-resources/src/crd/materialize.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,30 @@ pub mod v1alpha1 {
// Additional annotations and labels to include in the Certificate object.
pub secret_template: Option<CertificateSecretTemplate>,
}
#[derive(Clone, Debug, PartialEq, Deserialize, Serialize, JsonSchema)]
pub enum MaterializeRolloutStrategy {
// Default. Create a new generation of pods, leaving the old generation around until the
// new ones are ready to take over.
// This minimizes downtime, and is what almost everyone should use.
WaitForReady,

// WARNING!!!
// THIS WILL CAUSE YOUR MATERIALIZE INSTANCE TO BE UNAVAILABLE FOR SOME TIME!!!
// WARNING!!!
//
// Tear down the old generation of pods and promote the new generation of pods immediately,
// without waiting for the new generation of pods to be ready.
//
// This strategy should ONLY be used by customers with physical hardware who do not have
// enough hardware for the WaitForReady strategy. If you think you want this, please
// consult with Materialize engineering to discuss your situation.
ImmediatelyPromoteCausingDowntime,
}
impl Default for MaterializeRolloutStrategy {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can use derive(Default) and the #[default] variant annotation instead of deriving this by hand

fn default() -> Self {
Self::WaitForReady
}
}

#[derive(
CustomResource, Clone, Debug, Default, PartialEq, Deserialize, Serialize, JsonSchema,
Expand Down Expand Up @@ -147,11 +171,12 @@ pub mod v1alpha1 {
// even without making any meaningful changes.
#[serde(default)]
pub force_rollout: Uuid,
// If false (the default), orchestratord will use the leader
// promotion codepath to minimize downtime during rollouts. If true,
// it will just kill the environmentd pod directly.
// Deprecated and ignored. Use rollout_strategy instead.
#[serde(default)]
pub in_place_rollout: bool,
// Rollout strategy to use when upgrading this Materialize instance.
#[serde(default)]
pub rollout_strategy: MaterializeRolloutStrategy,
// The name of a secret containing metadata_backend_url and persist_backend_url.
// It may also contain external_login_password_mz_system, which will be used as
// the password for the mz_system user if authenticator_kind is Password.
Expand Down Expand Up @@ -368,10 +393,6 @@ pub mod v1alpha1 {
self.spec.request_rollout
}

pub fn in_place_rollout(&self) -> bool {
self.spec.in_place_rollout
}

pub fn rollout_requested(&self) -> bool {
self.requested_reconciliation_id()
!= self
Expand All @@ -386,6 +407,8 @@ pub mod v1alpha1 {

pub fn should_force_promote(&self) -> bool {
self.spec.force_promote == self.spec.request_rollout
|| self.spec.rollout_strategy
== MaterializeRolloutStrategy::ImmediatelyPromoteCausingDowntime
}

pub fn conditions_need_update(&self) -> bool {
Expand Down
18 changes: 5 additions & 13 deletions src/orchestratord/src/controller/materialize.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -431,8 +431,7 @@ impl k8s_controller::Context for Context {
let has_current_changes = status.resources_hash != active_resources.generate_hash();
let active_generation = status.active_generation;
let next_generation = active_generation + 1;
let increment_generation = has_current_changes && !mz.in_place_rollout();
let desired_generation = if increment_generation {
let desired_generation = if has_current_changes {
next_generation
} else {
active_generation
Expand Down Expand Up @@ -494,12 +493,7 @@ impl k8s_controller::Context for Context {

trace!("applying environment resources");
match resources
.apply(
&client,
increment_generation,
mz.should_force_promote(),
&mz.namespace(),
)
.apply(&client, mz.should_force_promote(), &mz.namespace())
.await
{
Ok(Some(action)) => {
Expand All @@ -511,11 +505,9 @@ impl k8s_controller::Context for Context {
// the previous environmentd until the new one is
// fully ready
resources.promote_services(&client, &mz.namespace()).await?;
if increment_generation {
resources
.teardown_generation(&client, mz, active_generation)
.await?;
}
resources
.teardown_generation(&client, mz, active_generation)
.await?;
self.update_status(
&mz_api,
mz,
Expand Down
Loading