Skip to content

restart redpanda on changes to the cluster configuration #603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
simon0191 wants to merge 1 commit into from
Closed

restart redpanda on changes to the cluster configuration #603

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .changes/unreleased/operator-Changed-20250403-131231.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
project: operator
kind: Changed
body: The operator is going to restart the cluster on any change to the cluster configuration
time: 2025-04-03T13:12:31.215439+02:00
5 changes: 5 additions & 0 deletions charts/redpanda/configmap.tpl.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,11 @@ func RedpandaConfigFile(dot *helmette.Dot, includeNonHashableItems bool) string
return helmette.ToYaml(redpandaYaml)
}

func RedpandaClusterConfig(dot *helmette.Dot) string {
values := helmette.Unwrap[Values](dot.Values)
return helmette.ToYaml(values.Config.Cluster.Translate())
}

// RPKProfile returns a [corev1.ConfigMap] for aiding users in connecting to
// the external listeners of their redpanda cluster.
// It is meant for external consumption via NOTES.txt and is not used within
Expand Down
2 changes: 1 addition & 1 deletion charts/redpanda/statefulset.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1051,7 +1051,7 @@ func statefulSetChecksumAnnotation(dot *helmette.Dot) string {
var dependencies []any
// NB: Seed servers is excluded to avoid a rolling restart when only
// replicas is changed.
dependencies = append(dependencies, RedpandaConfigFile(dot, false))
dependencies = append(dependencies, RedpandaConfigFile(dot, false), RedpandaClusterConfig(dot))
Copy link
Contributor

@chrisseto chrisseto Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mentioned in slack but saying it here as well, this won't work as expected. This will trigger a restart before the cluster config is applied.

I think our best option is to do some hackery within the operator when useFlux is false (This is now the default in Azure so we should be good to go).

In broad strokes, you'll want to get the cluster config's version and inject it into an annotation on the redpanda Pods.

In reconcileResources (which is reconcileDeflux in the release/2.3.x and release/2.4.x branch), you should acquire the version and set it:

# Everything here is nullable, so you'll need to do some nasty if chaining.
values.Statefulset.PodTemplate.Annotations["some-magic-key"] = clusterConfigVersion

I see ~2 options for getting the version. You can either make an admin client and pull it directly (1) or you could update reconcileClusterConfig stash the cluster config version onto the Status and/or Condition (2) and then read that within reconcileResources.

Option 1 feels kinda hacky but I think it'll be the fastest way to get this done.

Option 2 "fits" into the operator a bit better IMO. We're it me, I'd update the syncer.Sync to return (ClusterConfigVersion, error) and then stash the version in the Message of the cluster config condition:

	apimeta.SetStatusCondition(rp.GetConditions(), metav1.Condition{
		Type:               redpandav1alpha2.ClusterConfigSynced,
		Status:             metav1.ConditionTrue,
		ObservedGeneration: rp.Generation,
		Reason:             "ConfigSynced",
                 Message:            fmt.Sprintf("ClusterConfig at Version %d", version), 
	})

Then you can set the annotation to the value of message.

I'd strongly recommend adding a test case to RedpandaControllerSuite to make sure this works as expected.

You'll also likely want to work off the release/2.3.x branch as that's what's currently being deployed into cloud.

if values.External.Enabled {
dependencies = append(dependencies, ptr.Deref(values.External.Domain, ""))
if helmette.Empty(values.External.Addresses) {
Expand Down
49 changes: 30 additions & 19 deletions charts/redpanda/templates/_configmap.go.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,17 @@
{{- end -}}
{{- end -}}

{{- define "redpanda.RedpandaClusterConfig" -}}
{{- $dot := (index .a 0) -}}
{{- range $_ := (list 1) -}}
{{- $_is_returning := false -}}
{{- $values := $dot.Values.AsMap -}}
{{- $_is_returning = true -}}
{{- (dict "r" (toYaml (get (fromJson (include "redpanda.ClusterConfig.Translate" (dict "a" (list $values.config.cluster) ))) "r"))) | toJson -}}
{{- break -}}
{{- end -}}
{{- end -}}

{{- define "redpanda.RPKProfile" -}}
{{- $dot := (index .a 0) -}}
{{- range $_ := (list 1) -}}
Expand Down Expand Up @@ -121,23 +132,23 @@
{{- break -}}
{{- end -}}
{{- $kafkaTLS := (get (fromJson (include "redpanda.rpkKafkaClientTLSConfiguration" (dict "a" (list $dot) ))) "r") -}}
{{- $_175___ok_3 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $kafkaTLS "ca_file" (coalesce nil)) ))) "r") -}}
{{- $_ := (index $_175___ok_3 0) -}}
{{- $ok_3 := (index $_175___ok_3 1) -}}
{{- $_180___ok_3 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $kafkaTLS "ca_file" (coalesce nil)) ))) "r") -}}
{{- $_ := (index $_180___ok_3 0) -}}
{{- $ok_3 := (index $_180___ok_3 1) -}}
{{- if $ok_3 -}}
{{- $_ := (set $kafkaTLS "ca_file" "ca.crt") -}}
{{- end -}}
{{- $adminTLS := (get (fromJson (include "redpanda.rpkAdminAPIClientTLSConfiguration" (dict "a" (list $dot) ))) "r") -}}
{{- $_181___ok_4 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $adminTLS "ca_file" (coalesce nil)) ))) "r") -}}
{{- $_ := (index $_181___ok_4 0) -}}
{{- $ok_4 := (index $_181___ok_4 1) -}}
{{- $_186___ok_4 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $adminTLS "ca_file" (coalesce nil)) ))) "r") -}}
{{- $_ := (index $_186___ok_4 0) -}}
{{- $ok_4 := (index $_186___ok_4 1) -}}
{{- if $ok_4 -}}
{{- $_ := (set $adminTLS "ca_file" "ca.crt") -}}
{{- end -}}
{{- $schemaTLS := (get (fromJson (include "redpanda.rpkSchemaRegistryClientTLSConfiguration" (dict "a" (list $dot) ))) "r") -}}
{{- $_187___ok_5 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $schemaTLS "ca_file" (coalesce nil)) ))) "r") -}}
{{- $_ := (index $_187___ok_5 0) -}}
{{- $ok_5 := (index $_187___ok_5 1) -}}
{{- $_192___ok_5 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $schemaTLS "ca_file" (coalesce nil)) ))) "r") -}}
{{- $_ := (index $_192___ok_5 0) -}}
{{- $ok_5 := (index $_192___ok_5 1) -}}
{{- if $ok_5 -}}
{{- $_ := (set $schemaTLS "ca_file" "ca.crt") -}}
{{- end -}}
Expand Down Expand Up @@ -318,10 +329,10 @@
{{- if (gt ((get (fromJson (include "_shims.len" (dict "a" (list $tls_8) ))) "r") | int) (0 | int)) -}}
{{- $schemaRegistryTLS = $tls_8 -}}
{{- end -}}
{{- $_372_lockMemory_overprovisioned_flags := (get (fromJson (include "redpanda.RedpandaAdditionalStartFlags" (dict "a" (list $values) ))) "r") -}}
{{- $lockMemory := (index $_372_lockMemory_overprovisioned_flags 0) -}}
{{- $overprovisioned := (index $_372_lockMemory_overprovisioned_flags 1) -}}
{{- $flags := (index $_372_lockMemory_overprovisioned_flags 2) -}}
{{- $_377_lockMemory_overprovisioned_flags := (get (fromJson (include "redpanda.RedpandaAdditionalStartFlags" (dict "a" (list $values) ))) "r") -}}
{{- $lockMemory := (index $_377_lockMemory_overprovisioned_flags 0) -}}
{{- $overprovisioned := (index $_377_lockMemory_overprovisioned_flags 1) -}}
{{- $flags := (index $_377_lockMemory_overprovisioned_flags 2) -}}
{{- $result := (dict "additional_start_flags" $flags "enable_memory_locking" $lockMemory "overprovisioned" $overprovisioned "kafka_api" (dict "brokers" $brokerList "tls" $brokerTLS ) "admin_api" (dict "addresses" (get (fromJson (include "redpanda.Listeners.AdminList" (dict "a" (list $values.listeners ($values.statefulset.replicas | int) (get (fromJson (include "redpanda.Fullname" (dict "a" (list $dot) ))) "r") (get (fromJson (include "redpanda.InternalDomain" (dict "a" (list $dot) ))) "r")) ))) "r") "tls" $adminTLS ) "schema_registry" (dict "addresses" (get (fromJson (include "redpanda.Listeners.SchemaRegistryList" (dict "a" (list $values.listeners ($values.statefulset.replicas | int) (get (fromJson (include "redpanda.Fullname" (dict "a" (list $dot) ))) "r") (get (fromJson (include "redpanda.InternalDomain" (dict "a" (list $dot) ))) "r")) ))) "r") "tls" $schemaRegistryTLS ) ) -}}
{{- $result = (merge (dict ) $result (get (fromJson (include "redpanda.Tuning.Translate" (dict "a" (list $values.tuning) ))) "r")) -}}
{{- $result = (merge (dict ) $result (get (fromJson (include "redpanda.Config.CreateRPKConfiguration" (dict "a" (list $values.config) ))) "r")) -}}
Expand Down Expand Up @@ -565,17 +576,17 @@
{{- end -}}
{{- $enabledOptions := (dict "true" true "1" true "" true ) -}}
{{- $lockMemory := false -}}
{{- $_657_value_14_ok_15 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $flags "--lock-memory" "") ))) "r") -}}
{{- $value_14 := (index $_657_value_14_ok_15 0) -}}
{{- $ok_15 := (index $_657_value_14_ok_15 1) -}}
{{- $_662_value_14_ok_15 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $flags "--lock-memory" "") ))) "r") -}}
{{- $value_14 := (index $_662_value_14_ok_15 0) -}}
{{- $ok_15 := (index $_662_value_14_ok_15 1) -}}
{{- if $ok_15 -}}
{{- $lockMemory = (ternary (index $enabledOptions $value_14) false (hasKey $enabledOptions $value_14)) -}}
{{- $_ := (unset $flags "--lock-memory") -}}
{{- end -}}
{{- $overprovisioned := false -}}
{{- $_664_value_16_ok_17 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $flags "--overprovisioned" "") ))) "r") -}}
{{- $value_16 := (index $_664_value_16_ok_17 0) -}}
{{- $ok_17 := (index $_664_value_16_ok_17 1) -}}
{{- $_669_value_16_ok_17 := (get (fromJson (include "_shims.dicttest" (dict "a" (list $flags "--overprovisioned" "") ))) "r") -}}
{{- $value_16 := (index $_669_value_16_ok_17 0) -}}
{{- $ok_17 := (index $_669_value_16_ok_17 1) -}}
{{- if $ok_17 -}}
{{- $overprovisioned = (ternary (index $enabledOptions $value_16) false (hasKey $enabledOptions $value_16)) -}}
{{- $_ := (unset $flags "--overprovisioned") -}}
Expand Down
2 changes: 1 addition & 1 deletion charts/redpanda/templates/_statefulset.go.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -615,7 +615,7 @@
{{- $_is_returning := false -}}
{{- $values := $dot.Values.AsMap -}}
{{- $dependencies := (coalesce nil) -}}
{{- $dependencies = (concat (default (list ) $dependencies) (list (get (fromJson (include "redpanda.RedpandaConfigFile" (dict "a" (list $dot false) ))) "r"))) -}}
{{- $dependencies = (concat (default (list ) $dependencies) (list (get (fromJson (include "redpanda.RedpandaConfigFile" (dict "a" (list $dot false) ))) "r") (get (fromJson (include "redpanda.RedpandaClusterConfig" (dict "a" (list $dot) ))) "r"))) -}}
{{- if $values.external.enabled -}}
{{- $dependencies = (concat (default (list ) $dependencies) (list (get (fromJson (include "_shims.ptr_Deref" (dict "a" (list $values.external.domain "") ))) "r"))) -}}
{{- if (empty $values.external.addresses) -}}
Expand Down
1 change: 1 addition & 0 deletions operator/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ and is generated by [Changie](https://github.com/miniscruff/changie).
When `useFlux` is `false`, the equivalent of chart v5.9.19 will be deployed.

* Bumped the internal chart version to v5.9.20.
* The operator is going to restart the cluster on any change to the cluster configuration
### Removed
* Removed bundled FluxCD controllers, bundled FluxCD CRDs, and support for delegating control to FluxCD.

Expand Down