feat(Grafana): Update Grafana Deployment when referenced Secret or Configmap contents change by vignesh-codes · Pull Request #2525 · grafana/grafana-operator

vignesh-codes · 2026-02-22T20:37:23Z

Description

This PR adds automatic rolling restarts for Grafana pods when Secrets or ConfigMaps referenced in a Grafana CR are changed. Previously, the operator had no mechanism to react to secret changes, leaving Grafana pods running with stale credentials until manually restarted.

Related Issue

Fixes #2484

What Changed

Added a new SecretsReconciler to the grafana controller:

Watches all secrets and configmaps referenced in grafana CR's deployment spec
Computes a SHA-256 hash of the ResourceVersion of each referenced object without storing the actual secret data.
Injects the hash as a secret_hash env var on the grafana pod template
Triggers k8s rolling restart automatically when any referecned secret or configmap changes

References are collected from:

spec.deployment.spec.template.spec.containers[ * ] and initContainers[ * ] - secretKeyRef, secretRef, configMapKeyRef, configMapRef
spec.deployment.spec.template.spec.volumes[ * ] - secret, configMap
spec.external - apiKey, adminUser, adminPassword, tls.certSecretRef
spec.client.tls.certSecretRef

Tests

Deployed this grafana

# grafana.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: test-grafana
  namespace: grafana-operator-test
spec:
  deployment:
    spec:
      template:
        spec:
          containers:
            - name: grafana
              env:
                - name: GF_DATABASE_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: test-db-secret
                      key: password

Deployed this secret

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: test-db-secret
  namespace: grafana-operator-test
  labels:
    app.kubernetes.io/managed-by: grafana-operator
type: Opaque
stringData:
  password: initial-password
  username: grafana

Changed the secret to check if the grafana pod restarts

Pod restarted

Unit Tests

added basic unit tests

…er rolling restart on change

theSuess

Thanks for all the work you put into this! I took an initial look at this and will chat with the other maintainers about this in our sync today!

theSuess · 2026-02-23T09:58:36Z

controllers/reconcilers/grafana/secrets_reconciler.go

+func ReferencedSecretsAndConfigMaps(cr *v1beta1.Grafana) ([]string, []string) {
+	secretSet := make(map[string]struct{})
+	configMapSet := make(map[string]struct{})
+
+	addSecret := func(name string) {
+		if name != "" {
+			secretSet[name] = struct{}{}
+		}
+	}
+
+	addConfigMap := func(name string) {
+		if name != "" {
+			configMapSet[name] = struct{}{}
+		}
+	}
+
+	collectDeploymentRefs(cr, addSecret, addConfigMap)
+	collectExternalRefs(cr, addSecret)
+	collectClientRefs(cr, addSecret)
+
+	secretNames := make([]string, 0, len(secretSet))
+	for name := range secretSet {
+		secretNames = append(secretNames, name)
+	}
+
+	configMapNames := make([]string, 0, len(configMapSet))
+	for name := range configMapSet {
+		configMapNames = append(configMapNames, name)
+	}
+
+	sort.Strings(secretNames)
+	sort.Strings(configMapNames)
+
+	return secretNames, configMapNames
+}


This could be a struct function of the v1beta1.Grafana type directly

theSuess · 2026-02-23T10:00:51Z

controllers/reconcilers/grafana/secrets_reconciler.go

+	return secretNames, configMapNames
+}
+
+func collectDeploymentRefs(cr *v1beta1.Grafana, addSecret, addConfigMap func(string)) {


I think these functions would be cleaner when returning arrays directly rather than relying on passed down closures

theSuess · 2026-02-23T10:01:29Z

controllers/reconcilers/grafana/secrets_reconciler.go

+// Grafana CR's deployment spec and external config, hashes them, and stores the result in
+// vars.SecretsHash. The deployment reconciler then injects this as a SECRETS_HASH env var,
+// so any secret rotation causes a pod template change and triggers a rolling restart.
+func (r *SecretsReconciler) Reconcile(ctx context.Context, cr *v1beta1.Grafana, vars *v1beta1.OperatorReconcileVars, scheme *runtime.Scheme) (v1beta1.OperatorStageStatus, error) {


I think we can have this be part of the deployment reconciler

@Baarsgaard / @weisdd what do you think?

Talked about this in the weekly sync, please refactor this to be part of the deployment reconciler

theSuess · 2026-02-23T10:02:32Z

controllers/reconcilers/grafana/deployment_reconciler.go

+		{
+			// helps to restart Grafana when referenced secrets or configmaps are rotated
+			Name:  "SECRETS_HASH",
+			Value: vars.SecretsHash,
+		},


Since this is not actually used, it's better to have this be an annotation on the pod instead. This will also trigger a rollout without polluting the environment variables

…edSecretsAndConfigMaps on Grafana and used GetEnvVarConfigMapSource from tk8s

…ncedSecretsAndConfigMaps and deployment reconciler cheksum annotation process

vignesh-codes · 2026-03-09T01:04:02Z

Hey @theSuess I have updated based on ur comments. Lemme know if any more changes required. thanks :)

theSuess

Sorry for the late review

Looks good from my end, just have a few style suggestions

theSuess · 2026-03-17T11:55:27Z

api/v1beta1/grafana_types.go

+// Refs are collected from: (1) deployment pod template — container Env ValueFrom (SecretKeyRef/
+// ConfigMapKeyRef) and EnvFrom (SecretRef/ConfigMapRef), plus volume Secret and ConfigMap;
+// (2) external — AdminUser, AdminPassword, APIKey, TLS cert Secret if set; (3) client TLS cert
+// Secret if set. The deployment reconciler uses this to compute a hash of those resources'
+// ResourceVersions and sets the checksum/secrets pod template annotation.


nit: we can probably format this better so that each point has a linebreak

theSuess · 2026-03-17T11:56:15Z

api/v1beta1/grafana_types.go

+	return secretNames, configMapNames
+}
+
+func (in *Grafana) collectDeploymentRefs() (secrets, configMaps []string) {


Naming this deploymentRefs() would be more idiomatic go

theSuess · 2026-03-17T11:56:36Z

api/v1beta1/grafana_types.go

+	return secrets, configMaps
+}
+
+func collectContainerEnvRefs(c corev1.Container) (secrets, configMaps []string) {


Suggested change

func collectContainerEnvRefs(c corev1.Container) (secrets, configMaps []string) {

func containerEnvRefs(c corev1.Container) (secrets, configMaps []string) {

theSuess · 2026-03-17T12:09:03Z

api/v1beta1/grafana_types.go

+	return secrets, configMaps
+}
+
+func (in *Grafana) collectExternalRefs() []string {


Since external instances don't need restarting (and credentials are updated immediately), we don't need to index these

theSuess · 2026-03-17T12:57:12Z

api/v1beta1/grafana_types_test.go

 	})
 })
+
+func TestGrafana_ReferencedSecretsAndConfigMaps(t *testing.T) {


Since many test cases only test against the container env vars, can you extract these to their own test and run against collectContainerEnvRefs directly?

I think this would reduce the overhead a bit.

Another alternative would be to create a top level "blank" &Grafana{} on top and then set the required fields using struct accessors like basic.Spec.Deployment.Spec.Template.Spec.Containers = []corev1.Container{...} which would drastically cut down on } } } } } waterfalls 😅

vignesh-codes · 2026-03-30T02:46:06Z

Hey @theSuess,
was supposed to push my changes last week - completely forgot. I have updated the codes. lemme know if its fine.
Thanks :)

theSuess

One last minor thing and then this is good to go!

theSuess · 2026-03-31T08:30:04Z

api/v1beta1/grafana_types.go

+func (in *Grafana) collectClientRefs() []string {
+	if in.Spec.Client == nil || in.Spec.Client.TLS == nil || in.Spec.Client.TLS.CertSecretRef == nil {
+		return nil
+	}
+
+	return []string{in.Spec.Client.TLS.CertSecretRef.Name}
+}


Client secrets are not used in the deployment so this is not needed

feat: watch referenced secrets and configmaps in Grafana CR and trigg…

b4283f7

…er rolling restart on change

vignesh-codes requested review from Baarsgaard, hubeadmin, ishanjainn, nissessenap, pb82, theSuess and weisdd as code owners February 22, 2026 20:37

github-actions bot added the feature this PR introduces a new feature label Feb 22, 2026

theSuess requested changes Feb 23, 2026

View reviewed changes

weisdd mentioned this pull request Feb 24, 2026

chore(tk8s): add more helpers, simplify range checking in getReceivers #2532

Merged

vignesh-codes and others added 3 commits February 28, 2026 15:52

Merge branch 'master' into feat/grafana-cr-secret-configmap-watcher

2f633af

refactor: moved secrets hash into deployment reconciler, add Referenc…

0482a6e

…edSecretsAndConfigMaps on Grafana and used GetEnvVarConfigMapSource from tk8s

enhance: added comments that explains the purpose and usage of Refere…

e060210

…ncedSecretsAndConfigMaps and deployment reconciler cheksum annotation process

vignesh-codes requested a review from theSuess February 28, 2026 22:07

vignesh-codes added 2 commits March 3, 2026 19:08

Merge branch 'master' into feat/grafana-cr-secret-configmap-watcher

5a7d2b9

Merge branch 'master' into feat/grafana-cr-secret-configmap-watcher

ead02c6

theSuess reviewed Mar 17, 2026

View reviewed changes

vignesh-codes and others added 2 commits March 29, 2026 22:31

Merge branch 'master' into feat/grafana-cr-secret-configmap-watcher

3e2c30d

chore: code cleans, removed collectExternalRefs

9631e72

theSuess requested changes Mar 31, 2026

View reviewed changes

Baarsgaard changed the title ~~feat: watch referenced secrets and configmaps in Grafana CR and triger rolling restart on change~~ feat(Grafana): Update Grafana Deployment when referenced Secret or Configmap contents change Mar 31, 2026

	func collectContainerEnvRefs(c corev1.Container) (secrets, configMaps []string) {
	func containerEnvRefs(c corev1.Container) (secrets, configMaps []string) {

Conversation

vignesh-codes commented Feb 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

What Changed

Tests

Unit Tests

Uh oh!

theSuess left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vignesh-codes commented Mar 9, 2026

Uh oh!

theSuess left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vignesh-codes commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theSuess left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vignesh-codes commented Feb 22, 2026 •

edited

Loading

vignesh-codes commented Mar 30, 2026 •

edited

Loading