Using version v9.0.313 of reflector does not mirror the secrets.

[ioannispl41]

Description: In the latest version of Reflector, secrets annotated for reflection are not being mirrored in newly created namespaces. Despite properly annotating the secret, it does not appear in the newly created namespace (test-namespace). This issue did not exist in previous versions.Steps to Reproduce

1. Deploy the latest version of Reflector.

2. Create a secret in the default namespace with the following annotations:

kind: Secret
metadata:
  name: test-secret
  namespace: default
  annotations:
    reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
    reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: ""
    reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
    reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: ""
type: Opaque
data:
  key: dGVzdA==  # "test" in base64

3. Create a new namespace:

kubectl create namespace test-namespace

4. Check if the secret is mirrored in test-namespace:

kubectl get secrets -n test-namespace

Expected Behavior The secret should be automatically mirrored into the test-namespace as per the reflection-auto-enabled: "true" annotation.Actual Behavior The secret is not mirrored into the test-namespace.Logs from Reflector

2025-02-26 10:30:38.461 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2025-02-26 10:30:38.537 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2025-02-26 10:30:38.545 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources
2025-02-26 10:30:38.586 +00:00 [INF] (Microsoft.Hosting.Lifetime) Now listening on: http://[::]:8080
2025-02-26 10:30:38.586 +00:00 [INF] (Microsoft.Hosting.Lifetime) Application started. Press Ctrl+C to shut down.
2025-02-26 10:30:38.586 +00:00 [INF] (Microsoft.Hosting.Lifetime) Hosting environment: Production
2025-02-26 10:30:38.586 +00:00 [INF] (Microsoft.Hosting.Lifetime) Content root path: /app
2025-02-26 10:30:38.929 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-node-lease/dockerhub as a reflection of default/dockerhub
2025-02-26 10:30:38.941 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-public/dockerhub as a reflection of default/dockerhub
2025-02-26 10:30:38.951 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-system/dockerhub as a reflection of default/dockerhub
2025-02-26 10:30:38.964 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created local-path-storage/dockerhub as a reflection of default/dockerhub
2025-02-26 10:30:38.964 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected default/dockerhub where permitted. Created 4 - Updated 0 - Deleted 0 - Validated 0.
2025-02-26 10:30:38.995 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-node-lease/ghcr as a reflection of default/ghcr
2025-02-26 10:30:39.007 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-public/ghcr as a reflection of default/ghcr
2025-02-26 10:30:39.017 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-system/ghcr as a reflection of default/ghcr
2025-02-26 10:30:39.028 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created local-path-storage/ghcr as a reflection of default/ghcr
2025-02-26 10:30:39.029 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected default/ghcr where permitted. Created 4 - Updated 0 - Deleted 0 - Validated 0.

Output of kubectl get secrets -A (Secret Not Mirrored in test-namespace)**

NAMESPACE                       NAME                                                  TYPE                             DATA   AGE
default                         test-secret                                          Opaque                           1      4m39s
kube-node-lease                 test-secret                                          Opaque                           1      4m32s
kube-public                     test-secret                                          Opaque                           1      4m32s
kube-system                     test-secret                                          Opaque                           1      4m32s
local-path-storage              test-secret                                          Opaque                           1      4m32s
# test-namespace missing from the list

Possible Regression?
This behavior worked in previous versions but is broken in the latest update.

Additional Context: Secrets are correctly mirrored into default system namespaces (kube-node-lease, kube-public, kube-system, local-path-storage), but the newly created namespace test-namespace is missing.

---

[jaudiger]

We're also facing this issue for newly creating namespaces.

[sstobak]

We're also facing this issue for newly creating namespaces.

+1

[egbakou]

I've submitted a PR here #479

[winromulus]

I've merged the PR but it's a temp solution since I'm reworking most of that area for monitoring resources and disposing the client

[steinwelberg]

I have the feeling that version 9.0.318 does not solve the problem (at least so it seems for me). I still see the same behaviour as mentioned at the top of this issue.

I have a few secrets that should be automatically reflected to all namespaces. When I create a new namespace the secrets are not automatically created. The reflector does not log anything after the initial start. When I kill the reflector pod and it starts up again, then it does detect the new namespace and reflect the secrets into it.

[egbakou]

@steinwelberg I've tested the release on AKS and local k3s without any issues. Could you provide a reproduction example similar to what (@)ioannispl41 did?

[nhudson]

I have the feeling that version 9.0.318 does not solve the problem (at least so it seems for me). I still see the same behaviour as mentioned at the top of this issue.

I have a few secrets that should be automatically reflected to all namespaces. When I create a new namespace the secrets are not automatically created. The reflector does not log anything after the initial start. When I kill the reflector pod and it starts up again, then it does detect the new namespace and reflect the secrets into it.

I also have this issue. If reflector starts it will reflect the secret in all namespaces as expected, but if you create a new namespace the secret will not show up until you restart reflector.

This is running locally with kind (v9.0.318)

k get deploy -n reflector -oyaml | grep image
          image: docker.io/emberstack/kubernetes-reflector:9.0.318
          imagePullPolicy: IfNotPresent

k logs -n reflector reflector-dcc5cf554-hk55j
2025-03-05 21:01:24.561 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2025-03-05 21:01:24.588 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2025-03-05 21:01:24.607 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources
2025-03-05 21:01:24.626 +00:00 [INF] (Microsoft.Hosting.Lifetime) Now listening on: http://[::]:8080
2025-03-05 21:01:24.626 +00:00 [INF] (Microsoft.Hosting.Lifetime) Application started. Press Ctrl+C to shut down.
2025-03-05 21:01:24.626 +00:00 [INF] (Microsoft.Hosting.Lifetime) Hosting environment: Production
2025-03-05 21:01:24.627 +00:00 [INF] (Microsoft.Hosting.Lifetime) Content root path: /app
2025-03-05 21:02:20.671 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created calico-apiserver/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:20.675 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created calico-system/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:20.780 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created default/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:20.877 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-node-lease/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:20.978 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-public/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.081 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-system/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.180 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created local-path-storage/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.187 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created monitoring/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.279 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created reflector/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.381 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created tigera-operator/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.481 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created traefik/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.481 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected minio/s3creds where permitted. Created 11 - Updated 0 - Deleted 0 - Validated 0

 k get secrets -n monitoring s3creds
NAME      TYPE     DATA   AGE
s3creds   Opaque   2      22m

Then adding a new namespace called test-reflector

 k create ns test-reflector
namespace/test-reflector created

k get secrets -n test-reflector
No resources found in test-reflector namespace.

k logs -n reflector reflector-dcc5cf554-hk55j
2025-03-05 21:01:24.561 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2025-03-05 21:01:24.588 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2025-03-05 21:01:24.607 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources
2025-03-05 21:01:24.626 +00:00 [INF] (Microsoft.Hosting.Lifetime) Now listening on: http://[::]:8080
2025-03-05 21:01:24.626 +00:00 [INF] (Microsoft.Hosting.Lifetime) Application started. Press Ctrl+C to shut down.
2025-03-05 21:01:24.626 +00:00 [INF] (Microsoft.Hosting.Lifetime) Hosting environment: Production
2025-03-05 21:01:24.627 +00:00 [INF] (Microsoft.Hosting.Lifetime) Content root path: /app
2025-03-05 21:02:20.671 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created calico-apiserver/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:20.675 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created calico-system/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:20.780 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created default/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:20.877 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-node-lease/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:20.978 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-public/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.081 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created kube-system/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.180 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created local-path-storage/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.187 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created monitoring/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.279 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created reflector/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.381 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created tigera-operator/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.481 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created traefik/s3creds as a reflection of minio/s3creds
2025-03-05 21:02:21.481 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected minio/s3creds where permitted. Created 11 - Updated 0 - Deleted 0 - Validated 0.

Then restarting reflector

k rollout restart deployment reflector -n reflector
deployment.apps/reflector restarted

 k logs -n reflector reflector-67c79d7d5f-59z75
2025-03-05 21:27:25.037 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2025-03-05 21:27:25.058 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2025-03-05 21:27:25.070 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources
2025-03-05 21:27:25.086 +00:00 [INF] (Microsoft.Hosting.Lifetime) Now listening on: http://[::]:8080
2025-03-05 21:27:25.086 +00:00 [INF] (Microsoft.Hosting.Lifetime) Application started. Press Ctrl+C to shut down.
2025-03-05 21:27:25.086 +00:00 [INF] (Microsoft.Hosting.Lifetime) Hosting environment: Production
2025-03-05 21:27:25.086 +00:00 [INF] (Microsoft.Hosting.Lifetime) Content root path: /app
2025-03-05 21:27:25.257 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created cert-manager/s3creds as a reflection of minio/s3creds
2025-03-05 21:27:25.349 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created cnpg-test/s3creds as a reflection of minio/s3creds
2025-03-05 21:27:25.449 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created tembo-system/s3creds as a reflection of minio/s3creds
2025-03-05 21:27:25.552 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Created test-reflector/s3creds as a reflection of minio/s3creds
2025-03-05 21:27:25.552 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected minio/s3creds where permitted. Created 4 - Updated 0 - Deleted 0 - Validated 11.

k get secrets -n test-reflector
NAME      TYPE     DATA   AGE
s3creds   Opaque   2      84s

Reverting back to 7.1.288 works as expected.

[winromulus]

Please upgrade to the new version. There's a fix in there for a missing registration (got removed by mistake when Autofac was removed).

[codestation]

The bug is still present, upgraded to docker.io/emberstack/kubernetes-reflector:9.0.322, created a new namespace and the secrets weren't created. Restarted the pod and the secrets got created. There were no logs when i created the namespace.

Reverted back to 7.1.288.

[winromulus]

@codestation do you still have the most recent logs from it?

[codestation]

Yes, i still have a cluster with the latest version:

ka get deploy -n reflector -oyaml | grep image
          image: docker.io/emberstack/kubernetes-reflector:9.0.322
          imagePullPolicy: IfNotPresent

2025-03-10 18:46:00.270 +00:00 [INF[] (ES.Kubernetes.Reflector.Watchers.NamespaceWatcher) Requesting V1Namespace resources
2025-03-10 18:46:00.317 +00:00 [INF[] (ES.Kubernetes.Reflector.Watchers.SecretWatcher) Requesting V1Secret resources
2025-03-10 18:46:00.350 +00:00 [INF[] (ES.Kubernetes.Reflector.Watchers.ConfigMapWatcher) Requesting V1ConfigMap resources
2025-03-10 18:46:00.393 +00:00 [INF[] (Microsoft.Hosting.Lifetime) Now listening on: http://[::[]:8080
2025-03-10 18:46:00.394 +00:00 [INF[] (Microsoft.Hosting.Lifetime) Application started. Press Ctrl+C to shut down.
2025-03-10 18:46:00.394 +00:00 [INF[] (Microsoft.Hosting.Lifetime) Hosting environment: Production
2025-03-10 18:46:00.394 +00:00 [INF[] (Microsoft.Hosting.Lifetime) Content root path: /app
2025-03-10 18:46:01.746 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected credentials/cnpg-credentials where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 68.
2025-03-10 18:46:01.762 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected credentials/mariadb-credentials where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 68.
2025-03-10 18:46:02.664 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected monitoring/thanos-objectstorage where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 1.
2025-03-10 18:46:02.949 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected registry-creds/regcred-codestation where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 68.
2025-03-10 18:46:02.964 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected registry-creds/regcred-dockerhub where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 68.

Then created the namespace

$ ka create ns test
namespace/test created

No secrets were created and no new logs appeared

$ ka get secret -n test
No resources found in test namespace.

Restarted the pod

$ ka rollout restart -n reflector deployment reflector
deployment.apps/reflector restarted

Logs after restart

2025-03-10 18:53:06.673 +00:00 [INF[] (ES.Kubernetes.Reflector.Watchers.NamespaceWatcher) Requesting V1Namespace resources
2025-03-10 18:53:06.719 +00:00 [INF[] (ES.Kubernetes.Reflector.Watchers.SecretWatcher) Requesting V1Secret resources
2025-03-10 18:53:06.750 +00:00 [INF[] (ES.Kubernetes.Reflector.Watchers.ConfigMapWatcher) Requesting V1ConfigMap resources
2025-03-10 18:53:06.799 +00:00 [INF[] (Microsoft.Hosting.Lifetime) Now listening on: http://[::[]:8080
2025-03-10 18:53:06.800 +00:00 [INF[] (Microsoft.Hosting.Lifetime) Application started. Press Ctrl+C to shut down.
2025-03-10 18:53:06.800 +00:00 [INF[] (Microsoft.Hosting.Lifetime) Hosting environment: Production
2025-03-10 18:53:06.800 +00:00 [INF[] (Microsoft.Hosting.Lifetime) Content root path: /app
2025-03-10 18:53:07.945 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Created test/cnpg-credentials as a reflection of credentials/cnpg-credentials
2025-03-10 18:53:07.946 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected credentials/cnpg-credentials where permitted. Created 1 - Updated 0 - Deleted 0 - Validated 68.
2025-03-10 18:53:08.040 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Created test/mariadb-credentials as a reflection of credentials/mariadb-credentials
2025-03-10 18:53:08.040 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected credentials/mariadb-credentials where permitted. Created 1 - Updated 0 - Deleted 0 - Validated 68.
2025-03-10 18:53:08.956 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected monitoring/thanos-objectstorage where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 1.
2025-03-10 18:53:09.208 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Created test/regcred-codestation as a reflection of registry-creds/regcred-codestation
2025-03-10 18:53:09.209 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected registry-creds/regcred-codestation where permitted. Created 1 - Updated 0 - Deleted 0 - Validated 68.
2025-03-10 18:53:09.312 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Created test/regcred-dockerhub as a reflection of registry-creds/regcred-dockerhub
2025-03-10 18:53:09.312 +00:00 [INF[] (ES.Kubernetes.Reflector.Mirroring.SecretMirror) Auto-reflected registry-creds/regcred-dockerhub where permitted. Created 1 - Updated 0 - Deleted 0 - Validated 68.

After a restart the secrets were created. This cluster is on LKE (Linode), but my other cluster was k3s and it was the same before i downgraded.

[ZephireNZ]

I am also seeing the same behaviour as @codestation.

When a new namespace is created, reflector does not add configs/secrets as expected. However if reflector rollout is restarted then new configs/secrets are then created.

After downgrading to 7.1.288 it works as expected.

Annotations I'm using:

    reflector.v1.k8s.emberstack.com/reflection-allowed: 'true'
    reflector.v1.k8s.emberstack.com/reflection-auto-enabled: 'true'
    reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: my-prefix-.*