Flux fails when certSecretRef points to secret with empty ca.crt

[suneclausen]

Describe the bug

Observed in Flux 2.7.2 (helm chart 2.17.1), ImageRepository resources fail to scan when certSecretRef points to a secret containing an empty string for ca.crt. This configuration previously worked in Flux 2.6.4 (helm chart 2.16.3).

The error message received is:

$ flux -n debugtenant get images all
NAME                            LAST SCAN       SUSPENDED       READY   MESSAGE
imagerepository/example-app                     False           False   failed to configure authentication options: secret 'debugtenant/empty-cert-for-public' must contain either 'ca.crt' or both 'tls.crt' and 'tls.key'

NAME                    IMAGE   TAG     READY   MESSAGE
imagepolicy/example-app                 False   retrying in 30s error: no tags in database

This appears to be a regression as the same configuration worked correctly in the previous version where output for the same deployment was errors.

Steps to reproduce

Installed flux using Helm chart: https://github.com/fluxcd-community/helm-charts/releases/download/flux2-2.17.1/flux2-2.17.1.tgz

Then apply the following manifests

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: example-app
  namespace: debugtenant
spec:
  imageRepositoryRef:
    name: example-app
  policy:
    alphabetical:
      order: asc
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: example-app
  namespace: debugtenant
spec:
  image: quay.io/redhattraining/hello-world-nginx
  interval: 1m0s
  certSecretRef:
    name: empty-cert-for-public
---
apiVersion: v1
kind: Secret
metadata:
  name: empty-cert-for-public
  namespace: debugtenant
type: Opaque
data:
  ca.crt: ""

Expected behavior

ImageRepository should successfully scan container registries with publicly-signed certificates when certSecretRef points to a secret with an empty ca.crt value, matching the behavior in Flux 2.6.4 leading to result:

$ flux -n debugtenant get image all
NAME                            LAST SCAN                       SUSPENDED       READY   MESSAGE
imagerepository/example-app     2025-11-27T10:48:52+01:00       False           True    successful scan: found 2 tags

NAME                    LATEST IMAGE                                    READY   MESSAGE
imagepolicy/example-app quay.io/redhattraining/hello-world-nginx:v1.0   True    Latest image tag for quay.io/redhattraining/hello-world-nginx resolved to v1.0

Use case: This pattern allows maintaining consistent configuration templates across multiple projects, where some repositories use self-signed certificates (requiring a populated ca.crt) and others use publicly-signed certificates (requiring an empty ca.crt). The ability to use an empty string simplifies configuration management.

Current workaround: Removing certSecretRef entirely for repositories with publicly-signed certificates gives the desired outcome, but requires conditional configuration logic instead of always pointing to a secret and then populate that when needed.

Screenshots and recordings

No response

OS / Distro

N/A

Flux version

Issue on 2.7.2, works on 2.6.4

Flux check

On 2.7.2

$ flux check
► checking prerequisites
✗ flux 2.7.3 <2.7.4 (new CLI version is available, please upgrade)
✔ Kubernetes 1.34.0 >=1.32.0-0
► checking version in cluster
✔ distribution: flux-2.7.2
✔ bootstrapped: false
► checking controllers
(Omitted - using chart defaults)
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1
✔ externalartifacts.source.toolkit.fluxcd.io/v1
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1
✔ helmreleases.helm.toolkit.fluxcd.io/v2
✔ helmrepositories.source.toolkit.fluxcd.io/v1
✔ imagepolicies.image.toolkit.fluxcd.io/v1
✔ imagerepositories.image.toolkit.fluxcd.io/v1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

On 2.6.4

$ flux check
► checking prerequisites
✗ flux 2.2.2 <2.7.4 (new CLI version is available, please upgrade)
✔ Kubernetes 1.33.1+k3s1 >=1.26.0-0
► checking version in cluster
✔ distribution: flux-2.6.4
✔ bootstrapped: false
► checking controllers
(Omitted - using chart defaults)
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta3
✔ buckets.source.toolkit.fluxcd.io/v1
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1
✔ helmreleases.helm.toolkit.fluxcd.io/v2
✔ helmrepositories.source.toolkit.fluxcd.io/v1
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1
✔ providers.notification.toolkit.fluxcd.io/v1beta3
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[cappyzawa]

@suneclausen Thank you for reporting this issue.

In scenarios where deployments span multiple environments (dev/staging/prod), having a secret key present but with an empty value is a plausible pattern.

I'll work on fixing this.

[cappyzawa]

Following up on my earlier comment —

I’ve tracked this down to the migration to pkg/runtime/secrets in fluxcd/image-reflector-controller#791.

So behaviour-wise, we now have two possible contracts for certSecretRef:

1. Empty ca.crt = no custom CA (use system CA)

   • Keeps the 2.6.x behaviour
   • Makes templating easier (always point to a secret)
   • But can hide misconfigurations if someone expected a real CA

2. Empty ca.crt is invalid

   • Stricter / safer, catches TLS misconfig early
   • But breaks existing setups relying on empty ca.crt for public registries

Given that 2.6.x allowed the first behaviour and people are using it in the wild, my current thinking is:

• short term (likely starting in Flux v2.7.x patch releases and in v2.8):
  restore the legacy behaviour (empty ca.crt treated as “no CA”) to avoid breaking existing manifests, but emit a warning/condition indicating that this usage is deprecated and may change in the next minor release.

• longer term (Flux v2.9+):
  if strict validation is the desired contract, re-introduce enforcement (empty ca.crt becomes invalid) as a documented breaking change.

Since the regression was introduced through my PR, I’m happy to send the fix once we agree on which contract we want.

@stefanprodan @matheuscscp @suneclausen How does this plan sound?

[matheuscscp]

I think that it's a lot of effort fixing this in a patch just to reintroduce it later down the road.

And I also think people are not widely using this hack, impact should be minimal.

I apologize for the impact on this regression, but I'm not for supporting this setup. I don't think it simplifies configuration management, but rather supports bad practices.

[suneclausen]

Agree that it might be a bit much adding it to remove it again later. For me it would suffice adding a note in the next release about change of behaviour - just to indicate the consequence of the change?

In the end the impact is not huge on my end and I can work around it.

But it might be worth investigating if the same behaviour is present for HelmRepositories as well as it also exposes certSecretRef? That is for you to decide.

[matheuscscp]

Yes this behavior is everywhere in Flux now, as in Flux 2.7 we refactored the way we deal with secrets everywhere. I've renamed the issue to match this fact.

About the release notes, that's good feedback for future behavior changes going forward. We missed this one, unfortunately. I personally didn't even notice this behavior change while reviewing all the PRs, but if I did I'd definitely add a release note. Sorry about that!

@stefanprodan Is it worth updating the release notes or the blog post now?