Manifest with multiple images where one doesn't exist forces image to be updated with a blank tag

[dwilliams782]

Describe the bug
We are using image updater to update the images for ephemeral Kubernetes environments. When the environment is created, we use the Production ECR and the latest images, then image updater is configured to look for new images with a regex tag filter in a Staging ECR, and update to it when a new image is available (i.e. after release jobs have run in the ephemeral environment's Pull Request).

We have an Application containing a deployment with one container and an initContainer. The annotations on the Argo App looks like the following:

argocd-image-updater.argoproj.io/crm-backend.allow-tags=regexp:^eph-1438-.+$ 
argocd-image-updater.argoproj.io/crm-backend.force-update=true 
argocd-image-updater.argoproj.io/crm-backend.helm.image-name=container.imageRepository 
argocd-image-updater.argoproj.io/crm-backend.helm.image-tag=container.imageTag 
argocd-image-updater.argoproj.io/crm-backend.update-strategy=latest 
argocd-image-updater.argoproj.io/crm-db-migrations.allow-tags=regexp:^eph-1438-.+$ 
argocd-image-updater.argoproj.io/crm-db-migrations.force-update=true 
argocd-image-updater.argoproj.io/crm-db-migrations.helm.image-name=initContainers.db-migrations.imageRepository 
argocd-image-updater.argoproj.io/crm-db-migrations.helm.image-tag=initContainers.db-migrations.imageTag 
argocd-image-updater.argoproj.io/crm-db-migrations.update-strategy=latest 
argocd-image-updater.argoproj.io/image-list=crm-backend=<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com/crm-backend,crm-db-migrations=<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations 
argocd-image-updater.argoproj.io/write-back-method=argocd

When the new version of <staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com/crm-backend was published, it correctly detected this and updated it, but it also updated <staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations to an empty image tag:

Which causes a new ReplicaSet to be span up with an incorrect image for the initContainer:

Init:Failed to apply default image tag "<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations:": couldn't parse image name "<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations:": invalid reference format

Looking around this area in the code, it doesn't look like this is supposed to be possible for an image to be updated with no tag. We also did not get any Successfully updated image ... logs to indicate that image-updater thinks it updated the image here.

I think this is related to having force-update: true set, but AFAICT this is required to make it watch for images in a repository other than the one it started from?

Worth noting: the Staging ECR for crm-db-migrations does not have any images that matches this prefix, nor does it have any images with no tag. It should return zero images.

I'm going to enable trace logs and see if I can spot anything else going on, but I wanted to raise this in the mean time incase it is user error and a quick answer.

Expected behavior
The image referenced by the initContainer should not have been updated as no image exists that matches the regex.

Version

argocd:
  chart: argocd-image-updater
  repoURL: https://argoproj.github.io/argo-helm
  targetRevision: 0.12.3 # AppVersion v0.16.0

Logs

time="2025-08-19T17:49:47Z" level=debug msg="Considering this image for update" alias=crm-db-migrations application=crm-backend-1438 image_name=crm-db-migrations image_tag= registry=<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com

time="2025-08-19T17:49:47Z" level=debug msg="Using no version constraint when looking for a new tag" alias=crm-db-migrations application=crm-backend-1438 image_name=crm-db-migrations image_tag= registry=<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com

time="2025-08-19T17:49:47Z" level=warning msg="\"latest\" strategy has been renamed to \"newest-build\". Please switch to the new convention as support for the old naming convention will be removed in future versions." image_alias=crm-db-migrations image_name=<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations registry_url=<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com

time="2025-08-19T17:49:47Z" level=debug msg="target parameters: image-spec= image-name=initContainers.db-migrations.imageRepository, image-tag=initContainers.db-migrations.imageTag" application=crm-backend-1438 image=<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations namespace=argocd

time="2025-08-19T17:49:47Z" level=debug msg="Image '<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations' already on latest allowed version" alias=crm-db-migrations application=crm-backend-1438 image_name=crm-db-migrations image_tag= registry=<staging-ecr-id>.dkr.ecr.eu-west-2.amazonaws.com

[dwilliams782]

Trace logs:

> k logs argocd-image-updater-655f988fd-9lclf | grep migrations

time="2025-08-19T19:23:21Z" level=debug msg="Considering this image for update" alias=crm-db-migrations application=crm-backend-1438 image_name=crm-db-migrations image_tag= registry=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
time="2025-08-19T19:23:21Z" level=debug msg="Using no version constraint when looking for a new tag" alias=crm-db-migrations application=crm-backend-1438 image_name=crm-db-migrations image_tag= registry=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
time="2025-08-19T19:23:21Z" level=trace msg="Found update strategy latest" image_alias=crm-db-migrations image_name=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations registry_url=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
time="2025-08-19T19:23:21Z" level=warning msg="\"latest\" strategy has been renamed to \"newest-build\". Please switch to the new convention as support for the old naming convention will be removed in future versions." image_alias=crm-db-migrations image_name=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations registry_url=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
time="2025-08-19T19:23:21Z" level=trace msg="No ignore-tags annotation found" image_alias=crm-db-migrations image_name=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations registry_url=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
time="2025-08-19T19:23:21Z" level=trace msg="Using runtime platform constraint linux/amd64" image_alias=crm-db-migrations image_name=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations registry_url=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
time="2025-08-19T19:23:21Z" level=trace msg="No pull-secret annotation found" image_alias=crm-db-migrations image_name=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations registry_url=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
time="2025-08-19T19:23:21Z" level=trace msg="Performing HTTP GET https://<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/v2/crm-db-migrations/tags/list"
time="2025-08-19T19:23:21Z" level=trace msg="Removing tag 86ac873 because it either didn't match defined pattern or is ignored" alias=crm-db-migrations application=crm-backend-1438 image_name=crm-db-migrations image_tag= registry=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
.... Lots of removing tags
time="2025-08-19T19:23:21Z" level=trace msg="Removing tag c206dc6 because it either didn't match defined pattern or is ignored" alias=crm-db-migrations application=crm-backend-1438 image_name=crm-db-migrations image_tag= registry=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
time="2025-08-19T19:23:21Z" level=trace msg="List of available tags found: []" alias=crm-db-migrations application=crm-backend-1438 image_name=crm-db-migrations image_tag= registry=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com
time="2025-08-19T19:23:21Z" level=debug msg="target parameters: image-spec= image-name=initContainers.db-migrations.imageRepository, image-tag=initContainers.db-migrations.imageTag" application=crm-backend-1438 image=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations namespace=argocd
time="2025-08-19T19:23:21Z" level=debug msg="Image '<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations' already on latest allowed version" alias=crm-db-migrations application=crm-backend-1438 image_name=crm-db-migrations image_tag= registry=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com

Looking here:

		} else {
			// We need to explicitly set the up-to-date images in the spec too, so
			// that we correctly marshal out the parameter overrides to include all
			// images, regardless of those were updated or not.
			err = setAppImage(&updateConf.UpdateApp.Application, applicationImage.WithTag(updateableImage.ImageTag))
			if err != nil {
				imgCtx.Errorf("Error while trying to update image: %v", err)
				result.NumErrors += 1
			}
			imgCtx.Debugf("Image '%s' already on latest allowed version", updateableImage.GetFullNameWithTag())
		}

It looks like this intentionally updates the application, which is fine, but I would expect a little further up (around here):

		imgCtx.Tracef("List of available tags found: %v", tags.Tags())

		// Get the latest available tag matching any constraint that might be set
		// for allowed updates.

for a check to see if at least one valid tag was found?

This is making me think it is user error as it looks very intentional, please let me know if I've done something wrong! Happy to help test in any way.

[chengfang]

In your application annotations, such as argocd-image-updater.argoproj.io/write-back-method=argocd, ":" should be used instead of "=". Not sure if this has caused differences in parsing the annotation values.

see https://argocd-image-updater.readthedocs.io/en/stable/configuration/images/#annotation-format

On force-update: https://argocd-image-updater.readthedocs.io/en/stable/configuration/images/#forcing-image-updates
It's usually used to force the image updates when there is 0 replica (such as in a batch job)

[dwilliams782]

In your application annotations, such as argocd-image-updater.argoproj.io/write-back-method=argocd, ":" should be used instead of "=". Not sure if this has caused differences in parsing the annotation values.

see https://argocd-image-updater.readthedocs.io/en/stable/configuration/images/#annotation-format

Good morning, thanks for the reply.

I copied those annotations out of ArgoCD's UI where they are inserting = for whatever reason. Our actual manifest looks like:

  templatePatch: |
    metadata:
      annotations:
      {{- if eq .service "crm-backend" }}
        argocd-image-updater.argoproj.io/image-list: "{{ .service }}=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/{{ .service }},crm-db-migrations=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/crm-db-migrations"
        argocd-image-updater.argoproj.io/crm-db-migrations.update-strategy: latest
        argocd-image-updater.argoproj.io/crm-db-migrations.allow-tags: "regexp:^eph-{{ .number }}-.+$"
        argocd-image-updater.argoproj.io/crm-db-migrations.helm.image-tag: initContainers.db-migrations.imageTag
        argocd-image-updater.argoproj.io/crm-db-migrations.helm.image-name: initContainers.db-migrations.imageRepository
        argocd-image-updater.argoproj.io/crm-db-migrations.force-update: "true"
      {{- else }}
        argocd-image-updater.argoproj.io/image-list: "{{ .service }}=<staging-ecr>.dkr.ecr.eu-west-2.amazonaws.com/{{ .service }}"
      {{- end }}
        argocd-image-updater.argoproj.io/{{ .service }}.update-strategy: latest
        argocd-image-updater.argoproj.io/{{ .service }}.allow-tags: "regexp:^eph-{{ .number }}-.+$"
        argocd-image-updater.argoproj.io/{{ .service }}.helm.image-tag: container.imageTag
        argocd-image-updater.argoproj.io/{{ .service }}.helm.image-name: container.imageRepository
        argocd-image-updater.argoproj.io/{{ .service }}.force-update: "true"
        argocd-image-updater.argoproj.io/write-back-method: argocd

so we are using : as required.

On force-update: https://argocd-image-updater.readthedocs.io/en/stable/configuration/images/#forcing-image-updates It's usually used to force the image updates when there is 0 replica (such as in a batch job)

Because we are asking image-updater to change the initial image to a different registry, force is required as otherwise we got a message saying the new image is not used in the application (as it's not in the list of images used in the Argo app).

[chengfang]

you may want to try the master branch build if possible, since there are quite some changes since image updater v0.16.0 release. There is also a pending PR #1172 which will potentially affect the force-update behavior.