App of apps being overwritten by image-updater

[mehdicopter]

Describe the bug

I am using ArgoCD with the “App of Apps” pattern. After updating argo-cd-image-updater to version 0.15.0, I encountered an unexpected side effect.

When updating the image of a child application, ArgoCD also updates the parent application (“App of Apps”). This causes a resource conflict, as both the child application (“myapp”) and the parent application (“root”) end up supervising the same resources.

To Reproduce

1. Use ArgoCD with the “App of Apps” pattern.
2. Update argo-cd-image-updater to version 0.15.0.
3. Perform an image update for a child application.
4. Observe that the parent application also attempts to supervise the same resources.

Expected behavior

Only the child application (“myapp”) should be updated when its image is changed, without the parent application (“root”) taking control over the same resources.

Additional context

This issue was not present with the previous version of argo-cd-image-updater (0.14.x).

Version

{
    "Version": "v2.12.6+4dab5bd",
    "BuildDate": "2024-10-18T17:39:26Z",
    "GitCommit": "4dab5bd6a60adea12e084ad23519e35b710060a2",
    "GitTreeState": "clean",
    "GoVersion": "go1.22.4",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.4.2 2024-05-22T15:19:38Z",
    "HelmVersion": "v3.15.2+g1a500d5",
    "KubectlVersion": "v0.29.6",
    "JsonnetVersion": "v0.20.0"
}

[chengfang]

It's not obvious which commits may cause this regression in v0.15.0. #854 looks a bit suspicious.

Is it possible to filter out the parent app via command line options --match-application-label --match-application-name?

[mehdicopter]

I am trying to use those filters but I am having those errors:

klf argocd-image-updater-5bdb94f977-56hcv
Error: unknown flag: --match-application-label app.company.com/name

What am i doing wrong ? 😬

[mehdicopter]

I am using kustomize to update the deployment of argocd-image-updater

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-image-updater
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-image-updater
  template:
    spec:
      volumes:
        - name: scripts
          configMap:
            name: argocd-image-updater-scripts
            defaultMode: 0777
      containers:
        - name: argocd-image-updater
          args:
            - run
            - --match-application-label app.company.com/name=myapp
          volumeMounts:
            - name: scripts
              mountPath: /scripts

[jannfis]

In your snippet below, the command line switch and its parameter are being passed as a single argument.

To fix it, you can either use

          args:
            - run
            - --match-application-label=app.company.com/name=myapp

(note the equal sign between the parameter and the value) or

          args:
            - run
            - --match-application-label
            - app.company.com/name=myapp

[mehdicopter]

Even with the matching label it does the same... look at the screenshot.

[mehdicopter]

The root app-of-apps is behaving like being the app, resulting of having 2 argo apps responsible to handle resources, which causes SharedResourceWarning

[mehdicopter]

According to the logs, it does not even update the root application.

time="2024-10-24T22:54:12Z" level=debug msg="Applications listed: 12"
time="2024-10-24T22:54:12Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2024-10-24T22:54:12Z" level=debug msg="Processing application argocd/myapp-staging"
time="2024-10-24T22:54:12Z" level=debug msg="Considering this image for update" alias=adserver application=myapp-staging image_name=xxxx/oci-xxx/adserver image_tag="sha256:1a15f767519b1ce8d73130cc7b6e6a8787c12482f3a61dbe4c6d7bbc5d5d6c27" registry=europe-west1-docker.pkg.dev
time="2024-10-24T22:54:12Z" level=debug msg="Using version constraint 'staging' when looking for a new tag" alias=adserver application=myapp-staging image_name=xxxx/oci-xxx/adserver image_tag="sha256:1a15f767519b1ce8d73130cc7b6e6a8787c12482f3a61dbe4c6d7bbc5d5d6c27" registry=europe-west1-docker.pkg.dev
time="2024-10-24T22:54:13Z" level=debug msg="found 1 from 1 tags eligible for consideration" image="europe-west1-docker.pkg.dev/xxxx/oci-xxx/adserver@sha256:1a15f767519b1ce8d73130cc7b6e6a8787c12482f3a61dbe4c6d7bbc5d5d6c27"
time="2024-10-24T22:54:13Z" level=info msg="Setting new image to europe-west1-docker.pkg.dev/xxxx/oci-xxx/adserver:staging@sha256:63f7bebb86c43d8a1a71a8394f6f576731acf08a97dd0279a830d4bba8406c36" alias=adserver application=myapp-staging image_name=xxxx/oci-xxx/adserver image_tag="sha256:1a15f767519b1ce8d73130cc7b6e6a8787c12482f3a61dbe4c6d7bbc5d5d6c27" registry=europe-west1-docker.pkg.dev
time="2024-10-24T22:54:13Z" level=info msg="Successfully updated image 'europe-west1-docker.pkg.dev/xxxx/oci-xxx/adserver@sha256:1a15f767519b1ce8d73130cc7b6e6a8787c12482f3a61dbe4c6d7bbc5d5d6c27' to 'europe-west1-docker.pkg.dev/xxxx/oci-xxx/adserver:staging@sha256:63f7bebb86c43d8a1a71a8394f6f576731acf08a97dd0279a830d4bba8406c36', but pending spec update (dry run=false)" alias=adserver application=myapp-staging image_name=xxxx/oci-xxx/adserver image_tag="sha256:1a15f767519b1ce8d73130cc7b6e6a8787c12482f3a61dbe4c6d7bbc5d5d6c27" registry=europe-west1-docker.pkg.dev
time="2024-10-24T22:54:13Z" level=debug msg="Using commit message: "
time="2024-10-24T22:54:13Z" level=info msg="Committing 1 parameter update(s) for application myapp-staging" application=myapp-staging
time="2024-10-24T22:54:13Z" level=debug msg="Getting application myapp-staging across all namespaces"
time="2024-10-24T22:54:13Z" level=debug msg="Applications listed: 39"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: argo-cd in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: autopilot-bootstrap in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: blackbox-exporter in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: cluster-resources-in-cluster in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxxx-ui-back-dev in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxxx-ui-front-dev in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-dns-xxxx-dev in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-dns-in-cluster in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-dns-xxx-preprod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-dns-xxx-prod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-dns-xxx-staging in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-dns-xxxx-prod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-dns-xxxx-stng in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-dns-xxxx-test in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-secrets-xxxx-dev in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-secrets-gitlab-runner in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-secrets-in-cluster in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-secrets-xxx-preprod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-secrets-xxx-prod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-secrets-xxx-staging in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-secrets-xxxx-prod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-secrets-xxxx-stng in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: external-secrets-xxxx-test in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: gitlab-runner in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: grafana in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: prometheus in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: myapp-preprod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: myapp-prod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: myapp-staging in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Application myapp-staging matches the pattern"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxx-backend-preprod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxx-backend-prod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxx-backend-staging in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxx-frontend-preprod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxx-frontend-prod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxx-frontend-staging in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxx-fusionauth-preprod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxx-fusionauth-prod in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: xxx-fusionauth-staging in namespace argocd"
time="2024-10-24T22:54:13Z" level=debug msg="Found application: root in namespace argocd"
time="2024-10-24T22:54:13Z" level=info msg="Successfully updated the live application spec" application=myapp-staging
time="2024-10-24T22:54:13Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0"

[mehdicopter]

Does the logs help you @chengfang ?

[jannfis]

What are the SharedResourceWarning's details?

[mehdicopter]

What are the SharedResourceWarning's details?

time="2024-10-24T02:59:51Z" level=info msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2024-10-24T02:59:50Z\",\"message\":\"ConfigMap/datawiz-ui-front is part of applications argocd/datawiz-ui-front-dev and root\",\"type\":\"SharedResourceWarning\"},{\"lastTransitionTime\":\"2024-10-24T02:59:50Z\",\"message\":\"Deployment/datawiz-ui-front is part of applications argocd/datawiz-ui-front-dev and root\",\"type\":\"SharedResourceWarning\"},{\"lastTransitionTime\":\"2024-10-24T02:59:50Z\",\"message\":\"Service/datawiz-ui-front is part of applications argocd/datawiz-ui-front-dev and root\",\"type\":\"SharedResourceWarning\"}]}}" app-namespace=argocd app-qualified-name=argocd/datawiz-ui-front-dev application=datawiz-ui-front-dev project=datawiz

[jannfis]

At this point, I highly doubt that this has to do with the Image Updater. Image Updater itself doesn't manage or manipulate resources such as ConfigMaps, or other types.

Are you using a mono repo for all your apps, including the root app by any chance?

[mehdicopter]

At this point, I highly doubt that this has to do with the Image Updater. Image Updater itself doesn't manage or manipulate resources such as ConfigMaps, or other types.

Are you using a mono repo for all your apps, including the root app by any chance?

Ok I understand. But explain why it is working in 0.14.0 ?
I am using the same cluster, same repo which contains all ArgoCD apps.

[jannfis]

Can you post your root application's spec here?

[mehdicopter]

Can you post your root application's spec here?

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  labels:
    app.kubernetes.io/managed-by: argocd-autopilot
    app.kubernetes.io/name: root
  name: root
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  ignoreDifferences:
    - group: argoproj.io
      jsonPointers:
        - /status
      kind: Application
  project: default
  source:
    path: projects
    repoURL: https://gitlab.com/xxx/argocd.git
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true
    syncOptions:
      - allowEmpty=true
status:
  health: {}
  summary: {}
  sync:
    comparedTo:
      destination: {}
      source:
        repoURL: ""
    status: ""