ACK ACM constantly updating status ACK.LateInitialized for certificate

[demikl]

Describe the bug
The field .status.conditions[].lastTransitionTime for condition type "ACK.LateInitialized" is updated every minute. The metric workqueue_retries_total{controller="certificate"} is also increased each minute.

Steps to reproduce

Expected outcome
Once the certificate CR is synced with AWS, lastTransitionTime should not change.

Environment

• Kubernetes version : v1.32.6-eks-931bdca
• Using EKS (yes/no), if so version? yes ==> v1.32.6-eks-931bdca
• AWS service targeted (S3, RDS, etc.) : ACM
• version of controller: acm-controller:1.1.0

Logs

Here are the logs from the controller, with level DEBUG :

{"level":"debug","ts":"2025-08-28T09:38:54.567Z","logger":"ackrt","msg":"> r.Sync","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.567Z","logger":"ackrt","msg":">> r.resetConditions","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.567Z","logger":"ackrt","msg":"<< r.resetConditions","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.567Z","logger":"ackrt","msg":">> rm.ResolveReferences","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.567Z","logger":"ackrt","msg":"<< rm.ResolveReferences","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.567Z","logger":"ackrt","msg":">> rm.EnsureTags","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.567Z","logger":"ackrt","msg":"<< rm.EnsureTags","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.567Z","logger":"ackrt","msg":">> rm.ReadOne","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.567Z","logger":"ackrt","msg":">>> rm.sdkFind","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.664Z","logger":"ackrt","msg":">>>> rm.listTags","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.692Z","logger":"ackrt","msg":"<<<< rm.listTags","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.692Z","logger":"ackrt","msg":"<<< rm.sdkFind","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.692Z","logger":"ackrt","msg":"<< rm.ReadOne","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.692Z","logger":"ackrt","msg":">> r.updateResource","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.692Z","logger":"ackrt","msg":"<< r.updateResource","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.693Z","logger":"ackrt","msg":">> r.lateInitializeResource","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.693Z","logger":"ackrt","msg":">>> rm.LateInitialize","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.693Z","logger":"ackrt","msg":">>>> rm.sdkFind","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.749Z","logger":"ackrt","msg":">>>>> rm.listTags","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.773Z","logger":"ackrt","msg":"<<<<< rm.listTags","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.773Z","logger":"ackrt","msg":"<<<< rm.sdkFind","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.774Z","logger":"ackrt","msg":"<<< rm.LateInitialize","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.774Z","logger":"ackrt","msg":">>> r.patchResourceMetadataAndSpec","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.774Z","logger":"ackrt","msg":"no difference found between metadata and spec for desired and latest object.","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.774Z","logger":"ackrt","msg":"<<< r.patchResourceMetadataAndSpec","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.774Z","logger":"ackrt","msg":"<< r.lateInitializeResource","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.775Z","logger":"ackrt","msg":">> r.ensureConditions","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.775Z","logger":"ackrt","msg":"<< r.ensureConditions","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.775Z","logger":"ackrt","msg":"< r.Sync","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.775Z","logger":"ackrt","msg":"requeuing","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1,"after":60}
{"level":"debug","ts":"2025-08-28T09:38:54.775Z","logger":"ackrt","msg":"> r.patchResourceStatus","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.775Z","logger":"ackrt","msg":">> kc.Patch (status)","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.795Z","logger":"ackrt","msg":"patched resource status","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1,"json":"{\"metadata\":{\"resourceVersion\":\"867154530\"},\"spec\":{\"tags\":null},\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"ACK.ResourceSynced\"},{\"lastTransitionTime\":\"2025-08-28T09:38:54Z\",\"message\":\"Late initialization successful\",\"reason\":\"Late initialization successful\",\"status\":\"True\",\"type\":\"ACK.LateInitialized\"}]}}"}
{"level":"debug","ts":"2025-08-28T09:38:54.795Z","logger":"ackrt","msg":"<< kc.Patch (status)","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.795Z","logger":"ackrt","msg":"< r.patchResourceStatus","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1}
{"level":"debug","ts":"2025-08-28T09:38:54.795Z","logger":"ackrt","msg":"requeueing","kind":"Certificate","namespace":"kube-mgmt","name":"certificate","account":"REDACTED","role":"","region":"eu-west-3","is_adopted":true,"generation":1,"after":60}

[github-actions]

Hello @demikl 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[michaelhtm]

This issue seems to be happening because ACM certificate has a default one minute resync period

You can define your custom resync period if you'd like in the helm values in the following format:

  resourceResyncPeriods: 
    Certificate=3600 # for one hour resync period

https://github.com/aws-controllers-k8s/acm-controller/blob/main/helm/values.yaml#L136

[demikl]

Thanks for the explanation and the workaround.

[michaelhtm]

Of course!
Closing this issue as resolved. Feel free to reopen if otherwise
/close

[ack-prow]

@michaelhtm: Closing this issue.

In response to this:

Of course!
Closing this issue as resolved. Feel free to reopen if otherwise
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.