Adopting IAM Policy with adopt-or-create results in resource already exists error on subsequent reconcile

[Juricakov]

Describe the bug
When adopting Policy it results in error with resource already exists error.

It looks like ack metadata arn and adopted annotation are not set, finalizers are.

Second reconcile does not try to perform adoption due to finalizers existing, and it doesn't find the resource in AWS as arn is null, so it tries to create Policy and fails.

Steps to reproduce

• Apply Policy CRD for policy that already exists with adoption policy adopt or create and adoption fields containing arn
• IAM controller will try to create it and get resource already exists error

Expected outcome

• Policy is adopted instead of being recreated

Environment

• Kubernetes version 1.31
• Using EKS (yes/no), if so version? Yes
• AWS service targeted (S3, RDS, etc.) IAM

Additional info
CRD:

apiVersion: iam.services.k8s.aws/v1alpha1
kind: Policy
metadata:
  annotations:
    services.k8s.aws/adoption-fields: |
      {
        "arn": "arn:aws:iam::<accID>:policy/meta/<table-name>-reader"
      }
    services.k8s.aws/adoption-policy: adopt-or-create
    services.k8s.aws/deletion-policy: retain
  creationTimestamp: "2025-05-06T12:00:58Z"
  finalizers:
  - finalizers.iam.services.k8s.aws/Policy
  generation: 1
  name: <table-name>-reader
  namespace: ack-system
  resourceVersion: "2176756545"
  uid: fc0c540f-1cf9-4c68-b930-fbc8fed86073
spec:
  description: Read policies for the <table-name>.
  name: <table-name>-reader
  path: /meta/
  policyDocument: |
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "",
                "Effect": "Allow",
                "Action": [
                    "dynamodb:Scan",
                    "dynamodb:Query",
                    "dynamodb:ListTables",
                    "dynamodb:GetItem",
                    "dynamodb:DescribeTable",
                    "dynamodb:BatchGetItem"
                ],
                "Resource": [
                    "arn:aws:dynamodb:eu-central-1:<accID>:table/<table-name>",
                    "arn:aws:dynamodb:eu-central-1:<accID>:table/<table-name>/index/*"
                ]
            }
        ]
    }

CRD after first reconcile - note missing ackResourceMetadata.arn and adopted annotation:

apiVersion: iam.services.k8s.aws/v1alpha1
kind: Policy
metadata:
  annotations:
    services.k8s.aws/adoption-fields: |
      {
        "arn": "arn:aws:iam::<accID>:policy/meta/<table-name>-reader"
      }
    services.k8s.aws/adoption-policy: adopt-or-create
    services.k8s.aws/deletion-policy: retain
  creationTimestamp: "2025-05-06T12:00:58Z"
  finalizers:
  - finalizers.iam.services.k8s.aws/Policy
  generation: 1
  name: <table-name>-reader
  namespace: ack-system
  resourceVersion: "2176756552"
  uid: fc0c540f-1cf9-4c68-b930-fbc8fed86073
spec:
  description: Read policies for the <table-name>.
  name: <table-name>-reader
  path: /meta/
  policyDocument: |
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "",
                "Effect": "Allow",
                "Action": [
                    "dynamodb:Scan",
                    "dynamodb:Query",
                    "dynamodb:ListTables",
                    "dynamodb:GetItem",
                    "dynamodb:DescribeTable",
                    "dynamodb:BatchGetItem"
                ],
                "Resource": [
                    "arn:aws:dynamodb:eu-central-1:<accID>:table/<table-name>",
                    "arn:aws:dynamodb:eu-central-1:<accID>:table/<table-name>/index/*"
                ]
            }
        ]
    }
status:
  ackResourceMetadata:
    ownerAccountID: "<accID>"
    region: eu-central-1
  attachmentCount: 1
  conditions:
  - lastTransitionTime: "2025-05-06T12:00:58Z"
    message: Late initialization successful
    reason: Late initialization successful
    status: "True"
    type: ACK.LateInitialized
  - lastTransitionTime: "2025-05-06T12:00:58Z"
    message: Resource synced successfully
    reason: ""
    status: "True"
    type: ACK.ResourceSynced
  createDate: "2025-03-11T23:47:03Z"
  defaultVersionID: v1
  isAttachable: true
  permissionsBoundaryUsageCount: 0
  policyID: <policyID>
  updateDate: "2025-03-11T23:47:03Z"

Logs after first reconcile:

{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":"> r.Sync","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":">> r.resetConditions","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":"<< r.resetConditions","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":">> rm.ResolveReferences","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":"<< rm.ResolveReferences","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":">> r.handlePopulation","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":"Populating Resource","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":"<< r.handlePopulation","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":">> rm.EnsureTags","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":"<< rm.EnsureTags","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":">> rm.ReadOne","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.146Z","logger":"ackrt","msg":">>> rm.sdkFind","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.281Z","logger":"ackrt","msg":">>>> rm.getPolicyVersion","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.403Z","logger":"ackrt","msg":"<<<< rm.getPolicyVersion","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.404Z","logger":"ackrt","msg":"<<< rm.sdkFind","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.404Z","logger":"ackrt","msg":"<< rm.ReadOne","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.404Z","logger":"ackrt","msg":">> r.setResourceManaged","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.404Z","logger":"ackrt","msg":">>> r.patchResourceMetadataAndSpec","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.404Z","logger":"ackrt","msg":">>>> kc.Patch (metadata + spec)","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.412Z","logger":"ackrt","msg":"patched resource metadata + spec","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"json":"{\"metadata\":{\"finalizers\":[\"finalizers.iam.services.k8s.aws/Policy\"]}}"}
{"level":"debug","ts":"2025-05-06T12:00:58.412Z","logger":"ackrt","msg":"<<<< kc.Patch (metadata + spec)","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.412Z","logger":"ackrt","msg":"<<< r.patchResourceMetadataAndSpec","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.412Z","logger":"ackrt","msg":"marked resource as managed","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.413Z","logger":"ackrt","msg":"<< r.setResourceManaged","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.413Z","logger":"ackrt","msg":">> r.updateResource","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.413Z","logger":"ackrt","msg":"<< r.updateResource","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.413Z","logger":"ackrt","msg":">> r.lateInitializeResource","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.413Z","logger":"ackrt","msg":">>> rm.LateInitialize","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.413Z","logger":"ackrt","msg":">>>> rm.sdkFind","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.537Z","logger":"ackrt","msg":">>>>> rm.getPolicyVersion","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"<<<<< rm.getPolicyVersion","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"<<<< rm.sdkFind","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"<<< rm.LateInitialize","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":">>> r.patchResourceMetadataAndSpec","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"no difference found between metadata and spec for desired and latest object.","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"<<< r.patchResourceMetadataAndSpec","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"<< r.lateInitializeResource","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":">> r.ensureConditions","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":">>> rm.IsSynced","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"<<< rm.IsSynced","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"<< r.ensureConditions","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"< r.Sync","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"requeuing","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"after":36000}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":"> r.patchResourceStatus","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.665Z","logger":"ackrt","msg":">> kc.Patch (status)","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.679Z","logger":"ackrt","msg":"patched resource status","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"json":"{\"metadata\":{\"finalizers\":[\"finalizers.iam.services.k8s.aws/Policy\"],\"resourceVersion\":\"2176756552\"},\"spec\":{\"tags\":null},\"status\":{\"ackResourceMetadata\":{\"arn\":null,\"ownerAccountID\":\"<accID>\",\"region\":\"eu-central-1\"},\"attachmentCount\":1,\"conditions\":[{\"lastTransitionTime\":\"2025-05-06T12:00:58Z\",\"message\":\"Late initialization successful\",\"reason\":\"Late initialization successful\",\"status\":\"True\",\"type\":\"ACK.LateInitialized\"},{\"lastTransitionTime\":\"2025-05-06T12:00:58Z\",\"message\":\"Resource synced successfully\",\"reason\":\"\",\"status\":\"True\",\"type\":\"ACK.ResourceSynced\"}],\"createDate\":\"2025-03-11T23:47:03Z\",\"defaultVersionID\":\"v1\",\"isAttachable\":true,\"permissionsBoundaryUsageCount\":0,\"policyID\":\"<policyID>\",\"updateDate\":\"2025-03-11T23:47:03Z\"}}"}
{"level":"debug","ts":"2025-05-06T12:00:58.679Z","logger":"ackrt","msg":"<< kc.Patch (status)","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.679Z","logger":"ackrt","msg":"< r.patchResourceStatus","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:00:58.679Z","logger":"ackrt","msg":"requeueing","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"after":36000}

CRD after second reconcile:

apiVersion: iam.services.k8s.aws/v1alpha1
kind: Policy
metadata:
  annotations:
    services.k8s.aws/adoption-fields: |
      {
        "arn": "arn:aws:iam::<accID>:policy/meta/<table-name>-reader"
      }
    services.k8s.aws/adoption-policy: adopt-or-create
    services.k8s.aws/deletion-policy: retain
  creationTimestamp: "2025-05-06T12:00:58Z"
  finalizers:
    - finalizers.iam.services.k8s.aws/Policy
  generation: 1
  name: <table-name>-reader
  namespace: ack-system
  resourceVersion: "2176784567"
  uid: fc0c540f-1cf9-4c68-b930-fbc8fed86073
spec:
  description: Read policies for the <table-name>.
  name: <table-name>-reader
  path: /meta/
  policyDocument: |
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "",
                "Effect": "Allow",
                "Action": [
                    "dynamodb:Scan",
                    "dynamodb:Query",
                    "dynamodb:ListTables",
                    "dynamodb:GetItem",
                    "dynamodb:DescribeTable",
                    "dynamodb:BatchGetItem"
                ],
                "Resource": [
                    "arn:aws:dynamodb:eu-central-1:<accID>:table/<table-name>",
                    "arn:aws:dynamodb:eu-central-1:<accID>:table/<table-name>/index/*"
                ]
            }
        ]
    }
status:
  ackResourceMetadata:
    ownerAccountID: "<accID>"
    region: eu-central-1
  attachmentCount: 1
  conditions:
    - message: 'EntityAlreadyExists: A policy called <table-name>-reader already exists. Duplicate names are not allowed.'
      status: "True"
      type: ACK.Recoverable
    - lastTransitionTime: "2025-05-06T12:13:05Z"
      message: Unable to determine if desired resource state matches latest observed state
      reason: 'operation error IAM: CreatePolicy, https response error StatusCode: 409, RequestID: bc092234-d90c-40f8-aa74-ac7b374b98c8, EntityAlreadyExists: A policy called <table-name>-reader already exists. Duplicate names are not allowed.'
      status: Unknown
      type: ACK.ResourceSynced
  createDate: "2025-03-11T23:47:03Z"
  defaultVersionID: v1
  isAttachable: true
  permissionsBoundaryUsageCount: 0
  policyID: <policyID>
  updateDate: "2025-03-11T23:47:03Z"

Logs after second reconcile:

{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":"> r.Sync","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":">> r.resetConditions","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":"<< r.resetConditions","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":">> rm.ResolveReferences","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":"<< rm.ResolveReferences","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":">> rm.EnsureTags","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":"<< rm.EnsureTags","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":">> rm.ReadOne","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":">>> rm.sdkFind","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":"<<< rm.sdkFind","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"error":"resource not found"}
{"level":"debug","ts":"2025-05-06T12:12:31.080Z","logger":"ackrt","msg":"<< rm.ReadOne","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"error":"resource not found"}
{"level":"debug","ts":"2025-05-06T12:12:31.081Z","logger":"ackrt","msg":">> r.createResource","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.081Z","logger":"ackrt","msg":">>> rm.Create","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.081Z","logger":"ackrt","msg":">>>> rm.sdkCreate","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.198Z","logger":"ackrt","msg":"<<<< rm.sdkCreate","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"error":"operation error IAM: CreatePolicy, https response error StatusCode: 409, RequestID: 6b00623a-bccc-48c3-aa5c-0f8fc4c1f9ae, EntityAlreadyExists: A policy called <table-name>-reader already exists. Duplicate names are not allowed."}
{"level":"debug","ts":"2025-05-06T12:12:31.199Z","logger":"ackrt","msg":"<<< rm.Create","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"error":"operation error IAM: CreatePolicy, https response error StatusCode: 409, RequestID: 6b00623a-bccc-48c3-aa5c-0f8fc4c1f9ae, EntityAlreadyExists: A policy called <table-name>-reader already exists. Duplicate names are not allowed."}
{"level":"debug","ts":"2025-05-06T12:12:31.199Z","logger":"ackrt","msg":"<< r.createResource","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"error":"operation error IAM: CreatePolicy, https response error StatusCode: 409, RequestID: 6b00623a-bccc-48c3-aa5c-0f8fc4c1f9ae, EntityAlreadyExists: A policy called <table-name>-reader already exists. Duplicate names are not allowed."}
{"level":"debug","ts":"2025-05-06T12:12:31.199Z","logger":"ackrt","msg":">> r.ensureConditions","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.199Z","logger":"ackrt","msg":">>> rm.IsSynced","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.199Z","logger":"ackrt","msg":"<<< rm.IsSynced","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.199Z","logger":"ackrt","msg":"<< r.ensureConditions","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.199Z","logger":"ackrt","msg":"< r.Sync","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"error":"operation error IAM: CreatePolicy, https response error StatusCode: 409, RequestID: 6b00623a-bccc-48c3-aa5c-0f8fc4c1f9ae, EntityAlreadyExists: A policy called <table-name>-reader already exists. Duplicate names are not allowed."}
{"level":"debug","ts":"2025-05-06T12:12:31.199Z","logger":"ackrt","msg":"> r.patchResourceStatus","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.199Z","logger":"ackrt","msg":">> kc.Patch (status)","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.212Z","logger":"ackrt","msg":"patched resource status","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1,"json":"{\"metadata\":{\"resourceVersion\":\"2176783227\"},\"spec\":{\"tags\":null},\"status\":{\"conditions\":[{\"message\":\"EntityAlreadyExists: A policy called <table-name>-reader already exists. Duplicate names are not allowed.\",\"status\":\"True\",\"type\":\"ACK.Recoverable\"},{\"lastTransitionTime\":\"2025-05-06T12:12:31Z\",\"message\":\"Unable to determine if desired resource state matches latest observed state\",\"reason\":\"operation error IAM: CreatePolicy, https response error StatusCode: 409, RequestID: 6b00623a-bccc-48c3-aa5c-0f8fc4c1f9ae, EntityAlreadyExists: A policy called <table-name>-reader already exists. Duplicate names are not allowed.\",\"status\":\"Unknown\",\"type\":\"ACK.ResourceSynced\"}]}}"}
{"level":"debug","ts":"2025-05-06T12:12:31.212Z","logger":"ackrt","msg":"<< kc.Patch (status)","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"debug","ts":"2025-05-06T12:12:31.212Z","logger":"ackrt","msg":"< r.patchResourceStatus","kind":"Policy","namespace":"ack-system","name":"<table-name>-reader","account":"<accID>","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
{"level":"error","ts":"2025-05-06T12:12:31.212Z","msg":"Reconciler error","controller":"policy","controllerGroup":"iam.services.k8s.aws","controllerKind":"Policy","Policy":{"name":"<table-name>-reader","namespace":"ack-system"},"namespace":"ack-system","name":"<table-name>-reader","reconcileID":"7992a86e-ac7c-4d51-bff6-81a4f3db8f03","error":"operation error IAM: CreatePolicy, https response error StatusCode: 409, RequestID: 6b00623a-bccc-48c3-aa5c-0f8fc4c1f9ae, EntityAlreadyExists: A policy called <table-name>-reader already exists. Duplicate names are not allowed.","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255"}

[github-actions]

Hello @Juricakov 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[Juricakov]

@michaelhtm I saw some commits already, thanks so much for being so quick!

Just in case - I noticed DynamoDB table is also missing the adopted annotation. I assume the existing commit will fix that issue as well.

[michaelhtm]

Hi @Juricakov
Of course!
The changes should be part of the next release :)

[michaelhtm]

Hello @Juricakov
We just released a fix addressing your issue in all controllers! The IAM fix should be in v1.3.21

[Juricakov]

Hey, I tried, adoption works! Arn is set in status, subsequent reconciles succeed without error, however - adopted annotation no longer appears.

This is not critical for me, I updated scripts to rely on ACK.ResourceSynced status instead of waiting for adopted annotation.

[michaelhtm]

Thank you for the feedback @Juricakov. Seems like there was a bug during the update path of an adopt-or-create.

[Juricakov]

Hey unsure if related, let me know if I should create a different issue. One more thing I noticed is that sometimes we get panic on dynamodb controller. It looks like it tries to sync tags and panics when trying to read arn from status ack resource metadata.

Tags were completely in sync with AWS except operator version tag.

I am able to work around it by manually updating controller version tag in AWS and recreating the table CRD.

It does not happen for all table CRDs that are being adopted, but it happens on every subsequent retry for the affected Table CRDs.

Not sure how to reproduce it.

[michaelhtm]

Seems like this is a dynamoDB issue..No need to create a new issue :)