[SNS] cannot create sns subscription

[michaelhtm]

Description

Issue from kubernetes-sigs/kro#521
Observed Behavior:
Creating the topic manually using the same IAM role results in no error.
Creating the topic using a manifest file also works:

apiVersion: sns.services.k8s.aws/v1alpha1
kind: Topic
metadata:
  name: test
spec:
  name: test
  applicationFailureFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  applicationSuccessFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  applicationSuccessFeedbackSampleRate: "100"
  firehoseFailureFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  firehoseSuccessFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  firehoseSuccessFeedbackSampleRate: "100"
  httpFailureFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  httpSuccessFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  httpSuccessFeedbackSampleRate: "100"
  lambdaFailureFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  lambdaSuccessFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  lambdaSuccessFeedbackSampleRate: "100"
  sqsFailureFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  sqsSuccessFeedbackRoleARN: arn:aws:iam::12345:role/bitcoin-snsfeedback
  sqsSuccessFeedbackSampleRate: "100"

When creating a topic I see an error in the status field:

Status:
  Ack Resource Metadata:
    Owner Account ID:  123456
    Region:            eu-central-1
  Conditions:
    Message:               InvalidParameter: Invalid parameter: Attributes Reason: FirehoseFailureFeedbackRoleArn: arn:aws:iam::123456:role/bitcoin-snsfeedback is not a valid role to allow SNS to write to Cloudwatch Logs
    Status:                True
    Type:                  ACK.Terminal
    Last Transition Time:  2025-04-20T07:22:17Z
    Message:               Resource not synced
    Reason:                resource is in terminal condition
    Status:                False
    Type:                  ACK.ResourceSynced

In addition nothing happens in ack and kro controller logs, only after restarting the controller it detects the new topic and creates it successfully:

ack-sns-dff5b79fb-tzxv7 controller {"level":"info","ts":"2025-04-20T07:30:59.331Z","logger":"ackrt","msg":"created new resource","kind":"Topic","namespace":"pagerduty-inbound","name":"bitcoin","account":"123456","role":"","region":"eu-central-1","is_adopted":false,"generation":1}
ack-sns-dff5b79fb-tzxv7 controller {"level":"info","ts":"2025-04-20T07:30:59.399Z","logger":"ackrt","msg":"desired resource state has changed","kind":"Topic","namespace":"pagerduty-inbound","name":"bitcoin","account":"1234","role":"","region":"eu-central-1","is_adopted":false,"generation":2,"diff":[{"Path":{"Parts":["Spec","DeliveryPolicy"]},"A":"{\"http\":{\"defaultHealthyRetryPolicy\":{\"backoffFunction\":\"linear\",\"maxDelayTarget\":20,\"minDelayTarget\":20,\"numRetries\":3},\"defaultRequestPolicy\":{\"headerContentType\":\"text/plain; charset=UTF-8\"},\"disableSubscriptionOverrides\":false}}","B":"{\"http\":{\"defaultHealthyRetryPolicy\":{\"minDelayTarget\":20,\"maxDelayTarget\":20,\"numRetries\":3,\"backoffFunction\":\"linear\"},\"disableSubscriptionOverrides\":false,\"defaultRequestPolicy\":{\"headerContentType\":\"text/plain; charset=UTF-8\"}}}"},{"Path":{"Parts":["Spec","DisplayName"]},"A":null,"B":""}]}
ack-sns-dff5b79fb-tzxv7 controller {"level":"info","ts":"2025-04-20T07:30:59.440Z","logger":"ackrt","msg":"updated resource","kind":"Topic","namespace":"pagerduty-inbound","name":"bitcoin","account":"123456","role":"","region":"eu-central-1","is_adopted":false,"generation":2}

Expected Behavior:
Topic to be created successfully
Reproduction Steps (Please include ResourceGraphDefinition and Instances files):

Use the following RGD:

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: customintegration
spec:
  schema:
    apiVersion: v1alpha1
    kind: CustomIntegration
    spec:
      name: string
      sns:
        enabled: boolean | default=false
        name: string
        deliveryPolicy: string
        applicationSuccessFeedbackSampleRate: string | default="100"
        firehoseSuccessFeedbackSampleRate: string | default="100"
        httpSuccessFeedbackSampleRate: string | default="100"
        lambdaSuccessFeedbackSampleRate: string | default="100"
        sqsSuccessFeedbackSampleRate: string | default="100"
        region: string
        accountId: string
  resources:
    - id: iamPolicy
      includeWhen:
        - ${schema.spec.sns.enabled}
      template:
        apiVersion: iam.services.k8s.aws/v1alpha1
        kind: Policy
        metadata:
          name: ${schema.spec.name}-snsfeedback
        spec:
          name: ${schema.spec.name}
          policyDocument: |
            {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "logs:CreateLogGroup",
                    "logs:CreateLogStream",
                    "logs:PutLogEvents",
                    "logs:PutMetricFilter",
                    "logs:PutRetentionPolicy"
                  ],
                  "Resource": [
                    "*"
                  ]
                }
              ]
            }
    - id: iamRole
      includeWhen:
        - ${schema.spec.sns.enabled}
      template:
        apiVersion: iam.services.k8s.aws/v1alpha1
        kind: Role
        metadata:
          name: ${schema.spec.name}-snsfeedback
        spec:
          assumeRolePolicyDocument: |
            {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Principal": {
                    "Service": "sns.amazonaws.com"
                  },
                  "Action": "sts:AssumeRole"
                }
              ]
            }
          name: ${schema.spec.name}-snsfeedback
          policies:
          - ${iamPolicy.status.ackResourceMetadata.arn}
    - id: topic
      includeWhen:
        - ${schema.spec.sns.enabled}
      template:
        apiVersion: sns.services.k8s.aws/v1alpha1
        kind: Topic
        metadata:
          name: ${schema.spec.sns.name}
        spec:
          name: ${schema.spec.sns.name}
          deliveryPolicy: ${schema.spec.sns.deliveryPolicy}
          applicationFailureFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          applicationSuccessFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          applicationSuccessFeedbackSampleRate: ${schema.spec.sns.applicationSuccessFeedbackSampleRate}
          firehoseFailureFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          firehoseSuccessFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          firehoseSuccessFeedbackSampleRate: ${schema.spec.sns.firehoseSuccessFeedbackSampleRate}
          httpFailureFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          httpSuccessFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          httpSuccessFeedbackSampleRate: ${schema.spec.sns.httpSuccessFeedbackSampleRate}
          lambdaFailureFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          lambdaSuccessFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          lambdaSuccessFeedbackSampleRate: ${schema.spec.sns.lambdaSuccessFeedbackSampleRate}
          sqsFailureFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          sqsSuccessFeedbackRoleARN: ${iamRole.status.ackResourceMetadata.arn}
          sqsSuccessFeedbackSampleRate: ${schema.spec.sns.sqsSuccessFeedbackSampleRate}

Create an instance out of it
Watch the logs, nothing happends
Restart the sns-controller and the topic will be created

Versions:

kro version: 0.2.3
Kubernetes Version (kubectl version): 1.31

Involved Controllers:

Controller URLs and Versions (if applicable): sns

Which option describes the most your issue?

No response

[michaelhtm]

cc: @denzhel