Stand alone Subscription Filter CRD for ACK Cloudwatch logs controller

[vladkiner]

Is your feature request related to a problem?
We are having similar issue with Log Groups created by RDS for audit, error and slowquery logs as described issues/2174. Log group doesn't get imported and we suspecting due to ownership issue. We suspecting RDS/RDS Controller might own it and prevent Cloudwatchlogs Controller to adopt the resource. It would be beneficial to have Stand alone Subscription Filter CRD for ACK Cloudwatch logs controller

Describe the solution you'd like
A standalone crd for subscription filter in ACK cloudwatch logs controller

Describe alternatives you've considered
So far we have zero alternatives

[github-actions]

Hello @vladkiner 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[knottnt]

@vladkiner could you share the manifests and steps to reproduce the issue?

[vladkiner]

apiVersion: cloudwatchlogs.services.k8s.aws/v1alpha1
kind: LogGroup
metadata:
  name: aws-rds-cluster-hello-world-error
  namespace: {{namespace}}
spec:
  name: "/aws/rds/cluster/hello-world/error"
  retentionDays: 7
  subscriptionFilters:
    - destinationARN: "arn:aws:firehose:us-east-1:{{aws_account}}:deliverystream/{{delivery_stream_name}}"
      distribution: "ByLogStream"
      filterName: "filter_name"
      filterPattern: ""
      roleARN: "arn:aws:iam::{{aws_account}}:role/{{role_name}}"
---
apiVersion: services.k8s.aws/v1alpha1
kind: AdoptedResource
metadata:
  name: adopt-existing-logroup
  namespace: {{namespace}}
spec:  
  aws:
    nameOrID: "/aws/rds/cluster/hello-world/error"
  kubernetes:
    group: cloudwatchlogs.services.k8s.aws
    kind: LogGroup
    metadata:
      name: aws-rds-cluster-hello-world-error
      namespace: {{namespace}}

When I deploy it, I am getting this message when describing the log group

message: Resource already exists
    reason: This resource already exists but is not managed by ACK. To bring the resource
      under ACK management, you should explicitly adopt the resource by creating a
      services.k8s.aws/AdoptedResource

[knottnt]

@vladkiner The error you're seeing may be due to the LogGroup resource you're creating seeing that the AWS resource isn't yet managed by ACK. AdoptedResource attempts to create a new ACK resource so the fact that both AdoptedResource and LogGroup are being created could cause a conflict.

If you'd like to use the LogGroup manifest as the source of truth when adopting the AWS resource you can use the newer adoption-policy annotations with adopt-or-create policy.

apiVersion: cloudwatchlogs.services.k8s.aws/v1alpha1
kind: LogGroup
metadata:
  name: aws-rds-cluster-hello-world-error
  namespace: {{namespace}}
  annotations:
    service.k8s.aws/adoption-policy: adopt-or-create
spec:
  name: "/aws/rds/cluster/hello-world/error"
  retentionDays: 7
  subscriptionFilters:
    - destinationARN: "arn:aws:firehose:us-east-1:{{aws_account}}:deliverystream/{{delivery_stream_name}}"
      distribution: "ByLogStream"
      filterName: "filter_name"
      filterPattern: ""
      roleARN: "arn:aws:iam::{{aws_account}}:role/{{role_name}}"

[vladkiner]

Thank you for clarifying it. About adoption-policy-annotattions. I am a bit hesitant to use alpha feature in production. Is there a time frame when it will be transitioned to Beta?

[knottnt]

@vladkiner We don't have an official timeline, but are hoping to transition to Beta by the end of this year.

[vladkiner]

Got it, thank you for you help and provided information.

[knottnt]

Happy to help! I'll go ahead and close this issue. If you have any more problems or questions feel free to re-open this issue or open a new one.