DocumentDB deletion failure due to protection should be ACK.Recoverable instead of ACK.Terminal

[mmb]

Is your feature request related to a problem?

When trying to delete a DocumentDB with deletion protection turned on it fails and goes to ACK.Terminal.

Describe the solution you'd like

I'd like it to retry so when I turn off deletion protection the delete retries. This error seems recoverable, similar to deletion being blocked due to the resource being in use by another resource.

Describe alternatives you've considered

I couldn't find an easy way to get it to retry the deletion after it the ACK.Terminal status condition had been set.

[github-actions]

Hello @mmb 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[mmb]

If this changed is desired I'd be happy to send a PR.

[rushmash91]

Hey @mmb, can you share the manifest for the behavior you are encountering. You can also create a PR and we can continue the discussing there 🙂

[mmb]

I found some more detail on why this happens to clusters with deletionProtection: true. When you try to delete a DBCluster with this enabled the error returned by AWS is InvalidParameterCombination which gets a terminal status from https://github.com/aws-controllers-k8s/documentdb-controller/blob/18fab2bc7adc190c88f8cfc81051eeca491faae3/generator.yaml#L85.

Is the expectation that you would then update the DBCluster to set deleteProtection to false and it would finish deleting? I didn't try that but if that works there might be nothing to do here.

[rushmash91]

Hi @mmb, Thanks for reporting this issue! After replicating this, I can confirm that the current behavior where deletion protection errors result in ACK.Terminal status is working as intended based on ACK's error handling.

• Terminal conditions are for errors that require user intervention to modify the manifest before the operation can succeed
• Recoverable conditions are for transient errors that may resolve themselves over time without requiring manifest changes (e.g., rate limiting, temporary service unavailability)

When a DocumentDB cluster has deletionProtection: true, attempting to delete it will always fail with InvalidDBClusterStateFault until you explicitly:

1. Update your manifest to set deletionProtection: false
2. Apply the updated manifest

Simply retrying the deletion without manifest changes will never succeed, which is exactly what Terminal conditions are designed to handle.

The current reconciliation logic assumes deletion can always proceed immediately when a DeletionTimestamp is present. When a resource has a DeletionTimestamp set (from kubectl delete), the ACK runtime immediately moves to deletion logic without processing spec updates first. Even if the resource is changes deletionProtection: false in the spec, it will go directly to deletion, which fails because AWS resource still has protection enabled.

ACK Native Deletion Protection

This allows you to delete the Kubernetes resource while retaining the AWS resource, effectively providing deletion protection for any ACK resource without the terminal error. If you later want to manage the retained AWS resource again, you can adopt it back into Kubernetes using the adoption feature.

Docs for Adoption & ReadOnly: https://aws-controllers-k8s.github.io/community/docs/user-docs/features/

apiVersion: documentdb.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
  name: my-cluster
  annotations:
    services.k8s.aws/deletion-policy: retain  # Keeps AWS resource when K8s resource is deleted
spec:
  dbClusterIdentifier: my-cluster
  # ... other specs

Please feel free to reopen this issue if you have any further question!
/close

[ack-prow]

@rushmash91: Closing this issue.

In response to this:

Hi @mmb, Thanks for reporting this issue! After replicating this, I can confirm that the current behavior where deletion protection errors result in ACK.Terminal status is working as intended based on ACK's error handling.

• Terminal conditions are for errors that require user intervention to modify the manifest before the operation can succeed
• Recoverable conditions are for transient errors that may resolve themselves over time without requiring manifest changes (e.g., rate limiting, temporary service unavailability)

When a DocumentDB cluster has deletionProtection: true, attempting to delete it will always fail with InvalidDBClusterStateFault until you explicitly:

1. Update your manifest to set deletionProtection: false
2. Apply the updated manifest

Simply retrying the deletion without manifest changes will never succeed, which is exactly what Terminal conditions are designed to handle.

ACK Native Deletion Protection

This allows you to delete the Kubernetes resource while retaining the AWS resource, effectively providing deletion protection for any ACK resource without the terminal error. If you later want to manage the retained AWS resource again, you can adopt it back into Kubernetes using the adoption feature.

Docs for Adoption & ReadOnly: https://aws-controllers-k8s.github.io/community/docs/user-docs/features/

apiVersion: documentdb.services.k8s.aws/v1alpha1
kind: DBCluster
metadata:
 name: my-cluster
 annotations:
   services.k8s.aws/deletion-policy: retain  # Keeps AWS resource when K8s resource is deleted
spec:
 dbClusterIdentifier: my-cluster
 engine: docdb
 # ... other specs

Please feel free to reopen this issue if you have any further question!
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.