ACK RDS DBInstance Custom Resource disappears before the RDS is completely deleted on AWS RDS

[eTimS]

Hello

Describe the bug
When a DBInstance CR is deleted, ACK-RDS does not wait for the AWS RDS Database to be fully deleted before deleting the DBInstance CR from the etcd. This can lead to issues like, redeploying the DBInstance just after is was deleted, only to find out the controller notices the previous database is deleting and does nothing to create a new one on its own. (not tested : the controller will probably notice the DB is missing at the next reconcile step, but this is not practical when you just need to redeploy a database (and you automate the process))

Steps to reproduce
Use rds-chart:1.6.0
Create a DBInstance and then delete it.
Notice the DBInstance resource disappears directly while the AWS RDS Console will show that the database still exists and is deleting.
Create the same DBInstance again (while the DB is deleting on AWS) and notice that the new CR will say in its status that it is deleting. After the RDS DB is fully deleted in AWS, the controller doesn't update the new CR status, and does not create the database.

Expected outcome
When a DBInstance CR is deleted, wait for the "real" AWS RDS Database to be completely deleted before deleting the CR in kube. Also, keep the status of the DBInstance CR up to date during the deletion process (showing "deleting" or other appropriate values and update every 30 seconds for example).

Environment

• Kubernetes version
  1.31
• Using EKS (yes/no), if so version?
  eks.34
• AWS service targeted (S3, RDS, etc.)
  RDS

[github-actions]

Hello @eTimS 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[michaelhtm]

Hello @eTimS, Although I see the need for this feature, it introduces a behavioral change for existing users. We can discuss a potential feature flag solution for this change, ensuring ACK waits for a 404 error message before removing the finalizer from the custom resource.
cc: @a-hilaly @knottnt @rushmash91 @cheeseandcereal