EKS: Delete of Pod Identity Association is eventually consistent #2684
Description
Describe the bug
As mentioned in https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-considerations, PIAs are eventually consistent. In my experience, I notice it especially on the delete operation which can take a second or more to be effective.
This translate into an issue that when a PIA is deleted then immediately recreated (for example with ArgoCD force/purge recreation, or even manually in a script), the PIA will be marked as already existing with a ResourceInUseException error. (It also appears that the deletion was cancelled somehow because even though the Kubernetes resource was deleted at some point, the AWS resource is still there).
This would be fine, if the resource could be adopted-or-create. But the association ID is required for adoption to find the resource, which is not available on the resource by default since we re-create it with just the fields in the spec.
The adopt-or-create process attempts to read the source via https://github.com/aws-controllers-k8s/eks-controller/blob/main/pkg/resource/pod_identity_association/sdk.go#L56 and checks r.ko.Status.AssociationID == nil || r.ko.Spec.ClusterName == nil. Since the status will be empty on a new resource, this will always return NotFound and the source will not be adopted.
A potential workaround would be to actually list the PIAs: https://docs.aws.amazon.com/eks/latest/APIReference/API_ListPodIdentityAssociations.html
This uses cluster, namespace and service account which could be considered a private and unique key as well. This would allow the adoption process to go through if this is correct.
Steps to reproduce
Expected outcome
PIA recreation should always works
Environment
- Kubernetes version: N/A
- Using EKS (yes/no), if so version?
- EKS