Error: deleting Security Group because it has a dependent object

[def-]

Seen in https://buildkite.com/materialize/nightly/builds/14568#019b3403-f9b7-44ee-bbf5-b24a931dfdaf:

module.eks.module.eks.aws_security_group.node[0]: Still destroying... [id=sg-02cc5b629ad458c86, 15m00s elapsed]
╷
│ Warning: Helm uninstall returned an information message
│
│ These resources were kept due to the resource policy:
│ [CustomResourceDefinition] certificaterequests.cert-manager.io
│ [CustomResourceDefinition] certificates.cert-manager.io
│ [CustomResourceDefinition] challenges.acme.cert-manager.io
│ [CustomResourceDefinition] clusterissuers.cert-manager.io
│ [CustomResourceDefinition] issuers.cert-manager.io
│ [CustomResourceDefinition] orders.acme.cert-manager.io
│
╵
╷
│ Error: deleting Security Group (sg-02cc5b629ad458c86): operation error EC2: DeleteSecurityGroup, https response error StatusCode: 400, RequestID: 62dbc06f-0107-4c2b-aff4-e0ba3fa143cf, api error DependencyViolation: resource sg-02cc5b629ad458c86 has a dependent object

[jshiwamV]

Is it intermittent or has it been breaking consistently?
My first guess is, that this issue is similar to #106 (comment)

[def-]

Yes, that looks very similar. I'm manually cleaning up security groups, VPCs etc.
Every other run had other failures so far, but I think it's a flake.

[def-]

2nd try, same error:

module.eks.module.eks.aws_security_group.node[0]: Still destroying... [id=sg-092a1afece7a833f8, 15m00s elapsed]
╷
│ Warning: Helm uninstall returned an information message
│
│ These resources were kept due to the resource policy:
│ [CustomResourceDefinition] certificaterequests.cert-manager.io
│ [CustomResourceDefinition] certificates.cert-manager.io
│ [CustomResourceDefinition] challenges.acme.cert-manager.io
│ [CustomResourceDefinition] clusterissuers.cert-manager.io
│ [CustomResourceDefinition] issuers.cert-manager.io
│ [CustomResourceDefinition] orders.acme.cert-manager.io
│
╵
╷
│ Error: deleting Security Group (sg-092a1afece7a833f8): operation error EC2: DeleteSecurityGroup, https response error StatusCode: 400, RequestID: 8396b1be-454e-4f04-9618-54002d6cafab, api error DependencyViolation: resource sg-092a1afece7a833f8 has a dependent object

In https://buildkite.com/materialize/nightly/builds/14568#019b35b9-b0bf-4cc1-8814-ab1290d8cab9

[jshiwamV]

I will try to repro this locally, I can't access that link.

[def-]

https://gist.github.com/def-/9fb05769264a0847ee7e350fb5405fbb

[jshiwamV]

I ran the tests locally and didn't face this issue, but I had infra for simple example which i had been using for past 2 days. I got this issue there. The eni which was using the eks node security group looks as follows

|  eni-0d6df8b6561945150 |  aws-K8S-i-0a7ea53956cb895b4  |  available |  None |

From its description it looks like the one which VPC CNI manages, only reasonable explaination I have is the VPC CNI plugin couldn't cleanup the eni because the aws-node daemonset was deleted before it could remove the eni.

[jshiwamV]

for the test suites, can you confirm if it is CNI managed ENI that has dependency over the EKS Node shared security group? From the error log I can see that security group is from EKS module, so its most likely due to the reason I mentioned.

[def-]

@bobbyiliev probably knows best, but I can try to figure it out.

[jshiwamV]

ack, I will try to look on my side possible ways to fix this issue.

[bobbyiliev]

I think that the issue here is that we are leaking endpoints so the sg never gets deleted nor does the VPC itself.