Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Can't Delete EKS Cluster #382

Closed
rawkode opened this issue Apr 18, 2020 · 4 comments
Closed

Can't Delete EKS Cluster #382

rawkode opened this issue Apr 18, 2020 · 4 comments
Assignees
Milestone

Comments

@rawkode
Copy link

rawkode commented Apr 18, 2020

I spun up an EKS cluster using pulumi/eks, which worked great. Unfortunately, running a destroy afterwards leads to a failure. It seems it is trying to delete the security groups first, but there's some dependant resource.

Resources:
    - 114 to delete
Destroying (production):
 -  aws:ec2:SecurityGroup corp-production-eks-nodeSecurityGroup deleting 
@ destroying...............................................................
 -  aws:ec2:SecurityGroup corp-production-eks-nodeSecurityGroup deleting error: deleting urn:pulumi:production::infrastructure::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::corp-production-eks-nodeSecurityGroup: Error deleting security group: DependencyViolation: resource sg-0a35f8ecc789c705e has a dependent object
 -  aws:ec2:SecurityGroup corp-production-eks-nodeSecurityGroup **deleting failed** error: deleting urn:pulumi:production::infrastructure::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::corp-production-eks-nodeSecurityGroup: Error deleting security group: DependencyViolation: resource sg-0a35f8ecc789c705e has a dependent object
    pulumi:pulumi:Stack infrastructure-production  error: update failed
    pulumi:pulumi:Stack infrastructure-production **failed** 1 error
 
Diagnostics:
  aws:ec2:SecurityGroup (corp-production-eks-nodeSecurityGroup):
    error: deleting urn:pulumi:production::infrastructure::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::corp-production-eks-nodeSecurityGroup: Error deleting security group: DependencyViolation: resource sg-0a35f8ecc789c705e has a dependent object
    	status code: 400, request id: a7a3fe8b-0368-4008-b66d-ac5bd5dc816b
 
  pulumi:pulumi:Stack (infrastructure-production):
    error: update failed
@rawkode
Copy link
Author

rawkode commented Apr 18, 2020

I ran this again with --refresh and it deleted.

@metral
Copy link
Contributor

metral commented Apr 20, 2020

Thanks for the opening the issue.

Security groups can hang on deletion for a couple of reasons:

  • There's some network resources created in AWS (e.g. like a Service LB) that wasn't deleted first before the nodes were, or was not cleaned up in time before the sg was attempted to be deleted.
  • ENIs have been known to stick around either longer than usual, or don't get removed at all.
  • Or it can also be a random flake per eventual consistency not fully propagating in time.

I ran this again with --refresh and it deleted.

Glad to hear you were able to delete this. Unless you changed something outside of the pulumi program (e.g. using the AWS console) a re-run of the deletion should've worked without the need to specify --refresh. I suspect you may have just hit the 400 due to eventual consistency.

  • What does your program look like, and were there any changes?
  • Are you still hitting this issue in other programs?

@metral
Copy link
Contributor

metral commented May 5, 2020

@rawkode is this issue okay to close?

@rawkode rawkode closed this as completed May 5, 2020
@rawkode
Copy link
Author

rawkode commented May 5, 2020

The problem was ENIs, I deleted them manually.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants