Stack up date and rolback failed due to c5.2xLarge server not available on AWS.

[StudioDeussen]

@robinkwilson wilson Hi Robin, we tried to up-date the stack of our cloud enterprise instance to a larger server c5.2xLarge for an event we have been running. We got the following error message from AWS stating that the server size in not available in our Zone Irland (eu-west-1). This caused the up-date to fail. The stack went into rollback, but the rollback failed as well. Now the stack is stuck in a "UPDATE_ROLLBACK_FAILED". In the cloudformation section the up date uption is not available any more. Only the other stack actions (f.e. Continue update rollback) are there, but I don`t know, how or if I should use those. So we can't change the server size down again.

Intrestingly we can still reach our website: www.xrcreatorspaces.com, maybe its running on a back-up instance, but I dont know for sure. Its really crucial to find out what is happening here. The same error, server capacity not available in our region, happen to us two weeks ago as well. Back then we deleted the stack and created a new one with a smaller server size. Since this happend second time it seems to be a general problem. Its crucial for us to find out why this is happening, since its a real risk if we dont know if the up-grade in server size crashes our system.

Note: this is independt of the other discussion we had of up-dating from personal to an enterprise template by swaping the templates. Thanks for answering that! This current issue happens on a template we directly hosted as an enterprise version.

Any idea what we can do here?

Greets! Tim

Here is the orignal AWS error message:

2020-11-13 20:50:34 UTC+0100	AppASG	UPDATE_FAILED	Group did not stabilize. {current/minSize/maxSize} group size = {2/4/64}. Failed Scaling Activity: We currently do not have sufficient c5.2xlarge capacity in the Availability Zone you requested (eu-west-1a). Our system will be working on provisioning additional capacity. You can currently get c5.2xlarge capacity by not specifying an Availability Zone in your request or choosing eu-west-1b, eu-west-1c. Launching EC2 instance failed.

2020-11-13 21:22:04 UTC+0100	XRcreatorspaces3	UPDATE_ROLLBACK_FAILED	The following resource(s) failed to update: [StreamASG].

2020-11-13 21:22:02 UTC+0100 | StreamASG | UPDATE_FAILED | Received 0 SUCCESS signal(s) out of 2. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement

[robinkwilson]

Ah unfortunately this is an AWS error. They don't have enough c5.2xlarge machines available in your region, you'll need to update to a different instance size.

I'd be surprised if your site experienced any downtime during this action > error > rollback to the last working state.

The rollback will bring your instance state back to before the update. So if you were running t3.medium instances, it will rollback to t3.medium instances. Everything should be working properly, only you'll need to pick fewer machines or use different instance sizes.

I'm curious how many machines are you running?

[StudioDeussen]

Thanks for your suppoer @robinkwilson ! The rollback worked. We where only running 4 servers. (2 for the instance and 2 for the streaming). This was actually the second time that this happend. Its some how strange that AWS does not have enough large servers capacity in Europe. Do you know about the new Gaia-X initiative of the EU? https://en.wikipedia.org/wiki/GAIA-X It would be really cool for many reasons if Mozilla Hubs could be part of that.

The roolback error was solved by following your advice. AWS had also written to me. Maybe that is helpfull for other similar cases, so I will share it here:

Reviewing the case details, I see that a recent update to your CFN stack "XRcreatorspaces3" failed to update autoscaling resource "AppASG" due to lack of c5.2xlarge capacity in the specified AZ. The rollback process however cleaned up this resource, but this time failed at resource "StreamASG".

This StreamASG resource during the rollback process, had initiated a rolling update, setting the MinSize and DesiredCapacity to 4 temporarily. It spun up 2 new instances i-0b55c05accf3cea77 and i-053fb2ac0897ebe1f with the old Launch Config XRcreatorspaces3-StreamLaunchConfiguration-14AJUY2GOFK59 but appears to have failed to signal back to CFN within the specified timeout period causing the rolling update to be stuck midway.

As the main stack "XRcreatorspaces3" is currently now in Update Rollback Failed state, it wouldn't let you do further updates to it until we fix the failed resource StreamASG. To do this:

1. Perform a continue update rollback skipping the resource StreamASG - This process will skip the ASG and bring the mainstack to Update Complete state.

[+] https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-update-rollback-failed/

2. The next step is to manually bring StreamASG in sync with the stack. To do this, you would have to delete the old instances i-09eeca1863c9db3a9 and i-0a02537d5bae6d671 and bring the MinSize and DesiredCapacity back to 2 (from ASG console).

I also notice that the 5 child stacks managed by "XRcreatorspaces3" are all stack in UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS state which is unstable. While you do the above recommended steps, I'll be reaching out to our service team to flip the status of these child stacks to Update Complete so all the resources of "XRcreatorspaces3" are in sync and in workable state.

Once done, your stack will be ready for further updates. I'll keep you posted once the child stacks are fixed from our end. Meanwhile, if you have any further questions, please feel free to revert back to me and I'll be happy to address them for you.

1. You can terminate those instances from EC2 console Instance view.

2. To edit the MinSize and DesiredCapacity, navigate to ASG view > Select StreamASG > under Details section > Edit > Set MinSize and DesiredCapacity to 2.

[bemyevent]

Hello
I have the same problem for irland zone .
Here is the answer from aws support . Sorry it is long answer , i need to test this solution

We recommend that you launch the number of instances that you need in the placement group in a single launch request and that you use the same instance type for all instances in the placement group. If you try to add more instances to the placement group later, or if you try to launch more than one instance type in the placement group, you increase your chances of getting an insufficient capacity error.”

When a placement group is created, it will allocate resources physically close together in an availability zone. Then every time you try and launch a new instance in the same group, it will try to allocate resources in the same place. This situation could lead to “Insufficient Capacity Errors” in some situations, which is why we recommend to always launch all instances involved during the same request when possible.

In such situations, you can stop and start all of the instances in the placement group, and try the launch again. Starting the instances may migrate them to hardware that has capacity for all of the requested instances.

In order to solve this issue by updating stack, I tried to launched the stack at my end and I could reproduce your issue. From my testing, I was able to change EC2 instance types by following below steps. Please refer to the following steps for more information :

    - Please notice there will be “downtime” for a while if you would like to change the EC2 instance type during the process.
    - Since this stack is maintained by third-party. As AWS Support, we are not aware how mechanism operates in this system. Therefore, please always backup your data before updating the stack !

    1. There is a parameter named “StackOffline” in the stack template. You have to update the stack and select “Offline - Temporarily shut off servers”. According to the parameter description :
    ===================================================
    To save costs, you can switch your stack from being Online to Offline when it is not needed. This will shut down the servers while keeping all of your data. Once your stack is switched back to Online, it will come back up just how you left it.
    ===================================================

    Then, the DesiredCapacity of the ASG will be set to 0. All of your EC2 instance running in the ASG will be terminated.

    2. After the stack become “UPDATE_COMPLETE” state, you can update the stack again and select “StackOffline”: “Online” and change the EC2 instance type ( for example : “c5.2xlarge”). Then, you will see new EC2 instances be launched in the ASG.

[bemyevent]

After quick test , it works fine . The problem is solved .