EC2 volume run out of space, what should we do? Need documentation

[yalegria]

I am currently facing an issue with my Hubs Cloud instance. It displays "Hubs Cloud is currently offline. Please check later".

When I tried to ssh to my EC2 instance, and input my verification code I got the error: "No space left on device while writing config"

Checking the volume attached to my instance I realized it is a 8GB volume, so I need to increase it.

I haven't seen any documentation about increasing the volume size that applies specifically to Hubs Cloud. Some questions that this might rise:

• Should this be done at the template on CloudFormation?
• Do we need to shut down the instance and then modify the volume?
• Can we modify the volume while the instance is running?
• Can the template be modified to gives a choice to choose the volume size?

[yalegria]

I have performed the following:

• create a snapshot of the volume.
• monitored snapshot has finished.
• modify volume size.
• monitored modification was successful from volumes dashboard as well as the EC2 instance.

Tried to ssh to instance, still getting "no space left on device while writing config".

[brianpeiris]

Some diagnostic things that might help figure this out:

• How long has that particular instance been alive?
• How much did you increase the volume size to?

Suggestions for things to try:

• Just to check the obvious, I would ensure that it's not the ssh-client machine that's the issue and try ssh-ing from a different machine if possible.
• The EC2 instance should not have any persistent data on it, so it should be fine to terminate that instance and let the Auto Scaling Group automatically spin up a fresh instance.

[yalegria]

Some diagnostic things that might help figure this out:

* How long has that particular instance been alive?

This particular instance has been running for over a year with no problems.

* How much did you increase the volume size to?

I increased the volume to 32GB.

Suggestions for things to try:

* Just to check the obvious, I would ensure that it's not the ssh-client machine that's the issue and try ssh-ing from a different machine if possible.

* The EC2 instance should not have any persistent data on it, so it should be fine to terminate that instance and let the Auto Scaling Group automatically spin up a fresh instance.

[yalegria]

* Just to check the obvious, I would ensure that it's not the ssh-client machine that's the issue and try ssh-ing from a different machine if possible.

Tried to ssh from another machine, no luck.

[yalegria]

* The EC2 instance should not have any persistent data on it, so it should be fine to terminate that instance and let the Auto Scaling Group automatically spin up a fresh instance.

I tried this and my instance is back online!. Thanks @brianpeiris. I'm starting to know better the architecture; terminating an EC2 instance will also delete the volume attached. So the increased volume ( to 32GB) was gone. The auto scaling featured spin up a new EC2 instance with a brand new volume.

Why the instance have run out of space?
@brianpeiris suggests that some artifacts might have grown over time although there should be some mechanisms to clean them up. This is still a mystery.