AWS instance store, instance upgrades and recovery

[strf0x1]

We're using instance storage for my SO2 manager/search and search nodes, and am getting to point where we need to upgrade our instance sizes. AWS recommends to attach an EBS disk and rsync your data over. Am curious, has anyone done this? Curious if I'm thinking about this correctly...

• Stop ElasticSearch container
• Attach an EBS disk and copy over the /nsm/elasticsearch (/nsm for manager/search)
• Stop instance and upgrade instance size
• Bring server back up, copy data from EBS back to /nsm/elasticsearch
• Restart so-elasticsearch container.

Is that all there is to it? Most curious about when I bring the server back up, containers will try to start and probably fail. Don't see that there is a way to disable containers on startup, so was curious if this will cause problems because the backed-up data won't be there yet.

[TOoSmOotH]

If you are using the official AMI then you will need to move all of /nsm. I would recommend setting salt to disabled and reboot.

sudo systemctl disable salt-minion

Once you restart mount your new EBS volume that you are planning on copying data over to and you should be fine. If you have just mounted /nsm/elasticsearch on ephemeral then your above instructions should be fine. I would still disable salt and reboot so it doesn't fire it up during the copy.

After you are happy with your new instance enable salt-minion again and start the service.

[strf0x1]

Thank you! Really appreciate the help, you all are so awesome.

One issue I encountered on my test stack was that the fstab entry created by so-prepare-fs is defined like this (ephemeral mount):

/dev/nvme0n1p1 /nsm/elasticsearch xfs defaults 0 0

Turning down the node, upgrading and starting back produces a hang at Grub:

Dependency failed for /nsm/elasticsearch.

It's hard to tell if the ephemeral device name changed, but adding nofail to the fstab before stopping the instance seems to be the solution:

/dev/nvme0n1p1 /nsm/elasticsearch xfs defaults,nofail 0 0

Normally the solution would be to use UUID instead of /dev/nvme0n1p1, but I believe UUID will be changing with ephemeral storage and stop/starting an instance.

Might take a look at utilizing this script to autodiscover and format the new ephemeral disk:
https://gist.github.com/fideloper/40f7807920aa1198fa07b9e69dc82b56

Just FYI if anyone is running into this problem.

[jertel]

If you're using the official AMI then the ephemeral section of the AWS AMI documentation may help you with this particular question. Again, this is only applicable to official AMI users.