Bug in so-prepare-fs

[strf0x1]

Fairly sure this has been a bug for several releases, and have watched this occur with other people. In certain regions/instance types, ephemeral storage defaults to nvme0n1. So per the docs, the following syntax is used to customize the storage device.

so-prepare-fs /dev/nvme0n1 /nsm/elasticsearch

On first run, the error is:

/dev/nvme0n11: No such file or directory

But, on second run it works. Appreciate someone taking a look. This is consistent every single time we bootstrap a server in aws (dozens of deployments).

[jertel]

Please let us know which instance type and region will reproduce this behavior. Thanks.

[strf0x1]

i3.2xlarge, afraid I can't say what region it's in sorry. Device id of ephemeral storage is /dev/nvme0n1.

[jertel]

I am unable to reproduce this with that instance type.

The authenticity of host '107.21.190.59 (107.21.190.59)' can't be established.
ECDSA key fingerprint is SHA256:iIrwzE3bec0ZKgnNmXo3W0SD7XuFibh5J4Txf/Ca2rk.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '107.21.190.59' (ECDSA) to the list of known hosts.
''Preparing to run setup...
Getting started...
User cancelled setup.

-bash-4.2$ ls -altr /dev/nvme*
crw-------. 1 root root 245, 0 Nov  3 07:18 /dev/nvme0
brw-rw----. 1 root disk 259, 0 Nov  3 07:18 /dev/nvme0n1
-bash-4.2$ sudo so-prepare-fs /dev/nvme0n1 /nsm/elasticsearch
Partitioning device /dev/nvme0n1 to be mounted as /nsm/elasticsearch...
Discarding blocks...Done.
meta-data=/dev/nvme0n1p1         isize=512    agcount=4, agsize=115966720 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=463866880, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=226497, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
/dev/nvme0n1p1 on /nsm/elasticsearch type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
File system preparation successful.
-bash-4.2$ 

I understand you may be unable to divulge the region you're using for your production instances, but if you can reproduce it on any other region for the purposes of assisting our team with troubleshooting, that would be helpful.

In the meantime, I have added a short pause after the the so-prepare-fs tool finishes partitioning the device, before it moves on to the next steps. I believe this will likely resolve the problem, as I suspect it was caused by the new partition not yet reflecting on the filesystem. This will be available in the next release.

[strf0x1]

Thank you so much for your help. I will test with a small sleep() where you said and see if that fixes the issue. Would that be on line 23?

[strf0x1]

A small sleep of 2-3s appears to have fixed the problem. Thanks!