diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md index 89e4dac2e54..d68c723719b 100644 --- a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md @@ -1,6 +1,6 @@ --- title: How to configure and rebuild software RAID -excerpt: Find out how to verify the state of the software RAID of your server and rebuild it after a disk replacement +excerpt: Find out how to verify the state of your software RAID and rebuild it after a disk replacement updated: 2023-08-21 --- @@ -14,7 +14,7 @@ The default RAID level for OVHcloud server installations is RAID 1, which double ## Requirements -- A [dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration +- A [dedicated server](/links/bare-metal/bare-metal){.external} with a software RAID configuration - Administrative (sudo) access to the server via SSH ## Instructions @@ -382,4 +382,4 @@ mount /dev/md4 /home [Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) -Join our [community of users](/links/community). +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md new file mode 100644 index 00000000000..fdfdd7514c2 --- /dev/null +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md @@ -0,0 +1,604 @@ +--- +title: Managing and rebuilding software RAID on servers in BIOS boot mode +excerpt: Find out how to manage and rebuild software RAID after a disk replacement on your server in BIOS mode +updated: 2025-08-xx +--- + +## Objective + +Redundant Array of Independent Disks (RAID) is a technology that mitigates data loss on a server by replicating data across two or more disks. + +The default RAID level for OVHcloud server installations is RAID 1, which doubles the space taken up by your data, effectively halving the useable disk space. + +**This guide explains how to manage and rebuild software RAID after a disk replacement on your server in BIOS mode** + +Before we begin, please note that this guide focuses on Dedicated servers that use BIOS as the boot mode. If your server uses the UEFI mode (newer motherboards), refer to this guide [Managing and rebuilding software RAID on servers in UEFI boot mode](). + +## Requirements + +- A [Dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration +- Administrative (sudo) access to the server via SSH +- Understanding of RAID, partitions and GRUB + +## Instructions + +### Basic Information + +In a command line session, type the following code to determine the current RAID status: + +```sh +cat /proc/mdstat +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md3 : active raid1 nvme1n1p3[1] nvme0n1p3[0] + 497875968 blocks super 1.2 [2/2] [UU] + bitmap: 2/4 pages [8KB], 65536KB chunk + +md2 : active raid1 nvme1n1p2[1] nvme0n1p2[0] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: +``` + +This command shows us that we have two software RAID devices currently set up, with **md3** being the largest one. This array consists of two partitions, which are known as **nvme1n1p3** and **nvme0n1p3**. + +The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. + +If you have a server with SATA disks, you would get the following results: + +```sh +cat /proc/mdstat +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md3 : active raid1 sda3[0] sdb3[1] + 3904786432 blocks super 1.2 [2/2] [UU] + bitmap: 2/30 pages [8KB], 65536KB chunk + +md2 : active raid1 sda2[0] sdb2[1] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: +``` + +Although this command returns our RAID volumes, it doesn't tell us the size of the partitions themselves. We can find this information with the following command: + +```sh +sudo fdisk -l + +Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors +Disk model: HGST HUS724020AL +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +Disklabel type: gpt +Disk identifier: F92B6C5B-2518-4B2D-8FF9-A311DED5845F + +Device Start End Sectors Size Type +/dev/sdb1 2048 4095 2048 1M BIOS boot +/dev/sdb2 4096 1864177663 1864173568 888.9G Linux RAID +/dev/sdb3 1864177664 1865226239 1048576 512M Linux filesystem +/dev/sdb4 1865226240 3907024895 2041798656 973.6G Linux RAID + +Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors +Disk model: HGST HUS724020AL +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +Disklabel type: gpt +Disk identifier: 2E1DCCBA-8808-4D2B-BA33-9FEC3B96ADA8 + +Device Start End Sectors Size Type +/dev/sda1 2048 4095 2048 1M BIOS boot +/dev/sda2 4096 1864177663 1864173568 888.9G Linux RAID +/dev/sda3 1864177664 1865226239 1048576 512M Linux filesystem +/dev/sda4 1865226240 3907024895 2041798656 973.6G Linux RAID +/dev/sda5 3907025072 3907029134 4063 2M Linux filesystem + +Disk /dev/md4: 973.5 GiB, 1045265645568 bytes, 2041534464 sectors +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes + +Disk /dev/md2: 888.8 GiB, 954321600512 bytes, 1863909376 sectors +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +``` + +The `fdisk -l` command also allows you to identify your partition type. This is an important information when it comes to rebuilding your RAID in case of a disk failure. + +For **GPT** partitions, line 6 will display: `Disklabel type: gpt`. + +For **MBR** partitions, line 6 will display: `Disklabel type: dos`. + +Still going by the results of `fdisk -l`, we can see that `/dev/md2` consists of 888.8GB and `/dev/md4` contains 973.5GB. If we were to run the mount command we can also find out the layout of the disk. + +```sh +mount + +sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) +proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) +udev on /dev type devtmpfs (rw,nosuid,relatime,size=16315920k,nr_inodes=4078980,mode=755) +devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) +tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3266556k,mode=755) +/dev/md2 on / type ext4 (rw,relatime) +securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) +tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) +tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) +tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) +cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) +cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) +pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) +bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) +cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) +cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) +cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) +cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) +cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) +cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) +cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) +cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) +cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) +cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) +debugfs on /sys/kernel/debug type debugfs (rw,relatime) +hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) +mqueue on /dev/mqueue type mqueue (rw,relatime) +systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10340) +/dev/md4 on /home type ext3 (rw,relatime) +tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3266552k,mode=700,uid=1000,gid=1000) +``` + +Alternatively, the `lsblk` command offers a different view of the partitions: + +```sh +lsblk + +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +sda 8:0 0 1.8T 0 disk +├─sda1 8:1 0 1M 0 part +├─sda2 8:2 0 888.9G 0 part +│ └─md2 9:2 0 888.8G 0 raid1 / +├─sda3 8:3 0 512M 0 part [SWAP] +├─sda4 8:4 0 973.6G 0 part +│ └─md4 9:4 0 973.5G 0 raid1 /home +└─sda5 8:5 0 2M 0 part +sdb 8:16 0 1.8T 0 disk +├─sdb1 8:17 0 1M 0 part +├─sdb2 8:18 0 888.9G 0 part +│ └─md2 9:2 0 888.8G 0 raid1 / +├─sdb3 8:19 0 512M 0 part [SWAP] +└─sdb4 8:20 0 973.6G 0 part + └─md4 9:4 0 973.5G 0 raid1 /home +``` + +We take note of the devices, partitions and their mount points. From the above commands and results, we have: + +- Two RAID arrays: `/dev/md2` and `/dev/md4`. +- Partitions part of the RAID: `/` and `/home`. +- Partitions not part of the RAID: [SWAP]. + +### System verification + +To check whether a server runs on BIOS mode or BIOS mode, run the following command: + +```sh +[user@server_ip ~]# [ -d /sys/firmware/efi ] && echo BIOS || echo BIOS +``` + +### Simulating a disk failure + +Now that we have all the necessary information, we can simulate a disk failure and proceed with the tests. In this example, we will fail the main disk `sda`. + +The preferred way to do this is via the OVHcloud rescue mode environment. + +First reboot in rescue menu and log in with the credentials provided. + +To remove a disk from the RAID, the first step is to mark it as **Failed** and remove the partitions from their respective RAID arrays. + +```sh +cat /proc/mdstat + +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md2 : active raid1 sda2[1] sdb2[0] + 931954688 blocks super 1.2 [2/2] [UU] + bitmap: 2/7 pages [8KB], 65536KB chunk + +md4 : active raid1 sda4[0] sdb4[1] + 1020767232 blocks super 1.2 [2/2] [UU] + bitmap: 0/8 pages [0KB], 65536KB chunk + +unused devices: +``` +From the above output, sda consists of two partitions in RAID which are **sda2** and **sda4**. + +#### Removing the failed disk + +First we mark the partitions **sda2** and **sda4** as failed. + +```sh +sudo mdadm --manage /dev/md2 --fail /dev/sda2 + +# mdadm: set /dev/sda2 faulty in /dev/md2 +``` + +```sh +sudo mdadm --manage /dev/md4 --fail /dev/sda4 + +# mdadm: set /dev/sda4 faulty in /dev/md4 +``` + +We have now simulated a failure of the RAID, when we run the `cat /proc/mdstat` command, we have the following output + +```sh +cat /proc/mdstat + +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md2 : active raid1 sda2[1](F) sdb2[0] + 931954688 blocks super 1.2 [2/2] [_U] + bitmap: 2/7 pages [8KB], 65536KB chunk + +md4 : active raid1 sda4[0](F) sdb4[1] + 1020767232 blocks super 1.2 [2/2] [_U] + bitmap: 0/8 pages [0KB], 65536KB chunk +unused devices: +``` + +As we can see above, the [F] next to the partitions indicates that the disk has failed or is faulty. + +Next, we remove these partitions from the RAID arrays to completely remove the disk from RAID. + +```sh +sudo mdadm --manage /dev/md2 --remove /dev/sda2 + +# mdadm: hot removed /dev/sda2 from /dev/md2 +``` + +```sh +sudo mdadm --manage /dev/md3 --remove /dev/sda3 + +# mdadm: hot removed /dev/nvme0n1p3 from /dev/md3 +``` + +To make sure that we get a disk that is similar to an empty disk, we use the following command. Replace **sda** with your own values: + +```sh +sudo shred -s10M -n1 /dev/sda +``` + +The disk now appears as a new one: + +```sh +lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +sda 8:0 0 1.8T 0 disk +sdb 8:16 0 1.8T 0 disk +├─sdb1 8:17 0 1M 0 part +├─sdb2 8:18 0 888.9G 0 part +│ └─md2 9:2 0 888.8G 0 raid1 / +├─sdb3 8:19 0 512M 0 part [SWAP] +└─sdb4 8:20 0 973.6G 0 part + └─md4 9:4 0 973.5G 0 raid1 /home +``` + +### Rebuilding the RAID + +The following steps document the RAID rebuild in rescue mode. + +> [!primary] +> This process might be different depending on the operating system you have installed on your server. We recommend that you consult the official documentation of your operating system to have access to the proper commands. +> + +> [!warning] +> +> For most servers in software RAID, after a disk replacement, the server is able to reboot in normal mode (on the healthy disk) and the rebuild can be done in normal mode. However, if the server is not able to reboot in normal mode after a disk replacement, it will be rebooted in rescue mode to proceed with the RAID rebuild. +> + +#### Rebuilding the RAID after the main disk is replaced + +Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, sdb ) to the new one (sda). + +> [!tabs] +> **For GPT partitions** +>> +>> ```sh +>> sudo sgdisk -R /dev/sdX /dev/sdX +>> ``` +>> +>> The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` +>> +>> Example: +>> +>> ```sh +>> sudo sgdisk -R /dev/sda /dev/sdb +>> ``` +>> +>> Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: +>> +>> ```sh +>> sudo sgdisk -G /dev/sda +>> ``` +>> +>> If you receive a message like this: +>> +>> ```console +>> Warning: The kernel is still using the old partition table. +>> The new table will be used at the next reboot or after you +>> run partprobe(8) or kpartx(8) +>> The operation has completed successfully. +>> ``` +>> +>> You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. +>> +> **For MBR partitions** +>> +>> ```sh +>> sudo sfdisk -d /dev/sdb | sfdisk /dev/sda +>> ``` +>> +>> The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk` +>> + +We can now rebuild the RAID array. The following code snippet shows how we can add the new partitions (sda2 and sda4) back in the RAID array. + + +```sh +sudo mdadm --add /dev/md2 /dev/sda2 + +# mdadm: added /dev/sda2 + +sudo mdadm --add /dev/md4 /dev/sda4 + +# mdadm: re-added /dev/sda4 + +cat /proc/mdstat + +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md2 : active raid1 sda2[0] sdb2[1] + 931954688 blocks super 1.2 [2/2] [UU] + bitmap: 4/4 pages [16KB], 65536KB chunk + +md4 : active raid1 sda4[0](F) sdb4[1] + 1020767232 blocks super 1.2 [2/1] [UU] + [============>........] recovery = 64.8% (822969856/1020767232) finish=7.2min speed=401664K/sec + bitmap: 0/8 pages [0KB], 65536KB chunk +unused devices: +``` + +We can verify the RAID details with the following command: + +```sh +mdadm --detail /dev/md4 + +/dev/md4: + Version : 1.2 + Creation Time : Tue Jan 24 15:35:02 2023 + Raid Level : raid1 + Array Size : 1020767232 (973.48 GiB 1045.27 GB) + Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) + Raid Devices : 2 + Total Devices : 2 + Persistence : Superblock is persistent + + Intent Bitmap : Internal + + Update Time : Tue Jan 24 17:02:55 2023 + State : clean + Active Devices : 2 + Working Devices : 2 + Failed Devices : 0 + Spare Devices : 0 + + Rebuild Status : 21% complete + + UUID : 7f39d062:9f16a016:a4d2adc2:26fd5302 + Events : 0.95 + + Number Major Minor RaidDevice State + 0 8 2 0 spare rebuilding /dev/sda4 + 1 8 18 1 active sync /dev/sdb4 +``` + +The RAID has now been rebuilt. + +#### Adding the label to the SWAP partition (if applicable) + +First, we mount the partition containing our files on `/mnt`. In our example, that partition is `md4`. + +```sh +mount /dev/md4 /mnt +``` + +We add the label back to our swap partition: + +```sh +mkswap /dev/sda4 -L swap-sda4 +mkswap: /dev/sda4: warning: wiping old swap signature. +Setting up swapspace version 1, size = 512 MiB (536866816 bytes) +LABEL=swap-nvme0n1p4, UUID=b3c9e03a-52f5-4683-81b6-cc10091fcd +``` + +Next, we mount the following directories to make sure any manipulation we make in the chroot environment works properly: + +```sh +mount --types proc /proc /mnt/proc +mount --rbind /sys /mnt/sys +mount --make-rslave /mnt/sys +mount --rbind /dev /mnt/dev +mount --make-rslave /mnt/dev +mount --bind /run /mnt/run +mount --make-slave /mnt/run +``` + +Next, we access the `chroot` environment again: + +```sh +chroot /mnt +``` + +We retrieve the UUIDs of both swap partitions: + +```sh +blkid /dev/sda4 +blkid /dev/sdb4 +``` + +Example: + +```sh +blkid /dev/sda4 +/dev/sda4: LABEL="swap-sda4" UUID="b3c9e03a-52f5-4683-81b6-cc10091fcd15" TYPE="swap" PARTLABEL="logical" PARTUUID="cb6022d9-7631-449c-956a-116bcd0e2119" +``` + +```sh +blkid /dev/sdb4 +/dev/sdb4: LABEL="swap-sdb4" UUID="d6af33cf-fc15-4060-a43c-cb3b5537f58a" TYPE="swap" PARTLABEL="logical" PARTUUID="d037c35f-2ddb-40d5-be33-31cc496fe54b" +``` + +Next, we replace the old UUID of the swap partition (**nvme0n1p4)** with the new one in `/etc/fstab`: + +```sh +nano etc/fstab +``` + +Example: + +```sh +UUID=6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 / ext4 defaults 0 1 +UUID=f925a033-0087-40ec-817e-44efab0351ac /home ext4 defaults 0 0 +UUID=b7b5dd38-9b51-4282-8f2d-26c65e8d58ec swap swap defaults 0 0 +UUID=d6af33cf-fc15-4060-a43c-cb3b5537f58a swap swap defaults 0 0 +``` + +Make sure you replace the proper UUID. In our example above, the UUID to replace is `b7b5dd38-9b51-4282-8f2d-26c65e8d58ec` with the new one `b3c9e03a-52f5-4683-81b6-cc10091fcd15` since the other one belongs to **sdb4**. + +Next, we make sure everything is properly mounted: + +```sh + mount -av +/ : ignored +mount: (hint) your fstab has been modified, but systemd still uses + the old version; use 'systemctl daemon-reload' to reload. +/boot : successfully mounted +/boot/efi : successfully mounted +swap : ignored +swap : ignored +``` + +We enable the swap partition: + +```sh +swapon -av + +swapon: /dev/sda4: found signature [pagesize=4096, signature=swap] +swapon: /dev/sda4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/sda4 +swapon: /dev/sdb4: found signature [pagesize=4096, signature=swap] +swapon: /dev/sdb4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/sdb4 +``` + +We exit the Chroot environment with `Exit` and unmount all the disks: + +```sh +umount -a +``` + +We have now successfully completed the RAID rebuild on the server and we can now reboot the server in normal mode. + +/// details | **Rebuilding the RAID after the secondary disk is replaced** + +The following steps are done in normal mode since it is the secondary disk that is was replaced. In our example, our secondary disk is named **sdb**. + +Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, sda) to the new one (sdb). + +> [!tabs] +> **For GPT partitions** +>> +>> ```sh +>> sudo sgdisk -R /dev/sdX /dev/sdX +>> ``` +>> +>> The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk`. +>> +>> Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: +>> +>> ```sh +>> sudo sgdisk -G /dev/sdX +>> ``` +>> +>> If you receive a message like this: +>> +>> ```console +>> Warning: The kernel is still using the old partition table. +>> The new table will be used at the next reboot or after you +>> run partprobe(8) or kpartx(8) +>> The operation has completed successfully. +>> ``` +>> +>> You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. +>> +> **For MBR partitions** +>> +>> ```sh +>> sudo sfdisk -d /dev/sdX | sfdisk /dev/sdX +>> ``` +>> +>> The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk`. +>> + +Next, we add the partitions to the RAID: + +```sh +sudo mdadm --add /dev/md2 /dev/sdb2 + +# mdadm: added /dev/sdb2 + +sudo mdadm --add /dev/md4 /dev/sdb4 + +# mdadm: re-added /dev/sdb4 +``` + +Use the following command to follow the RAID rebuild `cat /proc/mdstat`. + +Lastly, we add a label and mount the [SWAP] partition (if applicable): + +```sh +sudo mkswap /dev/sdb4 -L swap-sdb4 +``` + +We retrieve the UUIDs of both swap partitions: + +```sh +sudo blkid /dev/sda4 +sudo blkid /dev/sdb4 +``` + +We replace the old UUID of the swap partition (**sdb4)** with the new one in `/etc/fstab`: + +```sh +sudo nano etc/fstab +``` + +Make sure you replace the correct UUID. Reload the system with the following command: + +```sh +sudo systemctl daemon-reload +``` + +Next, run the following command to enable it: + +```sh +sudo swapon -av +``` + +We have now successfully completed the RAID rebuild. + +## Go Further + +[Hot Swap - Software RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_soft) + +[OVHcloud API and Storage](/pages/bare_metal_cloud/dedicated_servers/partitioning_ovh) + +[Managing hardware RAID](/pages/bare_metal_cloud/dedicated_servers/raid_hard) + +[Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md new file mode 100644 index 00000000000..7d55cfeee5f --- /dev/null +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md @@ -0,0 +1,1013 @@ +--- +title: Managing and rebuilding software RAID on servers in UEFI boot mode +excerpt: Find out how to manage and rebuild software RAID after a disk replacement on your server in UEFI mode +updated: 2025-08-xx +--- + +## Objective + +Redundant Array of Independent Disks (RAID) is a technology that mitigates data loss on a server by replicating data across two or more disks. + +The default RAID level for OVHcloud server installations is RAID 1, which doubles the space taken up by your data, effectively halving the useable disk space. + +**This guide explains how to manage and rebuild software RAID after a disk replacement on your server in UEFI mode** + +Before we begin, please note that this guide focuses on Dedicated servers that use UEFI as the boot mode. This is the case with modern motherboards. If your server uses the BIOS mode (old motherboards), refer to this guide [How to configure and rebuild software RAID on a Dedicated server with BIOS boot mode](). + +For more information on UEFI, consult the following [guide](https://uefi.org/about){.external} + +## Requirements + +- A [dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration +- Administrative (sudo) access to the server via SSH +- Understanding of RAID, partitions and GRUB + +## Instructions + +When you purchase a new server, you might feel the need to perform a series of tests and actions. One of those actions could be simulating a disk failure in order to understand the process of rebuilding the raid and prepare yourself in case this happens. + +### Basic Information + +In a command line session, type the following code to determine the current RAID status: + +```sh +cat /proc/mdstat +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md3 : active raid1 nvme1n1p3[1] nvme0n1p3[0] + 497875968 blocks super 1.2 [2/2] [UU] + bitmap: 2/4 pages [8KB], 65536KB chunk + +md2 : active raid1 nvme1n1p2[1] nvme0n1p2[0] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: +``` + +This command shows us that we have two software RAID devices currently set up, with **md3** being the largest one. This array consists of two partitions, which are known as **nvme1n1p3** and **nvme0n1p3**. + +The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. + +If you have a server with SATA disks, you would get the following results: + +```sh +cat /proc/mdstat +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md3 : active raid1 sda3[0] sdb3[1] + 3904786432 blocks super 1.2 [2/2] [UU] + bitmap: 2/30 pages [8KB], 65536KB chunk + +md2 : active raid1 sda2[0] sdb2[1] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: +``` + +Although this command returns our RAID volumes, it doesn't tell us the size of the partitions themselves. We can find this information with the following command: + +```sh +sudo fdisk -l + +Disk /dev/nvme1n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors +Disk model: WDC CL SN720 SDAQNTW-512G-2000 +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +Disklabel type: gpt +Disk identifier: A11EDAA3-A984-424B-A6FE-386550A92435 + +Device Start End Sectors Size Type +/dev/nvme1n1p1 2048 1048575 1046528 511M EFI System +/dev/nvme1n1p2 1048576 3145727 2097152 1G Linux RAID +/dev/nvme1n1p3 3145728 999161855 996016128 474.9G Linux RAID +/dev/nvme1n1p4 999161856 1000210431 1048576 512M Linux files + + +Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors +Disk model: WDC CL SN720 SDAQNTW-512G-2000 +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +Disklabel type: gpt +Disk identifier: F03AC3C3-D7B7-43F9-88DB-9F12D7281D94 + +Device Start End Sectors Size Type +/dev/nvme0n1p1 2048 1048575 1046528 511M EFI System +/dev/nvme0n1p2 1048576 3145727 2097152 1G Linux RAID +/dev/nvme0n1p3 3145728 999161855 996016128 474.9G Linux RAID +/dev/nvme0n1p4 999161856 1000210431 1048576 512M Linux file +/dev/nvme0n1p5 1000211120 1000215182 4063 2M Linux file + + +Disk /dev/md2: 1022 MiB, 1071644672 bytes, 2093056 sectors +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes + + +Disk /dev/md3: 474.81 GiB, 509824991232 bytes, 995751936 sectors +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +``` + +The `fdisk -l` command also allows you to identify your partition type. This is an important information when it comes to rebuilding your RAID in case of a disk failure. + +For **GPT** partitions, line 6 will display: `Disklabel type: gpt`. + +For **MBR** partitions, line 6 will display: `Disklabel type: dos`. + +Still going by the results of `fdisk -l`, we can see that `/dev/md2` consists of 1022 MiB and `/dev/md3` contains 474.81 GiB. If we were to run the mount command we can also find out the layout of the disk. + +```sh +mount + +sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) +proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) +udev on /dev type devtmpfs (rw,nosuid,relatime,size=16348288k,nr_inodes=4087072,mode=755) +devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) +tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3275188k,mode=755) +/dev/md3 on / type ext4 (rw,relatime) +securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) +tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) +tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) +cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) +pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) +efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) +none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) +systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1462) +hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) +mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) +debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) +tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) +fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) +configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) +/dev/md2 on /boot type ext4 (rw,relatime) +/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro) +tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3275184k,nr_inodes=818796,mode=700,uid=1000,gid=1000) +``` + +Alternatively, the `lsblk` command offers a different view of the partitions: + +```sh +lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +nvme1n1 259:0 0 476.9G 0 disk +├─nvme1n1p1 259:7 0 511M 0 part +├─nvme1n1p2 259:8 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme1n1p3 259:9 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +└─nvme1n1p4 259:10 0 512M 0 part [SWAP] +nvme0n1 259:1 0 476.9G 0 disk +├─nvme0n1p1 259:2 0 511M 0 part /boot/efi +├─nvme0n1p2 259:3 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme0n1p3 259:4 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +├─nvme0n1p4 259:5 0 512M 0 part [SWAP] +└─nvme0n1p5 259:6 0 2M 0 part +``` + +We take note of the devices, partitions and their mount points. + +From the above commands and results, we have: + +- Two RAID arrays: `/dev/md2` and `/dev/md3`. +- Partitions part of the RAID: `/boot` and `/`. +- Partitions not part of the RAID: `/boot/efi` and [SWAP]. + +### System verification + +To check whether a server runs on BIOS mode or UEFI mode, run the following command: + +```sh +[user@server_ip ~]# [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS +``` + +### Understanding the EFI partition + +To begin let's understand what is UEFI mode. + +> [!primary] +> Please note that depending on your operating system, the process might be different. Ubuntu for example is able to keep both partitions synchronized at every GRUB update. However, it is the only operating system doing so. We recommend that you consult the official documentation of your operating system to understand how to manage this partition. +> +> In the guide, the operating system used is Debian. +> + +***What is an EFI partition?*** + +An EFI partition, is a partition which can contain the boot loaders, boot managers, or kernel images of an installed operating system. It also contains system utility programs designed to be run before the operating system boots, as well as data files such as error logs. + +***Is the EFI partition mirrored in RAID?*** + +No, by default, the EFI partition is a partition which cannot be included in the RAID. When you use one of our default installation templates to install your server in software RAID, two EFI partitions are created, one on each disk. However, only one EFI partition is used for the installation of the operating system. + +The partition is mounted on `/boot/efi` and the disk on which it is mounted is selected randomly by the operating system during installation. + +Example: + +```sh +lsblk + +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +nvme1n1 259:0 0 476.9G 0 disk +├─nvme1n1p1 259:7 0 511M 0 part +├─nvme1n1p2 259:8 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme1n1p3 259:9 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +└─nvme1n1p4 259:10 0 512M 0 part [SWAP] +nvme0n1 259:1 0 476.9G 0 disk +├─nvme0n1p1 259:2 0 511M 0 part /boot/efi +├─nvme0n1p2 259:3 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme0n1p3 259:4 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +├─nvme0n1p4 259:5 0 512M 0 part [SWAP] +└─nvme0n1p5 259:6 0 2M 0 part +``` + +From the example above, we see that we have two identical EFI partitions (nvme0n1p1 and nvme1n1p1) but only **nvme0n1p1** is mounted on `/boot/efi`. + +We can also use the following command to confirm that two EFI partitions were actually created after install: + +```sh +sudo lsblk -f +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT +nvme1n1 +├─nvme1n1p1 vfat FAT16 EFI_SYSPART B493-9DFA +├─nvme1n1p2 linux_raid_member 1.2 md2 baae988b-bef3-fc07-615f-6f9043cfd5ea +│ └─md2 ext4 1.0 boot 96850c4e-e2b5-4048-8c39-525194e441aa 851.8M 7% /boot +├─nvme1n1p3 linux_raid_member 1.2 md3 ce0c7fac-0032-054c-eef7-7463b2245519 +│ └─md3 ext4 1.0 root 6fea39e9-6297-4ea3-82f1-bf1a3e88106a 441.3G 0% / +└─nvme1n1p4 swap 1 swap-nvme1n1p4 483b9b41-ada3-4143-8cac-5bff7afb73c7 [SWAP] +nvme0n1 +├─nvme0n1p1 vfat FAT16 EFI_SYSPART B486-9781 504.9M 1% /boot/efi +├─nvme0n1p2 linux_raid_member 1.2 md2 baae988b-bef3-fc07-615f-6f9043cfd5ea +│ └─md2 ext4 1.0 boot 96850c4e-e2b5-4048-8c39-525194e441aa 851.8M 7% /boot +├─nvme0n1p3 linux_raid_member 1.2 md3 ce0c7fac-0032-054c-eef7-7463b2245519 +│ └─md3 ext4 1.0 root 6fea39e9-6297-4ea3-82f1-bf1a3e88106a 441.3G 0% / +├─nvme0n1p4 swap 1 swap-nvme0n1p4 51e7172b-adb0-4729-b0f8-613e5dede38b [SWAP] +└─nvme0n1p5 iso9660 Joliet Extension config-2 2025-08-05-14-55-41-00 +``` + +From the results above, we can see two partitions (**nvme0n1p1** and **nvme1n1p1**), with identical size (504.9M). Both partitions have the LABEL: `EFI_SYSPART` but only one is mounted on `/boot/efi`. + +***Does the content of EFI partition change regularly?*** + +In general, the contents of this partition do not change much, except when there are relevant updates such as grub/kernel updates or even simple `apt` or `yum` updates. + +In this case, we recommend running an automatic or manual script keep the partitions in sync. This is because in case of the failure of the main disk on which this partition in mounted, you can use the rescue mode to recover it. + +***What happens if my main disk fails and is replaced?*** + +If the main disk on your server fails and it is replaced. The server will be rebooted in rescue mode in other for you to rebuild the raid. Since the EFI partition is not mirrored, the server will not be able to boot on the secondary disk (in most cases). You can rebuild the raid in rescue mode, recreate the EFI partition and the disk should be able to boot. + +***What if the main disk fails and I did not synchronise my EFI partitions after a major update (kernel/grub)?*** + +If this is the case, you will have to rebuild the raid, create the EFI partition on the new disk and install the grub/kernel update in rescue mode. If you are not able to perform some manipulations, we recommend contacting a professional service. + +***How can I keep my EFI partitions synchronized or how often should I sychronize them?*** + +We recommend synchronizing both partitions daily. Below is a script you can use to manually synchronize your EFI partitions. You can also run an automated script to synchronize the partitions daily or when the service is booted. + +```sh +set -euo pipefail + +findmnt -n -o SOURCE /boot/efi +MAIN_PARTITION=/dev/ # sdX or nvmeXnXpX +MOUNTPOINT="/var/lib/grub/esp" + +mkdir -p "${MOUNTPOINT}" + +while read -r partition; do + if [[ "${partition}" == "${MAIN_PARTITION}" ]]; then + continue + fi + echo "Working on ${partition}" + mount "${partition}" "${MOUNTPOINT}" + rsync -ax "/boot/efi/" "${MOUNTPOINT}/" + umount "${MOUNTPOINT}" +done < <(blkid -o device -t LABEL=EFI_SYSPART) +``` + +In the script above, the `MAIN_PARTITION` is the one mounted on /boot/efi and `MOUNTPOINT` is where you want to sync the contents. + +Before you run the script, make sure `rsync` is installed: + +**Debian/Ubuntu** + +```sh +sudo apt install rsync +``` + +**CentOS, Red Hat and Fedora** + +```sh +sudo yum install rsync +``` + +Example: + +```sh +set -euo pipefail + +findmnt -n -o SOURCE /boot/efi +MAIN_PARTITION=/dev/nvme0n1p1 +MOUNTPOINT="/var/lib/grub/esp" + +mkdir -p "${MOUNTPOINT}" + +while read -r partition; do + if [[ "${partition}" == "${MAIN_PARTITION}" ]]; then + continue + fi + echo "Working on ${partition}" + mount "${partition}" "${MOUNTPOINT}" + rsync -ax "/boot/efi/" "${MOUNTPOINT}/" + umount "${MOUNTPOINT}" +done < <(blkid -o device -t LABEL=EFI_SYSPART) +``` + +In the example above, the main partition is `nvme0n1p1` and the mountpoint we use is `/var/lib/grub/esp`. + +### Simulating a disk failure + +Now that we have all the necessary information, we can simulate a disk failure and proceed with the tests. In this example, we will fail the main disk `nvme0n1`. + +The preferred way to do this is via the OVHcloud rescue mode environment. + +First reboot in rescue menu and log in with the credentials provided. + +To remove a disk from the RAID, the first step is to mark it as **Failed** and remove the partitions from their respective RAID arrays. + +```sh +root@rescue12-customer-ca (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme0n1p3[0] nvme1n1p3[1] + 497875968 blocks super 1.2 [2/2] [UU] + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: +``` + +From the above output, nvme0n1 consists of two partitions in RAID which are **nvme0n1p2** and **nvme0n1p3**. + +#### Removing the failed disk + +First we mark the partitions **nvme0n1p2** and **nvme0n1p3** as failed. + +```sh +sudo mdadm --manage /dev/md2 --fail /dev/nvme0n1p2 + +# mdadm: set /dev/nvme0n1p2 faulty in /dev/md2 +``` + +```sh +sudo mdadm --manage /dev/md3 --fail /dev/nvme0n1p3 + +# mdadm: set /dev/nvme0n1p3 faulty in /dev/md3 +``` + +We have now simulated a failure of the RAID, when we run the `cat /proc/mdstat` command, we have the following output + +```sh +root@rescue12-customer-ca (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme0n1p3[0](F) nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme0n1p2[2](F) nvme1n1p2[1] + 1046528 blocks super 1.2 [2/1] [_U] + +unused devices: +``` + +As we can see above, the [F] next to the partitions indicates that the disk has failed or is faulty. + +Next, we remove these partitions from the RAID arrays to completely remove the disk from RAID. + +```sh +sudo mdadm --manage /dev/md2 --remove /dev/nvme0n1p2 + +# mdadm: hot removed /dev/nvme0n1p2 from /dev/md2 +``` + +```sh +sudo mdadm --manage /dev/md3 --remove /dev/nvme0n1p3 + +# mdadm: hot removed /dev/nvme0n1p3 from /dev/md3 +``` + +To make sure that we get a disk that is similar to an empty disk, we use the following command on each partition, then on the disk. + +```sh +shred -s10M -n1 /dev/nvme0n1p1 +shred -s10M -n1 /dev/nvme0n1p2 +shred -s10M -n1 /dev/nvme0n1p3 +shred -s10M -n1 /dev/nvme0n1p4 +shred -s10M -n1 /dev/nvme0n1 +``` + +The disk now appears as a new one: + +```sh +lsblk -A + +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +nvme1n1 259:0 0 476.9G 0 disk +├─nvme1n1p1 259:1 0 511M 0 part +├─nvme1n1p2 259:2 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 +├─nvme1n1p3 259:3 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 +└─nvme1n1p4 259:4 0 512M 0 part +nvme0n1 259:5 0 476.9G 0 disk +``` + +If we run the following command, we see that our disk has been successfully "wiped": + +```sh +parted /dev/nvme0n1 +GNU Parted 3.5 +Using /dev/nvme0n1 +Welcome to GNU Parted! Type 'help' to view a list of commands. +(parted) p +Error: /dev/nvme0n1: unrecognised disk label +Model: WDC CL SN720 SDAQNTW-512G-2000 (nvme) +Disk /dev/nvme0n1: 512GB +Sector size (logical/physical): 512B/512B +Partition Table: unknown +Disk Flags: +``` + +Our RAID status should now look like this: + +```sh +root@rescue12-customer-ca (ns5009452.ip-51-222-254.net) ~ # cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme1n1p2[1] + 1046528 blocks super 1.2 [2/1] [_U] + +unused devices: n +``` + +From the results above, we can see that only two partitions now appear in the RAID arrays. We have successfully failed the disk **nvme0n1** and we can now proceed with the hypothetical disk replacement. + +For more information on how to prepare and request for a disk replacement, consult this [guide](/pages/bare_metal_cloud/dedicated_servers/disk_replacement) + +If you run the following command, you can have more details on the RAID array (s): + + +```sh +mdadm --detail /dev/md3 + +/dev/md3: + Version : 1.2 + Creation Time : Fri Aug 1 14:51:13 2025 + Raid Level : raid1 + Array Size : 497875968 (474.81 GiB 509.82 GB) + Used Dev Size : 497875968 (474.81 GiB 509.82 GB) + Raid Devices : 2 + Total Devices : 1 + Persistence : Superblock is persistent + + Intent Bitmap : Internal + + Update Time : Fri Aug 1 15:56:17 2025 + State : clean, degraded + Active Devices : 1 + Working Devices : 1 + Failed Devices : 0 + Spare Devices : 0 + +Consistency Policy : bitmap + + Name : md3 + UUID : b383c3d5:7fb1bb5e:6b7c4d96:6ea817ff + Events : 215 + + Number Major Minor RaidDevice State + - 0 0 0 removed + 1 259 4 1 active sync /dev/nvme1n1p3 +``` + +### Rebuilding the RAID + +The following steps document the RAID rebuild in rescue mode. + +> [!primary] +> This process might be different depending on the operating system you have installed on your server. We recommend that you consult the official documentation of your operating system to have access to the proper commands. +> + +> [!warning] +> +> For most servers in software RAID, after a disk replacement, the server is able to reboot in normal mode (on the healthy disk) and the rebuild can be done in normal mode. However, if the server is not able to reboot in normal mode after a disk replacement, it will be rebooted in rescue mode to proceed with the raid rebuild. +> +> If your server is able to boot in normal mode after the RAID rebuilding, simply proceed with the steps from [this section](). + +#### Rebuilding the RAID after the main disk is replaced + +Here, we assume that the EFI partitions have been kept in sync (including after GRUB/kernel updates. If that is not your case, we have provided additional information on how to proceed). + +Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, nvme1n1) to the new one (nvme0n1). + +> [!tabs] +> **For GPT partitions** +>> +>> ```sh +>> sgdisk -R /dev/nvmeXnX /dev/nvmeAnA +>> ``` +>> +>> The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` +>> +>> In our example: +>> +>> ```sh +>> sgdisk -R /dev/nvme0n1 /dev/nvme1n1 +>> ``` +>> +>> We run `lsblk` to make sure the partition tables have been properly copied: +>> +>> ```sh +>> lsblk +>> +>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +>> nvme1n1 259:0 0 476.9G 0 disk +>> ├─nvme1n1p1 259:1 0 511M 0 part +>> ├─nvme1n1p2 259:2 0 1G 0 part +>> │ └─md2 9:2 0 1022M 0 raid1 +>> ├─nvme1n1p3 259:3 0 474.9G 0 part +>> │ └─md3 9:3 0 474.8G 0 raid1 +>> └─nvme1n1p4 259:4 0 512M 0 part +>> nvme0n1 259:5 0 476.9G 0 disk +>> ├─nvme0n1p1 259:10 0 511M 0 part +>> ├─nvme0n1p2 259:11 0 1G 0 part +>> ├─nvme0n1p3 259:12 0 474.9G 0 part +>> └─nvme0n1p4 259:13 0 512M 0 part +>> ``` +>> +>> Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: +>> +>> ```sh +>> sgdisk -G /dev/nvme0n1 +>> ``` +>> +>> If you receive a message like this: +>> +>> ```console +>> Warning: The kernel is still using the old partition table. +>> The new table will be used at the next reboot or after you +>> run partprobe(8) or kpartx(8) +>> The operation has completed successfully. +>> ``` +>> +>> You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. +>> +> **For MBR partitions** +>> +>> ```sh +>> sudo sfdisk -d /dev/sdX | sfdisk /dev/sdX +>> ``` +>> +>> The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk`. +>> + +We can now rebuild the RAID array. The following code snippet shows how we can add the new partitions (nvme0n1p2 and nvme0n1p3) back in the RAID array. You need to replace the appropriate values with your own. + +```sh +sudo mdadm --add /dev/md2 /dev/nvme0n1p2 + +# mdadm: added /dev/nvme0n1p2 + +sudo mdadm --add /dev/md3 /dev/nvme0n1p3 + +# mdadm: re-added /dev/nvme0n1p3 +``` + +You can check the rebuild process + +```sh +cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme0n1p3[2] nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + [>....................] recovery = 0.1% (801920/497875968) finish=41.3min speed=200480K/sec + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] + 1046528 blocks super 1.2 [2/2] [UU] +``` + +Once the RAID rebuild is complete, run the following command to make sure that the partitions were properly added to the RAID: + +```sh +lsblk -fA +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS +nvme1n1 +├─nvme1n1p1 vfat FAT16 EFI_SYSPART 4629-D183 +├─nvme1n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d +├─nvme1n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f +└─nvme1n1p4 swap 1 swap-nvme1n1p4 9bf292e8-0145-4d2f-b891-4cef93c0d209 +nvme0n1 +├─nvme0n1p1 +├─nvme0n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d +├─nvme0n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f +└─nvme0n1p4 +``` + +From the above results, we can see that the partitions of our newly added disk have been properly added to the RAID, however, the EFI partition was not duplicated, which is normal since it is not included in the RAID. We also see that the [SWAP] partition **nvme0n1p4** does not have a label anymore (this will not be the same for all cases, especially if you customise your partitions before your server's installation). + +> [!warning] +> The examples above are merely illustrating the necessary steps based on a default server configuration. The information in the output table depends on your server's hardware and its partition scheme. When in doubt, consult the documentation of your operating system. +> +> If you require professional assistance with server administration, consider the details in the Go further section of this guide. +> + +#### Recreating the EFI partition + +The next step is to format **nvme0n1p1** to create the EFI partition, then replicate the content of this partition from the healthy drive (in our example: nvme1n1) to the new one. + +This is possible because we kept the partitions in sync while the server was running normally before the disk failure. So we have both partitions up to date. + +> [!warning] +> If there was a kernel or grub update or a major update and both partitions were not synchronized, consult the following guide [Repairing the GRUB bootloader]() once you are done creating the new EFI partition. +> + +First, we format the partition: + +```sh +mkfs.vfat /dev/nvme0n1p1 +``` + +Next, we label the partition as `EFI_SYSPART` (this naming is proper to OVHcloud) + +```sh +fatlabel /dev/nvme0n1p1 EFI_SYSPART +``` + +Now we have to duplicate the content from the partition nvme1n1p1 (healthy partition), to nvme0n1p1 (new partition). To do so, we start by creating two folders named `old` and `new`. + +```sh +mkdir old new +``` + +Next, we mount **nvme1n1p1** in the `old` folder and **nvme0n1p1** in the `new` folder to make the distinction: + +```sh +mount /dev/nvme1n1p1 old +mount /dev/nvme0n1p1 new +``` + +Next, we copy the files from `old` to `new`. Depending on your operating system, you will have a similar output. Here we are using debian: + +```sh +rsync -axv old/ new/ +``` + +Once this is done, we unmount the **nvme0n1p1** partition. + +```sh +umount /dev/nvme0n1p1 +``` + +Next, we mount the partition containing our files on `/mnt`. In our example, that partition is `md3`. + +```sh +mount /dev/md3 /mnt +``` + +Next, we mount the following directories to make sure any manipulation we make in the chroot environment works properly: + +```sh +mount --types proc /proc /mnt/proc +mount --rbind /sys /mnt/sys +mount --make-rslave /mnt/sys +mount --rbind /dev /mnt/dev +mount --make-rslave /mnt/dev +mount --bind /run /mnt/run +mount --make-slave /mnt/run +``` + +Next, we use the `chroot` command to access the mountpoint and make sure the new EFI partition has been properly created and the system recongnises both EFI partitions: + +```sh +chroot /mnt +``` + +```sh +blkid -t LABEL=EFI_SYSPART +/dev/nvme1n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="EFI_SYSPART" LABEL="EFI_SYSPART" UUID="4629-D183" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="889f241b-49c3-4031-b5c9-60df0746f98f" +/dev/nvme0n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="EFI_SYSPART" LABEL="EFI_SYSPART" UUID="521F-300B" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="02bf2b2d-7ada-4461-ba50-07683519f65d" +``` + +/// details | **GRUB/kernel update with EFI partitions not synchronized** + +If GRUB/kernal updates were made and the EFI partitions were not kept in sync, this additional step could necessary. + +Still in the `chroot` environment, we create the `/boot/efi` folder in order to mount the EFI partition **nvme0n1p1** in it: + +```sh +cd /mnt +mkdir /boot/efi +mount /dev/nvme0n1p1 /boot/efi +``` + +Next, we install the grub bootloader to make sure the server can reboot in normal mode on the new disk (you won't have to do this if the disk replaced is the secondary disk. Simply duplicate the EFI partition and proceed to the RAID rebuild, then enable the [SWAP] partition (if applicable)): + +```sh +grub-install --efi-directory=/boot/efi /dev/nvme0n1p1 +``` +/// + + + +#### Adding the label to the SWAP partition + +We exit the `chroot` environment, then we recreate our [SWAP] partition **nvme0n1p4** and add the label `swap-nvmenxxx`: + +```sh +mkswap /dev/nvme0n1p4 -L swap-nvme0n1p4 +mkswap: /dev/nvme0n1p4: warning: wiping old swap signature. +Setting up swapspace version 1, size = 512 MiB (536866816 bytes) +LABEL=swap-nvme0n1p4, UUID=b3c9e03a-52f5-4683-81b6-cc10091fcd +``` + +We verify that the label has been properly applied: + +```sh +lsblk -fA +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS +nvme1n1 + +├─nvme1n1p1 +│ vfat FAT16 EFI_SYSPART +│ BA77-E844 504.9M 1% /root/old +├─nvme1n1p2 +│ linux_ 1.2 md2 53409058-480a-bc65-4e1d-6acc848fe233 +│ └─md2 +│ ext4 1.0 boot f925a033-0087-40ec-817e-44efab0351ac +├─nvme1n1p3 +│ linux_ 1.2 md3 a3b8816c-a5c3-7f01-ee17-e1aa9685c35c +│ └─md3 +│ ext4 1.0 root 6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 441.2G 0% /mnt +└─nvme1n1p4 + swap 1 swap-nvme1n1p4 + d6af33cf-fc15-4060-a43c-cb3b5537f58a +nvme0n1 + +├─nvme0n1p1 +│ vfat FAT16 EFI_SYSPART +│ 477D-6658 +├─nvme0n1p2 +│ linux_ 1.2 md2 53409058-480a-bc65-4e1d-6acc848fe233 +│ └─md2 +│ ext4 1.0 boot f925a033-0087-40ec-817e-44efab0351ac +├─nvme0n1p3 +│ linux_ 1.2 md3 a3b8816c-a5c3-7f01-ee17-e1aa9685c35c +│ └─md3 +│ ext4 1.0 root 6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 441.2G 0% /mnt +└─nvme0n1p4 + swap 1 swap-nvme0n1p4 + b3c9e03a-52f5-4683-81b6-cc10091fcd15 +``` + +Next, we access the `chroot` environment again: + +```sh +chroot /mnt +``` + +We retrieve the UUIDs of both swap partitions: + +```sh +blkid /dev/nvme0n1p4 +blkid /dev/nvme1n1p4 +``` + +Example: + +```sh +blkid /dev/nvme0n1p4 +/dev/nvme0n1p4: LABEL="swap-nvme0n1p4" UUID="b3c9e03a-52f5-4683-81b6-cc10091fcd15" TYPE="swap" PARTLABEL="logical" PARTUUID="cb6022d9-7631-449c-956a-116bcd0e2119" +``` + +```sh +blkid /dev/nvme1n1p4 +/dev/nvme1n1p4: LABEL="swap-nvme1n1p4" UUID="d6af33cf-fc15-4060-a43c-cb3b5537f58a" TYPE="swap" PARTLABEL="logical" PARTUUID="d037c35f-2ddb-40d5-be33-31cc496fe54b" +``` + +Next, we replace the old UUID of the swap partition (**nvme0n1p4)** with the new one in `/etc/fstab`: + +```sh +nano etc/fstab +``` + +Example: + +```sh +UUID=6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 / ext4 defaults 0 1 +UUID=f925a033-0087-40ec-817e-44efab0351ac /boot ext4 defaults 0 0 +LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1 +UUID=b7b5dd38-9b51-4282-8f2d-26c65e8d58ec swap swap defaults 0 0 +UUID=d6af33cf-fc15-4060-a43c-cb3b5537f58a swap swap defaults 0 0 +``` + +Make sure you replace the proper UUID. In our example above, the UUID to replace is `b7b5dd38-9b51-4282-8f2d-26c65e8d58ec` with the new one `b3c9e03a-52f5-4683-81b6-cc10091fcd15` since the other one belongs to **nvme1n1p4**. + +Next, we make sure everything is properly mounted: + +```sh +mount -av +/ : ignored +mount: (hint) your fstab has been modified, but systemd still uses + the old version; use 'systemctl daemon-reload' to reload. +/boot : successfully mounted +/boot/efi : successfully mounted +swap : ignored +swap : ignored +``` + +We enable the swap partition: + +```sh +swapon -av + +swapon: /dev/nvme0n1p4: found signature [pagesize=4096, signature=swap] +swapon: /dev/nvme0n1p4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/nvme0n1p4 +swapon: /dev/nvme1n1p4: found signature [pagesize=4096, signature=swap] +swapon: /dev/nvme1n1p4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/nvme1n1p4 +``` + +We exit the Chroot environment with `Exit` and unmount all the disks: + +```sh +umount -a +``` + +We have now successfully completed the RAID rebuild on the server and we can now reboot the server in normal mode. + +/// details | **Rebuilding the RAID after the secondary disk is replaced** + +The following are to be followed in normal mode since it is the secondary disk that is being replaced and the server will be booted in normal mode. Our secondary disk is **nvme1n1**. + +Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, nvme0n1) to the new one (nvme1n1). + +> [!tabs] +> **For GPT partitions** +>> +>> ```sh +>> sgdisk -R /dev/nvmeXnX /dev/nvmeAnA +>> ``` +>> +>> The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk`. +>> +>> Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: +>> +>> ```sh +>> sgdisk -G /dev/nvmeXnX +>> ``` +>> +>> If you receive a message like this: +>> +>> ```console +>> Warning: The kernel is still using the old partition table. +>> The new table will be used at the next reboot or after you +>> run partprobe(8) or kpartx(8) +>> The operation has completed successfully. +>> ``` +>> +>> You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. +>> +> **For MBR partitions** +>> +>> ```sh +>> sudo sfdisk -d /dev/sdX | sfdisk /dev/sdX +>> ``` +>> +>> The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk`. +>> + +Next, we add the partitions to the RAID: + +```sh +sudo mdadm --add /dev/md2 /dev/nvme1n1p2 + +# mdadm: added /dev/nvme1n1p2 + +sudo mdadm --add /dev/md3 /dev/nvme1n1p3 + +# mdadm: re-added /dev/nvme1n1p3 +``` + +Use the following command to follow the RAID rebuild `cat /proc/mdstat`. + +**Recreating the EFI partition on the disk** + +First, we install the necessary tools: + +**Debian and Ubuntu** + +```sh +sudo apt install dosfstools +``` + +**CentOS** + +```sh +sudo yum install dosfstools +``` + +Next, we format the partition. In our example nvme1n1p1: + +```sh +sudo mkfs.vfat /dev/nvme1n1p1 +``` + +Next, we label the partition as `EFI_SYSPART` (this naming is proper to OVHcloud) + +```sh +sudo fatlabel /dev/nvme1n1p1 EFI_SYSPART +``` + +We sync both partitions using a script. + +Example: + +```sh +set -euo pipefail + +findmnt -n -o SOURCE /boot/efi +MAIN_PARTITION=/dev/nvme0n1p1 +MOUNTPOINT="/var/lib/grub/esp" + +mkdir -p "${MOUNTPOINT}" + +while read -r partition; do + if [[ "${partition}" == "${MAIN_PARTITION}" ]]; then + continue + fi + echo "Working on ${partition}" + mount "${partition}" "${MOUNTPOINT}" + rsync -ax "/boot/efi/" "${MOUNTPOINT}/" + umount "${MOUNTPOINT}" +done < <(blkid -o device -t LABEL=EFI_SYSPART) +``` + +Lastly, we enable the [SWAP] partition (if applicable): + +```sh +sudo mkswap /dev/nvme1n1p4 -L swap-nvme1n1p4 +``` + +We retrieve the UUIDs of both swap partitions: + +```sh +sudo blkid /dev/nvme0n1p4 +sudo blkid /dev/nvme1n1p4 +``` + +We replace the old UUID of the swap partition (**nvme1n1p4)** with the new one in `/etc/fstab`: + +```sh +sudo nano /etc/fstab +``` +Make sure you replace the correct UUID. Reload the system with the following command: + +```sh +sudo systemctl daemon-reload +``` + +Next, run the following command to enable it: + +```sh +sudo swapon -av +``` + +We have now successfully completed the RAID rebuild. + +## Go Further + +[Hot Swap - Software RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_soft) + +[OVHcloud API and Storage](/pages/bare_metal_cloud/dedicated_servers/partitioning_ovh) + +[Managing hardware RAID](/pages/bare_metal_cloud/dedicated_servers/raid_hard) + +[Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) + +Join our [community of users](/links/community). \ No newline at end of file