|
| 1 | +--- |
| 2 | +title: Troubleshoot Linux VM Boot Failure After Enabling Azure Disk Encryption |
| 3 | +description: Provides troubleshooting steps for an issue where an Azure Linux virtual machine fails to boot after deploying Azure Disk Encryption. |
| 4 | +ms.date: 05/08/2025 |
| 5 | +ms.service: azure-virtual-machines |
| 6 | +ms.custom: sap:Azure Disk Encryption (ADE) for Linux, linux-related-content |
| 7 | +ms.reviewer: divargas, elcorral, v-weizhu |
| 8 | +--- |
| 9 | +# Troubleshoot Linux VM boot failure after enabling Azure Disk Encryption |
| 10 | + |
| 11 | +**Applies to:** :heavy_check_mark: Linux VMs |
| 12 | + |
| 13 | +When you deploy Azure Disk Encryption (ADE) for an Azure Linux virtual machine (VM), various essential settings related to the boot process and system components are modified by editing files. If the ADE deployment fails or is interrupted, the VM might be stuck in emergency mode or unusable, especially when the operating system (OS) disk is encrypted. |
| 14 | + |
| 15 | +This article lists common scenarios where a Linux VM fails to boot after an ADE deployment and provides steps to troubleshoot the issue. |
| 16 | + |
| 17 | +## Prerequisites |
| 18 | + |
| 19 | +In all scenarios, you should [take a snapshot](/azure/virtual-machines/linux/snapshot-copy-managed-disk) or create a backup before disks are encrypted. Backups ensure that a recovery option is possible if an unexpected failure occurs during encryption. For more information about how to back up and restore encrypted VMs, see [Azure Backup](/azure/backup/backup-azure-vms-encryption). |
| 20 | + |
| 21 | +## Extension logs |
| 22 | + |
| 23 | +To troubleshoot Linux VM boot issues, check the extension logs in the Azure serial console or the extension log file `/var/log/azure/Microsoft.Azure.Security.AzureDiskEncryptionForLinux/extension.log`. |
| 24 | + |
| 25 | +## <a id="initram-miss"> </a> Scenario 1: ADE modules are missing from the initramfs image |
| 26 | + |
| 27 | +If the OS disk uses Logical Volume Manager (LVM) and you see a warning message like the following text, the required modules aren't included in the initial RAM disk image: |
| 28 | + |
| 29 | + ```output |
| 30 | + Warning: /dev/mapper/rootvg-rootlv does not exist |
| 31 | + ... |
| 32 | + Entering emergency mode. Exit the shell to continue. |
| 33 | + dracut:/# |
| 34 | + ``` |
| 35 | + |
| 36 | +To resolve this issue, follow these steps: |
| 37 | + |
| 38 | +1. [Restore the VM from a backup](/azure/backup/restore-azure-encrypted-virtual-machines) and attempt the encryption again. |
| 39 | +2. If a restore isn't feasible, use the Azure CLI extension [az vm repair](unlock-encrypted-linux-disk-offline-repair.md#method1) to create a repair VM, or use the [manual method](unlock-encrypted-linux-disk-offline-repair.md#method2) to create a rescue VM. Then, attach and unlock the OS disk of the faulty VM to that repair/rescue VM. |
| 40 | +3. Once you're in [chroot](chroot-environment-linux.md), execute the following commands: |
| 41 | + |
| 42 | + > [!NOTE] |
| 43 | + > Replace the kernel and extension version accordingly in the following commands. |
| 44 | +
|
| 45 | + ### [RHEL 8 and 9](#tab/redhat) |
| 46 | + |
| 47 | + 1. Copy the following files from the extension configuration directory to the initramfs scripts directory: |
| 48 | + |
| 49 | + ```bash |
| 50 | + sudo cp /var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-X.X.X.X/main/oscrypto/91adeOnline /usr/lib/dracut/modules.d/ |
| 51 | + ``` |
| 52 | + |
| 53 | + 2. Regenerate the initramfs image: |
| 54 | + |
| 55 | + ```bash |
| 56 | + sudo dracut -f -v /boot/initramfs-X.XX.X-XXX.XX.X.x86_64.img <kernel version> |
| 57 | + ``` |
| 58 | + |
| 59 | + 3. Test the modified kernel by booting the VM from it. If it works, regenerate any remaining initramfs files. |
| 60 | + |
| 61 | + ### [Ubuntu](#tab/ubuntu) |
| 62 | + |
| 63 | + > [!NOTE] |
| 64 | + > This procedure might also apply to boot issues that occur after a VM is upgraded from Ubuntu 18 to Ubuntu 20. |
| 65 | + |
| 66 | + 1. Copy the following files from the extension configuration directory to the initramfs scripts directory: |
| 67 | + |
| 68 | + ```bash |
| 69 | + sudo cd /var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-X.X.X.X/main/oscrypto/ubuntu_2004/encryptscripts |
| 70 | + sudo cp crypt-ade-boot /usr/share/initramfs-tools/scripts/init-premount/ |
| 71 | + sudo cp crypt-ade-hook /usr/share/initramfs-tools/hooks/ |
| 72 | + ``` |
| 73 | + |
| 74 | + 2. Once the `crypt-ade-boot` file is copied, list the OS partition from `/dev/disk/by-partuuid/` by Universally Unique Identifier (UUID): |
| 75 | + |
| 76 | + ```bash |
| 77 | + sudo ls -l /dev/disk/by-partuuid/ | grep -w <partition containing the OS> |
| 78 | + lrwxrwxrwx 1 root root 10 May 18 17:33 <ROOTPARTUUID> -> ../../sda1 |
| 79 | + ``` |
| 80 | + |
| 81 | + 3. Unlock the root partition by running the following command. Replace the `ROOTPARTUUID` variable with the one obtained in the preceding step. |
| 82 | + |
| 83 | + ```bash |
| 84 | + cryptsetup luksOpen /dev/disk/by-partuuid/ROOTPARTUUID osencrypt --header /boot/luks/osluksheader -d /mnt/azure_bek_disk/LinuxPassPhraseFileName |
| 85 | + ``` |
| 86 | + |
| 87 | + 4. Regenerate the initramfs image: |
| 88 | + |
| 89 | + ```bash |
| 90 | + update-initramfs -u -k all |
| 91 | + ``` |
| 92 | + |
| 93 | + An output similar to the following text is expected: |
| 94 | + |
| 95 | + ```output |
| 96 | + update-initramfs: Generating /boot/initrd.img-5.15.0-1038-azure |
| 97 | + cryptsetup: WARNING: target 'osencrypt' not found in /etc/crypttab |
| 98 | + + PREREQS=udev |
| 99 | + + mount -a |
| 100 | + + cryptsetup luksOpen /dev/disk/by-partuuid/<ROOTPARTUUID> osencrypt --header /boot/luks/osluksheader -d /mnt/azure_bek_disk/LinuxPassPhraseFileName |
| 101 | + Device osencrypt already exists. |
| 102 | + + exit 0 |
| 103 | + ``` |
| 104 | + |
| 105 | + --- |
| 106 | + |
| 107 | +4. Swap the failed OS disk with the one containing the fix. |
| 108 | + |
| 109 | +5. Review the extension logs and serial console logs to ensure the encryption process finishes successfully: |
| 110 | + |
| 111 | + ```output |
| 112 | + [AzureDiskEncryption] 3670: [Info] ======= MACHINE STATE: completed ======= |
| 113 | + [AzureDiskEncryption] 3670: [Info] Encryption succeeded for all volumes |
| 114 | + [AzureDiskEncryption] 3670: [Info] Executing: lvs --noheadings --nameprefixes --unquoted -o lv_name,vg_name,lv_kernel_major,lv_kernel_minor |
| 115 | + [AzureDiskEncryption] 3670: [Info] OS PV is encrypted |
| 116 | + [AzureDiskEncryption] 3670: [Info] found one ide with vmbus: <GUID> and the sdx path is: sda |
| 117 | + [AzureDiskEncryption] 3670: [Info] found one ide with vmbus: <GUID> and the sdx path is: sdb |
| 118 | + [AzureDiskEncryption] 3670: [Info] found one ide with vmbus: <GUID> and the sdx path is: sdc |
| 119 | + [AzureDiskEncryption] 3670: [Info] Executing: pvs |
| 120 | + [AzureDiskEncryption] 3670: [Info] Found OS block device: /dev/mapper/osencrypt |
| 121 | + ``` |
| 122 | + |
| 123 | +6. Review the initramfs files to ensure that the ADE modules are properly added: |
| 124 | + |
| 125 | + > [!NOTE] |
| 126 | + > Replace the initramfs file name accordingly in the following commands. |
| 127 | + |
| 128 | + ### [RHEL 8 and 9](#tab/redhat) |
| 129 | + |
| 130 | + ```bash |
| 131 | + sudo lsinitrd /boot/initramfs-4.18.0-553.45.1.el8_10.x86_64.img | egrep -w "ade|adeOnline" |
| 132 | + ``` |
| 133 | + |
| 134 | + An output similar to the following one is expected: |
| 135 | + |
| 136 | + ```output |
| 137 | + -rwxr--r-- 1 root root 1126 Jan 15 2024 usr/lib/dracut/hooks/cmdline/30-parse-crypt-ade.sh |
| 138 | + -rwxr--r-- 1 root root 681 Jan 15 2024 usr/sbin/crypt-run-generator-ade |
| 139 | + ``` |
| 140 | + |
| 141 | + ### [Ubuntu](#tab/ubuntu) |
| 142 | + |
| 143 | + ```bash |
| 144 | + lsinitramfs /boot/initrd.img-5.15.0-1082-azure | egrep -i ade |
| 145 | + ``` |
| 146 | + |
| 147 | + An output similar to the following one is expected: |
| 148 | + |
| 149 | + ```output |
| 150 | + boot/luks/osluksheader |
| 151 | + conf/conf.d/cryptheader |
| 152 | + scripts/init-premount/crypt-ade-boot |
| 153 | + ``` |
| 154 | + |
| 155 | + --- |
| 156 | + |
| 157 | +## Scenario 2: The encryption is interrupted |
| 158 | + |
| 159 | +The steps for troubleshooting an interrupted encryption depend on where the process is interrupted. In some scenarios, [restoring from a backup](/azure/backup/restore-azure-encrypted-virtual-machines) might be the only option. |
| 160 | + |
| 161 | +1. Check the serial console logs for any error messages. |
| 162 | + |
| 163 | + Extension deployment issues are typically indicated by Python errors. For example: |
| 164 | + |
| 165 | + ```output |
| 166 | + [AzureDiskEncryption] 7293: [Info] Command /sbin/e2fsck -f -y /dev/korn-fromme failed with return code 8 |
| 167 | + stdout: |
| 168 | + Possibly non-existent device? |
| 169 | + stderr: |
| 170 | + e2fsck 1.46.5 (30-Dec-2021) |
| 171 | + /sbin/e2fsck: No such file or directory while trying to open /dev/korn-fromme |
| 172 | + [AzureDiskEncryption] 7293: [Error] check shrink fs failed with code 8 for /dev/korn-fromme |
| 173 | + [AzureDiskEncryption] 7293: [Info] Your file system may not have enough space to do the encryption or file System may not support resizing |
| 174 | + [AzureDiskEncryption] 7293: [Error] Failed to encrypt data volumes with error: Encryption failed for name:korn-fromme type:lvm fstype:ext4 mountpoint:/someone label: model: size:6442450944 majmin:None device_id:, stack trace: Traceback (most recent call last): |
| 175 | + File '/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.4.0.7/main/handle.py', line 2172, in daemon_encrypt |
| 176 | + while not daemon_encrypt_data_volumes(encryption_marker=encryption_marker, |
| 177 | + File '/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux-1.4.0.7/main/handle.py', line 2438, in daemon_encrypt_data_volumes |
| 178 | + raise Exception(message) |
| 179 | + Exception: Encryption failed for name:korn-fromme type:lvm fstype:ext4 mountpoint:/someone label: model: size:6442450944 majmin:None device_id: |
| 180 | + ``` |
| 181 | +
|
| 182 | +2. Ensure all the [extension prerequisites](/azure/virtual-machines/linux/disk-encryption-overview#additional-vm-requirements) are met. |
| 183 | +
|
| 184 | +3. If necessary, work on a rescue VM and analyze the failed disk. For the OS disk, ensure the following things: |
| 185 | + - The required partitions are in place, and the data is healthy. |
| 186 | + - The [OS LUKS header file](unlock-encrypted-linux-disk-offline-repair.md#identify-the-header-file), `osluksheader`, is stored separately under the boot partition. If the disk is encrypted and this file is missing or corrupted, you can't recover the VM without a working backup. |
| 187 | + - The initramfs contains the required ADE modules. If the required modules are missing, follow the steps in the [ADE modules are missing from the initram image](#initram-miss) section. |
| 188 | + - The BEK volume contains the [ADE key file](unlock-encrypted-linux-disk-offline-repair.md#identify-the-ade-key-file). |
| 189 | +
|
| 190 | +4. If the ADE key file is missing, create a test VM and encrypt it (volume type data) using the original encryption settings that encrypt the faulty VM. Once encrypted, follow these steps: |
| 191 | + 1. Copy the ADE key file in the BEK volume on the test VM. |
| 192 | + 2. Start the faulty VM. |
| 193 | + 3. In the emergency mode, [identify the ADE key file](unlock-encrypted-linux-disk-offline-repair.md#identify-the-ade-key-file) and [the header file](unlock-encrypted-linux-disk-offline-repair.md#identify-the-header-file). |
| 194 | + 4. Based on the disk layout (LVM or raw), [open the disk from encryption manually](unlock-encrypted-linux-disk-offline-repair.md#unlock-by-files). |
| 195 | + 5. Boot the VM. |
| 196 | + 6. If the ADE key file is still missing and the BEK volume is mounted, manually create a file called `/mnt/azure_bek_disk/LinuxPassPhraseFileName` with the ADE key file contents. |
| 197 | + 7. Reboot the VM. |
| 198 | + 8. Redeploy the VM. |
| 199 | +
|
| 200 | +## Scenario 3: There isn't enough space in the boot partition (Ubuntu) |
| 201 | +
|
| 202 | +> [!NOTE] |
| 203 | +> Starting with [Ubuntu 24](https://azuremarketplace.microsoft.com/marketplace/apps/canonical.ubuntu-24_04-lts?tab=Overview), images come with a separate boot partition of at least 1 GB. |
| 204 | +
|
| 205 | +ADE requires a separate boot partition. During an extension deployment, it creates `/boot` as a separate partition and restores the original files. At the end of this process, a new initial RAM disk file is created. If there's insufficient space, this step fails. This scenario is complex due to many variants. When the OS disk uses ADE, [resizing the OS disk](/azure/virtual-machines/linux/how-to-resize-encrypted-lvm#scenarios) isn't supported. Currently, only Ubuntu images might fall under this boot split process. |
| 206 | +
|
| 207 | +To prevent this issue, perform the following actions: |
| 208 | +
|
| 209 | +1. Delete unused old kernels. |
| 210 | +2. Ensure only the necessary files are under `/boot`. |
| 211 | +
|
| 212 | +## Scenario 4: The VFAT kernel module is disabled |
| 213 | +
|
| 214 | +The VFAT kernel module must be enabled to mount the BEK volume. If not, the ADE key file isn't available, which prevents the disk from being unlocked. To proceed with the encryption, [enable the VFAT module](vfat-disabled-boot-issues.md#ade-encrypted-vm-is-unable-to-access-root-volume). |
| 215 | +
|
| 216 | +## Scenario 5: Required packages fail to install |
| 217 | +
|
| 218 | +The ADE extension installs required packages if they aren't installed by default. If this package installation fails, the encryption fails. |
| 219 | +
|
| 220 | +To determine the cause of package installation failures, follow these steps: |
| 221 | +
|
| 222 | +1. Review the extension logs from the Azure serial console and locate a message like this: |
| 223 | +
|
| 224 | + ```output |
| 225 | + [Info] Installing pre-requisites |
| 226 | + ``` |
| 227 | +
|
| 228 | +2. Ensure all the packages are successfully installed. |
| 229 | +
|
| 230 | + For a full list of the required packages based on the Linux distro, see [Package management](/azure/virtual-machines/linux/disk-encryption-isolated-network#package-management). |
| 231 | +
|
| 232 | +3. If there are issues with the package installation, determine which package fails and the reason. |
| 233 | +
|
| 234 | +4. Ensure the faulty VM has access to the package repositories. |
| 235 | +
|
| 236 | + If the faulty VM has specific network requirements, go to [Azure Disk Encryption on an isolated network](/azure/virtual-machines/linux/disk-encryption-isolated-network). |
| 237 | +
|
| 238 | + For more information about troubleshooting repository issues, see [Troubleshoot common issues in the yum and dnf package management tools for Linux](yum-dnf-common-issues.md) and [Troubleshoot common issues with APT on Ubuntu](apt-common-issues-in-ubuntu.md). |
| 239 | +
|
| 240 | +## Scenario 6: Missing parameters in the GRUB configuration |
| 241 | +
|
| 242 | +During the encryption process, the extension adds several parameters to the kernel options in the `/etc/default/grub` file. These parameters are related to the UUIDs of the boot and root partitions: `rd.luks.ade.partuuid` and `rd.luks.ade.bootuuid`. |
| 243 | +
|
| 244 | +The parameters must be correctly set to the UUIDs. If not, use an [offline troubleshooting](unlock-encrypted-linux-disk-offline-repair.md) method to add the parameters manually. You can obtain the UUIDs in the `chroot` environment using the `blkid` command. |
| 245 | +
|
| 246 | +For more information about regenerating the GRUB file, see [Reinstall GRUB and regenerate the GRUB configuration file manually](troubleshoot-vm-boot-error.md#reinstall-grub-regenerate-grub-configuration-file). |
| 247 | +
|
| 248 | +## Scenario 7: Missing or corrupted osluksheader file |
| 249 | +
|
| 250 | +LUKS stores its encryption metadata in the LUKS header at the beginning of the encrypted partition. This header includes crucial information such as the cipher, mode, hash function, and key slots. The partition is encrypted using a master key. |
| 251 | +
|
| 252 | +When ADE is used in the OS disk, the header is stored in a file named `osluksheader`, located in the boot partition. If this file is corrupted or missing, it can only be restored from a backup. Use an [offline troubleshooting](unlock-encrypted-linux-disk-offline-repair.md) method to mount the boot partition of the affected disk and place the `osluksheader` file from the backup. |
| 253 | +
|
| 254 | +## Resources |
| 255 | +
|
| 256 | +- [Azure Disk Encryption for Linux VMs](/azure/virtual-machines/linux/disk-encryption-overview) |
| 257 | +- [Azure Disk Encryption for Linux VMs troubleshooting guide](/azure/virtual-machines/linux/disk-encryption-troubleshooting) |
| 258 | +- [Azure Disk Encryption for Linux virtual machines FAQ](/azure/virtual-machines/linux/disk-encryption-faq) |
| 259 | +
|
| 260 | +[!INCLUDE [Azure Help Support](../../../includes/azure-help-support.md)] |
0 commit comments