Skip to content

Commit c7cda97

Browse files
megvanhuygenmegvanhuygen
authored
Merge pull request #114067 from v-miegge/v-miegge/reinstate-fstab-file
V miegge/reinstate fstab file
2 parents bd63092 + df476d9 commit c7cda97

File tree

4 files changed

+243
-9
lines changed

4 files changed

+243
-9
lines changed

‎.openpublishing.redirection.json

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -52343,11 +52343,6 @@
5234352343
"source_path": "articles/load-balancer/azure-media-player/components-limitations.md",
5234452344
"redirect_url": "/articles/load-balancer/concepts",
5234552345
"redirect_document_id": false
52346-
},
52347-
{
52348-
"source_path": "articles/virtual-machines/troubleshooting/linux-virtual-machine-cannot-start-fstab-errors.md",
52349-
"redirect_url": "/articles/virtual-machines/troubleshooting",
52350-
"redirect_document_id": false
5235152346
}
5235252347

5235352348
]

‎articles/virtual-machines/linux/toc.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -702,8 +702,6 @@
702702
href: disks-shared-enable.md
703703
- name: Upload a vhd to a disk - CLI
704704
href: disks-upload-vhd-to-managed-disk-cli.md
705-
- name: Resize OS disk with a GPT partition
706-
href: resize-os-disk-gpt-partition.md
707705
- name: Resize a disk
708706
href: expand-disks.md
709707
- name: Use Storage Explorer to manage disks

‎articles/virtual-machines/troubleshooting/linux-virtual-machine-cannot-start-fstab-errors.md

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
---
2+
title: Troubleshoot Linux VM starting issues due to fstab errors | Microsoft Docs
3+
description: Explains why Linux VM cannot start and how to solve the problem.
4+
services: virtual-machines-linux
5+
documentationcenter: ''
6+
author: v-miegge
7+
manager: dcscontentpm
8+
editor: ''
9+
tags: ''
10+
11+
ms.service: virtual-machines-linux
12+
ms.topic: troubleshooting
13+
ms.workload: infrastructure-services
14+
ms.tgt_pltfrm: vm-linux
15+
ms.devlang: azurecli
16+
ms.date: 10/09/2019
17+
ms.author: v-six
18+
19+
---
20+
21+
# Troubleshoot Linux VM starting issues due to fstab errors
22+
23+
You can't connect to an Azure Linux Virtual Machine (VM) by using a Secure Shell (SSH) connection. When you run the [Boot Diagnostics](https://docs.microsoft.com/azure/virtual-machines/troubleshooting/boot-diagnostics) feature on the [Azure portal](https://portal.azure.com/), you see log entries that resemble the following examples:
24+
25+
## Examples
26+
27+
The following are examples of possible errors.
28+
29+
### Example 1: A disk is mounted by the SCSI ID instead of the universally unique identifier (UUID)
30+
31+
```
32+
[K[[1;31m TIME [0m] Timed out waiting for device dev-incorrect.device.
33+
[[1;33mDEPEND[0m] Dependency failed for /data.
34+
[[1;33mDEPEND[0m] Dependency failed for Local File Systems.
35+
…
36+
Welcome to emergency mode! After logging in, type "journalctl -xb" to viewsystem logs, "systemctl reboot" to reboot, "systemctl default" to try again to boot into default mode.
37+
Give root password for maintenance
38+
(or type Control-D to continue)
39+
```
40+
41+
### Example 2: An unattached device is missing on CentOS
42+
43+
```
44+
Checking file systems…
45+
fsck from util-linux 2.19.1
46+
Checking all file systems.
47+
/dev/sdc1: nonexistent device ("nofail" fstab option may be used to skip this device)
48+
/dev/sdd1: nonexistent device ("nofail" fstab option may be used to skip this device)
49+
/dev/sde1: nonexistent device ("nofail" fstab option may be used to skip this device)
50+
51+
[/sbin/fsck.ext3 (1) — /CODE] sck.ext3 -a /dev/sdc1
52+
fsck.ext3: No such file or directory while trying to open /dev/sdc1
53+
54+
/dev/sdc1:
55+
The superblock could not be read or does not describe a correct ext2
56+
filesystem. If the device is valid and it really contains an ext2
57+
filesystem (and not swap or ufs or something else), then the superblock
58+
is corrupt, and you might try running e2fsck with an alternate superblock:
59+
60+
e2fsck -b 8193 <device>
61+
62+
[/sbin/fsck.xfs (1) — /GLUSTERDISK] fsck.xfs -a /dev/sdd1
63+
/sbin/fsck.xfs: /dev/sdd1 does not exist
64+
[/sbin/fsck.ext3 (1) — /DATATEMP] fsck.ext3 -a /dev/sde1 fsck.ext3: No such file or directory while trying to open /dev/sde1
65+
```
66+
67+
### Example 3: A VM cannot start because of an fstab misconfiguration or because the disk is no longer attached
68+
69+
```
70+
The disk drive for /var/lib/mysql is not ready yet or not present.
71+
Continue to wait, or Press S to skip mounting or M for manual recovery
72+
```
73+
74+
### Example 4: A serial log entry shows an incorrect UUID
75+
76+
```
77+
Checking filesystems
78+
Checking all file systems.
79+
[/sbin/fsck.ext4 (1) — /] fsck.ext4 -a /dev/sda1
80+
/dev/sda1: clean, 70442/1905008 files, 800094/7608064 blocks
81+
[/sbin/fsck.ext4 (1) — /datadrive] fsck.ext4 -a UUID="<UUID>"
82+
fsck.ext4: Unable to resolve UUID="<UUID>"
83+
[FAILED
84+
85+
*** An error occurred during the file system check.
86+
*** Dropping you to a shell; the system will reboot
87+
*** when you leave the shell.
88+
*** Warning — SELinux is active
89+
*** Disabling security enforcement for system recovery.
90+
*** Run 'setenforce 1' to reenable.
91+
type=1404 audit(1428047455.949:4): enforcing=0 old_enforcing=1 auid=<AUID> ses=4294967295
92+
Give root password for maintenance
93+
(or type Control-D to continue)
94+
```
95+
96+
This problem may occur if the file systems table (fstab) syntax is incorrect or if a required data disk that is mapped to an entry in the "/etc/fstab" file is not attached to the VM.
97+
98+
## Resolution
99+
100+
To resolve this problem, start the VM in emergency mode by using the serial console for Azure Virtual Machines. Then use the tool to repair the file system. If the serial console isn't enabled on your VM, go to the [Repair the VM offline](#repair-the-vm-offline) section.
101+
102+
## Use the serial console
103+
104+
### Using Single User Mode
105+
106+
1. Connect to [the serial console](https://docs.microsoft.com/azure/virtual-machines/troubleshooting/serial-console-linux).
107+
2. Use serial console to take single user mode [single user mode](https://docs.microsoft.com/azure/virtual-machines/linux/serial-console-grub-single-user-mode)
108+
3. Once the vm has booted into single user mode. Use your favorite text editor to open the fstab file.
109+
110+
```
111+
# nano /etc/fstab
112+
```
113+
114+
4. Review the listed file systems. Each line in the fstab file indicates a file system that is mounted when the VM starts. For more information about the syntax of the fstab file, run the man fstab command. To troubleshoot a start failure, review each line to make sure that it's correct in both structure and content.
115+
116+
> [!Note]
117+
> * Fields on each line are separated by tabs or spaces. Blank lines are ignored. Lines that have a number sign (#) as the first character are comments. Commented lines can remain in the fstab file, but they won't be processed. We recommend that you comment fstab lines that you're unsure about instead of removing the lines.
118+
> * For the VM to recover and start, the file system partitions should be the only required partitions. The VM may experience application errors about additional commented partitions. However, the VM should start without the additional partitions. You can later uncomment any commented lines.
119+
> * We recommend that you mount data disks on Azure VMs by using the UUID of the file system partition. For example, run the following command: ``/dev/sdc1: LABEL="cloudimg-rootfs" UUID="<UUID>" TYPE="ext4" PARTUUID="<PartUUID>"``
120+
> * To determine the UUID of the file system, run the blkid command. For more information about the syntax, run the man blkid command.
121+
> * The nofail option helps make sure that the VM starts even if the file system is corrupted or the file system doesn't exist at startup. We recommend that you use the nofail option in the fstab file to enable startup to continue after errors occur in partitions that are not required for the VM to start.
122+
123+
5. Change or comment out any incorrect or unnecessary lines in the fstab file to enable the VM to start correctly.
124+
125+
6. Save the changes to the fstab file.
126+
127+
7. Reboot the vm using the below command.
128+
129+
```
130+
# reboot -f
131+
```
132+
> [!Note]
133+
> You can also use "ctrl+x" command which would also reboot the vm.
134+
135+
136+
8. If the entries comment or fix was successful, the system should reach a bash prompt in the portal. Check whether you can connect to the VM.
137+
138+
### Using Root Password
139+
140+
1. Connect to [the serial console](https://docs.microsoft.com/azure/virtual-machines/troubleshooting/serial-console-linux).
141+
2. Sign-in to the system by using a local user and password.
142+
143+
> [!Note]
144+
> You can't use an SSH key to sign in to the system in the serial console.
145+
146+
3. Look for the error that indicates that the disk wasn't mounted. In the following example, the system was trying to attach a disk that was no longer present:
147+
148+
```
149+
[DEPEND] Dependency failed for /datadisk1.
150+
[DEPEND] Dependency failed for Local File Systems.
151+
[DEPEND] Dependency failed for Relabel all filesystems, if necessary.
152+
[DEPEND] Dependency failed for Migrate local... structure to the new structure.
153+
Welcome to emergency mode! After logging in, type "journalctl -xb" to view
154+
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to try again to boot into default mode.
155+
Give root password for maintenance
156+
(or type Control-D to continue):
157+
```
158+
159+
4. Connect to the VM by using the root password (Red Hat-based VMs).
160+
161+
5. Use your favorite text editor to open the fstab file. After the disk is mounted, run the following command for Nano:
162+
163+
```
164+
$ nano /mnt/troubleshootingdisk/etc/fstab
165+
```
166+
167+
6. Review the listed file systems. Each line in the fstab file indicates a file system that is mounted when the VM starts. For more information about the syntax of the fstab file, run the man fstab command. To troubleshoot a start failure, review each line to make sure that it's correct in both structure and content.
168+
169+
> [!Note]
170+
> * Fields on each line are separated by tabs or spaces. Blank lines are ignored. Lines that have a number sign (#) as the first character are comments. Commented lines can remain in the fstab file, but they won't be processed. We recommend that you comment fstab lines that you're unsure about instead of removing the lines.
171+
> * For the VM to recover and start, the file system partitions should be the only required partitions. The VM may experience application errors about additional commented partitions. However, the VM should start without the additional partitions. You can later uncomment any commented lines.
172+
> * We recommend that you mount data disks on Azure VMs by using the UUID of the file system partition. For example, run the following command: ``/dev/sdc1: LABEL="cloudimg-rootfs" UUID="<UUID>" TYPE="ext4" PARTUUID="<PartUUID>"``
173+
> * To determine the UUID of the file system, run the blkid command. For more information about the syntax, run the man blkid command.
174+
> * The nofail option helps make sure that the VM starts even if the file system is corrupted or the file system doesn't exist at startup. We recommend that you use the nofail option in the fstab file to enable startup to continue after errors occur in partitions that are not required for the VM to start.
175+
176+
7. Change or comment out any incorrect or unnecessary lines in the fstab file to enable the VM to start correctly.
177+
178+
8. Save the changes to the fstab file.
179+
180+
9. Restart the virtual machine.
181+
182+
10. If the entries comment or fix was successful, the system should reach a bash prompt in the portal. Check whether you can connect to the VM.
183+
184+
11. Check your mount points when you test any fstab change by running the mount –a command. If there are no errors, your mount points should be good.
185+
186+
## Repair the VM offline
187+
188+
1. Attach the system disk of the VM as a data disk to a recovery VM (any working Linux VM). To do this, you can use [CLI commands](https://docs.microsoft.com/azure/virtual-machines/troubleshooting/troubleshoot-recovery-disks-linux) or you can automate setting up the recovery VM using the [VM repair commands](repair-linux-vm-using-azure-virtual-machine-repair-commands.md).
189+
190+
2. After you mount the system disk as a data disk on the recovery VM, back up the fstab file before you make changes, and then follow the next steps to correct the fstab file.
191+
192+
3. Look for the error that indicates the disk wasn't mounted. In the following example, the system was trying to attach a disk that was no longer present:
193+
194+
```
195+
[DEPEND] Dependency failed for /datadisk1.
196+
[DEPEND] Dependency failed for Local File Systems.
197+
[DEPEND] Dependency failed for Relabel all filesystems, if necessary.
198+
[DEPEND] Dependency failed for Migrate local... structure to the new structure.
199+
Welcome to emergency mode! After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to try again to boot into default mode.
200+
Give root password for maintenance (or type Control-D to continue):
201+
```
202+
203+
4. Connect to the VM by using the root password (Red Hat-based VMs).
204+
205+
5. Use your favorite text editor to open the fstab file. After the disk is mounted, run the following command for Nano. Make sure that you're working on the fstab file that is located on the mounted disk and not the fstab file that's on the rescue VM.
206+
207+
```
208+
$ nano /mnt/troubleshootingdisk/etc/fstab
209+
```
210+
211+
6. Review the listed file systems. Each line in the fstab file indicates a file system that is mounted when the VM starts. For more information about the syntax of the fstab file, run the man fstab command. To troubleshoot a start failure, review each line to make sure that it's correct in both structure and content.
212+
213+
> [!Note]
214+
> * Fields on each line are separated by tabs or spaces. Blank lines are ignored. Lines that have a number sign (#) as the first character are comments. Commented lines can remain in the fstab file, but they won't be processed. We recommend that you comment fstab lines that you're unsure about instead of removing the lines.
215+
> * For the VM to recover and start, the file system partitions should be the only required partitions. The VM may experience application errors about additional commented partitions. However, the VM should start without the additional partitions. You can later uncomment any commented lines.
216+
> * We recommend that you mount data disks on Azure VMs by using the UUID of the file system partition. For example, run the following command: ``/dev/sdc1: LABEL="cloudimg-rootfs" UUID="<UUID>" TYPE="ext4" PARTUUID="<PartUUID>"``
217+
> * To determine the UUID of the file system, run the blkid command. For more information about the syntax, run the man blkid command. Notice that the disk that you want to recover is now mounted on a new VM. Although the UUIDs should be consistent, the device partition IDs (for example, "/dev/sda1") are different on this VM. The file system partitions of the original failing VM that are located on a non-system VHD are not available to the recovery VM [using CLI commands](https://docs.microsoft.com/azure/virtual-machines/troubleshooting/troubleshoot-recovery-disks-linux).
218+
> * The nofail option helps make sure that the VM starts even if the file system is corrupted or the file system doesn't exist at startup. We recommend that you use the nofail option in the fstab file to enable startup to continue after errors occur in partitions that are not required for the VM to start.
219+
220+
7. Change or comment out any incorrect or unnecessary lines in the fstab file to enable the VM to start correctly.
221+
222+
8. Save the changes to the fstab file.
223+
224+
9. Restart the virtual machine or rebuild the original VM.
225+
226+
10. If the entries comment or fix was successful, the system should reach a bash prompt in the portal. Check whether you can connect to the VM.
227+
228+
11. Check your mount points when you test any fstab change by running the mount –a command. If there are no errors, your mount points should be good.
229+
230+
12. Unmount and detach the original virtual hard disk, and then create a VM from the original system disk. To do this, you can use [CLI commands](troubleshoot-recovery-disks-linux.md) or the [VM repair commands](repair-linux-vm-using-azure-virtual-machine-repair-commands.md) if you used them to create the recovery VM.
231+
232+
13. After you create the VM again and you can connect to it through SSH, take the following actions:
233+
* Review any of the fstab lines that were changed or commented out during the recovery.
234+
* Make sure that you're using UUID and the nofail option appropriately.
235+
* Test any fstab changes before you restart the VM. To do this, use the following command: ``$ sudo mount -a``
236+
* Create an additional copy of the corrected fstab file for use in future recovery scenarios.
237+
238+
## Next steps
239+
240+
* [Troubleshoot a Linux VM by attaching the OS disk to a recovery VM with the Azure CLI 2.0](https://docs.microsoft.com/azure/virtual-machines/virtual-machines-linux-troubleshoot-recovery-disks)
241+
* [Troubleshoot a Linux VM by attaching the OS disk to a recovery VM using the Azure portal](https://docs.microsoft.com/azure/virtual-machines/linux/troubleshoot-recovery-disks-portal)

‎articles/virtual-machines/troubleshooting/toc.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -143,14 +143,14 @@
143143
items:
144144
- name: SSH troubleshooting
145145
href: troubleshoot-ssh-connection.md
146-
- name: Chroot in Linux Rescue VM
147-
href: chroot-environment-linux.md
148146
- name: Troubleshoot file system errors
149147
href: linux-recovery-cannot-start-file-system-errors.md
150148
- name: Troubleshoot kernel-related boot problems
151149
href: https://support.microsoft.com/help/4091524
152150
- name: Recover LVM configured VMs and no boot
153151
href: chroot-logical-volume-manager.md
152+
- name: Troubleshoot fstab errors
153+
href: linux-virtual-machine-cannot-start-fstab-errors.md
154154
- name: Detailed SSH troubleshooting
155155
href: detailed-troubleshoot-ssh-connection.md
156156
- name: Application access issues

0 commit comments

Comments
 (0)