Debian not booting: ALERT /dev/disk/by-uuid does not exist

Written by - 0 comments

Published on - Listed in Linux Proxmox Virtualization


On a test server running a Proxmox PVE installation (based on Debian Wheezy) I launched a apt-get dist-upgrade which installed a new kernel and launched grub-update.
In the output of dist-upgrade I saw some errors but didn't look closer at it and immediately rebooted the server. Mistake.

Once the grub menu appeared and the first entry was auto-selected, grub couldn't find the root disk and alerted with the following error message on the console:

ALERT! /dev/disk/by-uuid/5b9f271d-3e7a-49cf-b892-2c6c8e194684 does not exist.
Dropping to a shell!

Here the screenshot:

ALERT disk does not exist

Argh. I have already seen such cases (see my article grub2 boot issues after hdd replacement) where grub2 didn't boot anymore.

I launched the Knoppix ISO from the virtual DVD drive (with boot option knoppix64) to see what's going on.

Mounted /dev/sda1 (the /boot partition of the Proxmox installation) as /mnt and verified some grub stuff:

mount /dev/sda1 /mnt

cat /mnt/grub/device.map
(hd0) /dev/sda

cat /mnt/grub/grub.cfg
[...]
menuentry 'Proxmox Virtual Environment GNU/Linux, with Linux 2.6.32-23-pve' --class proxmox --class gnu-linux --class gnu --class os {
[...]
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 29efb07c-be6e-44dc-b637-70efda447bd7
linux /vmlinuz-2.6.32-23-pve root=UUID=5b9f271d-3e7a-49cf-b892-2c6c8e194684 ro quiet
[...]
}

Let's check out the UUID of the root partition which is a Logical Volume. First activate the volume group pve:

vgchange -ay pve
  4 logical volume(s) in volume group "pve" now active

Where does the root LV point to?

ls -l /dev/mapper | grep pve-root
lrwxrwxrwx 1 root root 7 Feb  2 09:06 pve-root -> ../dm-1

Check out UUID of /dev/dm-1:

ls -l /dev/disk/by-uuid | grep dm-1
lrwxrwxrwx 1 root root 10 Feb  2 09:06 5b9f271d-3e7a-49cf-b892-2c6c8e194684

Hmm... that's the correct UUID also used in the grub config. So we need to find other errors by launching update-grub within the "real" Proxmox OS (Debian Wheezy). To do that, the root and runtime filesystems need to be mounted:

umount /mnt

mount /dev/mapper/pve-root /mnt
mount /dev/sda1 /mnt/boot
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys

Change root (get into the "real" Linux environment):

chroot /mnt

Launch update-grub command:

update-grub
Generating grub.cfg ..
/usr/sbin/grub-probe: error: Couldn't find PV pv1. Check your device.map.
Found linux image: /boot/vmlinuz-2.6.32-23-pve
Found initrd image: /boot/initrd.img-2.6.32-23-pve
/usr/sbin/grub-probe: error: Couldn't find PV pv1. Check your device.map.
[...]
done

pvdisplay didn't show anything. Activated VG pve within chroot environment again:

vgchange -ay pve
  4 logical volume(s) in volume group "pve" now active

pvdisplay
--- Physical volume ---
PV Name        /dev/sda2
VG Name        pve
[...]
PV UUID        1YMsls-ycW4-aqWL-MEsC-EOFM-VVyI-4UrgFY
[...]

Launch update-grub again:

update-grub
Generating grub.cfg ..
Found linux image: /boot/vmlinuz-2.6.32-23-pve
Found initrd image: /boot/initrd.img-2.6.32-23-pve
[...]
done

So this time it worked without an error message. Let's check the contents of the device.map and grub.cfg:

cat /boot/grub/device.map
(fd0) /dev/fd0
(hd0) /dev/sda
(hd1) /dev/sdb

cat /boot/grub/grub.cfg
[...]
menuentry 'Proxmox Virtual Environment GNU/Linux, with Linux 2.6.32-23-pve' --class proxmox --class gnu-linux --class gnu --class os {
[...]
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 29efb07c-be6e-44dc-b637-70efda447bd7
linux /vmlinuz-2.6.32-23-pve root=/dev/mapper/pve-root ro quiet
[...]
}

Well well well... device.map has changed a lot and so did the root path of the linux entry in grub.cfg.

Let's reboot and see... Hurray! Successfully booted:

Boot fixed


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder