Taking LVM snapshots is a widely used and handy way to take a quick "backup" of the current state, test something (e.g. upgrade) and revert to the LVM snapshot if something bad happened. Once everything is deemed OK with the original LV, the snapshot can be removed again.
Been there, done that. Probably douzens if not more than a hundred times in my Linux years. But even Seniors like me can run into previously unseen errors.
As mentioned I wanted to remove the snapshot of a Logical Volume:
root@debian:~# lvremove -f /dev/vgdata/app-snapshot
Logical volume vgdata/app-snapshot in use.
The most common reason for this error is that the snapshot is somewhere mounted. But interestingly, that was not the case:
root@debian:~# mount|grep snap
(no output)
That's strange... Maybe the "parent" LV, which is still mounted, causes some weird problem?
root@debian:~# lvdisplay /dev/vgdata/app-snapshot
--- Logical volume ---
LV Path /dev/vgdata/app-snapshot
LV Name app-snapshot
VG Name vgdata
LV UUID EuF2W9-7v62-zB1J-ypGN-mpfF-YhtF-OQRJXe
LV Write Access read/write
LV Creation host, time debian, 2022-09-26 16:52:24 +0200
LV snapshot status active destination for myapp
LV Status available
# open 1
LV Size 20.00 GiB
Current LE 5120
COW-table size 10.00 GiB
COW-table LE 2560
Allocated to snapshot 70.13%
Snapshot chunk size 4.00 KiB
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 254:14
"myapp" in this case represents the parent LV from which the snapshot was taken.
OK so let's stop the LXC container using the parent LV and fully unmount the parent LV:
root@debian:~# lxc-stop -n myapp
root@debian:~# df -h |grep myapp
/dev/mapper/vgdata-myapp ext4 20G 6.1G 13G 33% /var/lib/lxc/myapp/rootfs
root@debian:~# umount /var/lib/lxc/myapp/rootfs
This should resolve any possible "lock" on the snapshot, right? That's what I thought - but I was wrong:
root@debian:~# lvremove -f /dev/vgdata/app-snapshot
Logical volume vgdata/app-snapshot in use.
The snapshot still refuses to be deleted.
After having tested out a hint from Rob Hallam where dmsetup could be used to directly delete the DM device behind the Logical Volume (but didn't work either), I was still stuck with the same problem: The system still believes that the snapshot is in use.
But so far all possible proof would show otherwise:
Eventually my search engine paths made me come across a Serverfault answer, which mentions to use findmnt | grep <lv-name>. It also mentions something very interesting (which at first did not catch my eye):
It turns out that grub runs a program called os-prober it's purpose is to add OS's to the Grub menu, so what I think is happening is it's seeing my snapshot in /dev/mapper and mounting the device to ensure it's a mountable filesystem and something is happening to keep it from unmounting the filesystem.
By looking at the full output of findmnt I was shocked to see, indeed, the grub-mount entry at the very bottom:
root@debian:~# findmnt
TARGET SOURCE FSTYPE OPTIONS
/ /dev/sda1 ext4 rw,relatime,errors=remount-ro
|-/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime
[...]
|-/tmp /dev/mapper/vgdata-lvtmp ext4 rw,relatime
`-/var /dev/mapper/vgdata-lvvar ext4 rw,relatime
|-/var/lib/lxc/fileserver/rootfs /dev/mapper/fileserver
| ext4 rw,relatime
[...]
|-/var/lib/lxcfs lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0
`-/var/lib/os-prober/mount grub-mount fuse.grub- rw,nosuid,nodev,relatime,user_id=0,group_id=0
Is there really a process still hanging on to something?
root@debian:~# ps aux|grep grub-mount
root 1662497 0.0 0.0 11476 2640 ? Ss 2023 0:00 grub-mount /dev/mapper/vgdata-app--snapshot /var/lib/os-prober/mount
root 1995845 0.0 0.0 6348 712 pts/9 S+ 17:03 0:00 grep --color=auto grub-mount
Wow. Just... wow! There has indeed been a process (grub-mount) which mounted the snapshot and has never released it ever since!
Let's kill the process and verify the mount:
root@debian:~# kill -9 1662497
root@debian:~# findmnt | tail -n 1
`-/var/lib/os-prober/mount grub-mount fuse.grub- rw,nosuid,nodev,relatime,user_id=0,group_id=0
So this entry is still here (maybe I should not have used -9, my bad), let's manually unmount it:
root@debian:~# umount /var/lib/os-prober/mount
The grub-mount entry has now disappeared from findmnt's output.
Let's try if that would solve it:
root@debian:~# lvremove -f /dev/vgdata/app-snapshot
Logical volume "app-snapshot" successfully removed
Hurraaayy! The snapshot could finally be deleted!
As it turns out, it was really the grub-mount process (which is triggered by os-prober after a package update of grub2) holding on to that snapshot and never unmounting it. This prevents the snapshot from being deleted.
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb Icingaweb2 Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder