Unable to remove LV snapshot: Logical volume in use (by grub-mount process!)

Written by - 0 comments

Published on - Listed in Linux LVM

Taking LVM snapshots is a widely used and handy way to take a quick "backup" of the current state, test something (e.g. upgrade) and revert to the LVM snapshot if something bad happened. Once everything is deemed OK with the original LV, the snapshot can be removed again.

Been there, done that. Probably douzens if not more than a hundred times in my Linux years. But even Seniors like me can run into previously unseen errors.

Can't remove LV snapshot: Logical volume in use

As mentioned I wanted to remove the snapshot of a Logical Volume:

root@debian:~# lvremove -f /dev/vgdata/app-snapshot
  Logical volume vgdata/app-snapshot in use.

The most common reason for this error is that the snapshot is somewhere mounted. But interestingly, that was not the case:

root@debian:~# mount|grep snap
(no output)

That's strange... Maybe the "parent" LV, which is still mounted, causes some weird problem?

root@debian:~# lvdisplay /dev/vgdata/app-snapshot
  --- Logical volume ---
  LV Path                /dev/vgdata/app-snapshot
  LV Name                app-snapshot
  VG Name                vgdata
  LV UUID                EuF2W9-7v62-zB1J-ypGN-mpfF-YhtF-OQRJXe
  LV Write Access        read/write
  LV Creation host, time debian, 2022-09-26 16:52:24 +0200
  LV snapshot status     active destination for myapp
  LV Status              available
  # open                 1
  LV Size                20.00 GiB
  Current LE             5120
  COW-table size         10.00 GiB
  COW-table LE           2560
  Allocated to snapshot  70.13%
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:14

"myapp" in this case represents the parent LV from which the snapshot was taken.

OK so let's stop the LXC container using the parent LV and fully unmount the parent LV:

root@debian:~# lxc-stop -n myapp
root@debian:~# df -h |grep myapp
/dev/mapper/vgdata-myapp ext4              20G  6.1G   13G  33% /var/lib/lxc/myapp/rootfs
root@debian:~# umount /var/lib/lxc/myapp/rootfs

This should resolve any possible "lock" on the snapshot, right? That's what I thought - but I was wrong:

root@debian:~# lvremove -f /dev/vgdata/app-snapshot
  Logical volume vgdata/app-snapshot in use.

The snapshot still refuses to be deleted.

findmnt reveals a weird mount entry from grub-mount

After having tested out a hint from Rob Hallam where dmsetup could be used to directly delete the DM device behind the Logical Volume (but didn't work either), I was still stuck with the same problem: The system still believes that the snapshot is in use.

But so far all possible proof would show otherwise:

  • mount shows no mount entries for the snapshot (and not even for the parent LV anymore)
  • df shows no snapshot
  • lsof ran for an eternity and didn't show the snapshot either (lots of uid errors due to unprivileged lxc containers)
  • ps and grep combination - nada

Eventually my search engine paths made me come across a Serverfault answer, which mentions to use findmnt | grep <lv-name>. It also mentions something very interesting (which at first did not catch my eye):

It turns out that grub runs a program called os-prober it's purpose is to add OS's to the Grub menu, so what I think is happening is it's seeing my snapshot in /dev/mapper and mounting the device to ensure it's a mountable filesystem and something is happening to keep it from unmounting the filesystem.

By looking at the full output of findmnt I was shocked to see, indeed, the grub-mount entry at the very bottom:

root@debian:~# findmnt
TARGET                               SOURCE                   FSTYPE     OPTIONS
/                                    /dev/sda1                ext4       rw,relatime,errors=remount-ro
|-/sys                               sysfs                    sysfs      rw,nosuid,nodev,noexec,relatime
|-/tmp                               /dev/mapper/vgdata-lvtmp ext4       rw,relatime
`-/var                               /dev/mapper/vgdata-lvvar ext4       rw,relatime
  |-/var/lib/lxc/fileserver/rootfs  /dev/mapper/fileserver
  |                                                           ext4       rw,relatime
  |-/var/lib/lxcfs                   lxcfs                    fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0
  `-/var/lib/os-prober/mount         grub-mount               fuse.grub- rw,nosuid,nodev,relatime,user_id=0,group_id=0

Is there really a process still hanging on to something?

root@debian:~# ps aux|grep grub-mount
root     1662497  0.0  0.0  11476  2640 ?        Ss    2023   0:00 grub-mount /dev/mapper/vgdata-app--snapshot /var/lib/os-prober/mount
root     1995845  0.0  0.0   6348   712 pts/9    S+   17:03   0:00 grep --color=auto grub-mount

Wow. Just... wow! There has indeed been a process (grub-mount) which mounted the snapshot and has never released it ever since!

Let's kill the process and verify the mount:

root@debian:~# kill -9 1662497
root@debian:~# findmnt | tail -n 1
  `-/var/lib/os-prober/mount         grub-mount               fuse.grub- rw,nosuid,nodev,relatime,user_id=0,group_id=0

So this entry is still here (maybe I should not have used -9, my bad), let's manually unmount it:

root@debian:~# umount /var/lib/os-prober/mount

The grub-mount entry has now disappeared from findmnt's output.

Let's try if that would solve it:

root@debian:~# lvremove -f /dev/vgdata/app-snapshot
  Logical volume "app-snapshot" successfully removed

Hurraaayy! The snapshot could finally be deleted!

As it turns out, it was really the grub-mount process (which is triggered by os-prober after a package update of grub2) holding on to that snapshot and never unmounting it. This prevents the snapshot from being deleted.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Icingaweb   Icingaweb2   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder   

Update cookies preferences