Rsnapshot error: /bin/rm: cannot remove ... Directory not empty

Written by - 0 comments

Published on - Listed in Linux


For the last two days I got the following error message from the nightly rsnapshot run:

/bin/rm: cannot remove ‘/backup/rsnapshot/daily.14/
p01/mnt/containerdata/admin/[...]/Auth’:
Directory not empty
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot daily
----------------------------------------------------------------------------
ERROR: Warning! /bin/rm failed.
ERROR: Error! rm_rf("/backup/rsnapshot/daily.14/")

But even when I manually tried to delete the folder, it didn't work:

# rm -rfv /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth
rm: cannot remove ‘/backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth’: Directory not empty

Why would that happen? Let's check out the contents of that particular path:

# la -la /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth
ls: /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth: Input/output error
ls: cannot open directory /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth: Input/output error

 Oops, IO error! That doesn't sound too good! I unmounted the /backup path and ran a forced file system check:

# fsck.ext4 -f /dev/vgdata/lvbackup
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Inode 2900778 has a bad extended attribute block 524288.  Clear<y>? yes
Inode 12765722 has a bad extended attribute block 524288.  Clear<y>? yes
Inode 42844666 has a bad extended attribute block 524288.  Clear<y>? yes
Pass 2: Checking directory structure
Entry '.' in /rsnapshot/daily.5/p01/mnt/containerdata/[...] (89989522) is a link to '.'  Clear<y>? yes
Entry '..' in /rsnapshot/daily.5/p01/mnt/containerdata/[...] (89989522) is duplicate '..' entry.
Fix<y>? yes
Entry '.' in /rsnapshot/daily.7/p01/mnt/containerdata/[...] (71123012) is a link to '.'  Clear<y>? yes
Entry '..' in /rsnapshot/daily.7/p01/mnt/containerdata/[...] (71123012) is duplicate '..' entry.
Fix<y>? yes
Entry '.' in /rsnapshot/daily.8/p01/mnt/containerdata/[...] (70746560) is a link to '.'  Clear<y>? yes
Entry '..' in /rsnapshot/daily.8/p01/mnt/containerdata/[...] (70746560) is duplicate '..' entry.
Fix<y>? yes
Pass 3: Checking directory connectivity
'..' in /rsnapshot/daily.8/p01/mnt/containerdata/[...] (70746560) is <The NULL inode> (0), should be /rsnapshot/daily.8/p01/mnt/containerdata/[...] (70746549).
Fix<y>? yes
'..' in /rsnapshot/daily.7/p01/mnt/containerdata/[...] (71123012) is <The NULL inode> (0), should be /rsnapshot/daily.7/p01/mnt/containerdata/[...] (71123004).
Fix<y>? yes
'..' in /rsnapshot/daily.5/p01/mnt/containerdata/[...] (89989522) is <The NULL inode> (0), should be /rsnapshot/daily.5/p01/mnt/containerdata/[...] (89989521).
Fix<y>? yes
Pass 4: Checking reference counts
Inode 70746549 ref count is 51, should be 50.  Fix<y>? yes
Inode 70746560 ref count is 2, should be 1.  Fix<y>? yes
Inode 71123004 ref count is 7, should be 6.  Fix<y>? yes
Inode 71123012 ref count is 2, should be 1.  Fix<y>? yes
Inode 89989521 ref count is 6, should be 5.  Fix<y>? yes
Inode 89989522 ref count is 2, should be 1.  Fix<y>? yes
Pass 5: Checking group summary information

/dev/vgdata/lvbackup: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vgdata/lvbackup: 11160908/170393600 files (0.2% non-contiguous), 610481642/681574400 blocks

 Note: To protect the privacy of the customer's data, I removed the full paths and directory names from the output.

Once the file system check passed, I was able to check the directory:

# ls -la /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth/
total 52
drwxr-xr-x  3 1002 1002 4096 Dec  2 13:04 .
drwxr-xr-x  3 1002 1002 4096 May  3 02:05 ..
drwxr-xr-x  2 1002 1002 4096 Dec  2 13:05 Access
[...]

And of course rm was now working, too:

# rm -rfv /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth
removed directory: ‘/backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth/Access’
[...]
removed directory: ‘/backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth’

TL;DR: When rsnapshot fails with such an error, it's worth to check out the file system's health.


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder