Rsnapshot error: /bin/rm: cannot remove ... Directory not empty

Written by Claudio Kuenzler - 0 comments

Published on May 4th 2018 - Listed in Linux

For the last two days I got the following error message from the nightly rsnapshot run:

/bin/rm: cannot remove ‘/backup/rsnapshot/daily.14/
p01/mnt/containerdata/admin/[...]/Auth’:
Directory not empty
----------------------------------------------------------------------------
rsnapshot encountered an error! The program was invoked with these options:
/usr/bin/rsnapshot daily
----------------------------------------------------------------------------
ERROR: Warning! /bin/rm failed.
ERROR: Error! rm_rf("/backup/rsnapshot/daily.14/")

But even when I manually tried to delete the folder, it didn't work:

# rm -rfv /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth
rm: cannot remove ‘/backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth’: Directory not empty

Why would that happen? Let's check out the contents of that particular path:

# la -la /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth
ls: /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth: Input/output error
ls: cannot open directory /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth: Input/output error

 Oops, IO error! That doesn't sound too good! I unmounted the /backup path and ran a forced file system check:

# fsck.ext4 -f /dev/vgdata/lvbackup
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Inode 2900778 has a bad extended attribute block 524288.  Clear<y>? yes
Inode 12765722 has a bad extended attribute block 524288.  Clear<y>? yes
Inode 42844666 has a bad extended attribute block 524288.  Clear<y>? yes
Pass 2: Checking directory structure
Entry '.' in /rsnapshot/daily.5/p01/mnt/containerdata/[...] (89989522) is a link to '.'  Clear<y>? yes
Entry '..' in /rsnapshot/daily.5/p01/mnt/containerdata/[...] (89989522) is duplicate '..' entry.
Fix<y>? yes
Entry '.' in /rsnapshot/daily.7/p01/mnt/containerdata/[...] (71123012) is a link to '.'  Clear<y>? yes
Entry '..' in /rsnapshot/daily.7/p01/mnt/containerdata/[...] (71123012) is duplicate '..' entry.
Fix<y>? yes
Entry '.' in /rsnapshot/daily.8/p01/mnt/containerdata/[...] (70746560) is a link to '.'  Clear<y>? yes
Entry '..' in /rsnapshot/daily.8/p01/mnt/containerdata/[...] (70746560) is duplicate '..' entry.
Fix<y>? yes
Pass 3: Checking directory connectivity
'..' in /rsnapshot/daily.8/p01/mnt/containerdata/[...] (70746560) is <The NULL inode> (0), should be /rsnapshot/daily.8/p01/mnt/containerdata/[...] (70746549).
Fix<y>? yes
'..' in /rsnapshot/daily.7/p01/mnt/containerdata/[...] (71123012) is <The NULL inode> (0), should be /rsnapshot/daily.7/p01/mnt/containerdata/[...] (71123004).
Fix<y>? yes
'..' in /rsnapshot/daily.5/p01/mnt/containerdata/[...] (89989522) is <The NULL inode> (0), should be /rsnapshot/daily.5/p01/mnt/containerdata/[...] (89989521).
Fix<y>? yes
Pass 4: Checking reference counts
Inode 70746549 ref count is 51, should be 50.  Fix<y>? yes
Inode 70746560 ref count is 2, should be 1.  Fix<y>? yes
Inode 71123004 ref count is 7, should be 6.  Fix<y>? yes
Inode 71123012 ref count is 2, should be 1.  Fix<y>? yes
Inode 89989521 ref count is 6, should be 5.  Fix<y>? yes
Inode 89989522 ref count is 2, should be 1.  Fix<y>? yes
Pass 5: Checking group summary information

/dev/vgdata/lvbackup: ***** FILE SYSTEM WAS MODIFIED *****
/dev/vgdata/lvbackup: 11160908/170393600 files (0.2% non-contiguous), 610481642/681574400 blocks

 Note: To protect the privacy of the customer's data, I removed the full paths and directory names from the output.

Once the file system check passed, I was able to check the directory:

# ls -la /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth/
total 52
drwxr-xr-x  3 1002 1002 4096 Dec  2 13:04 .
drwxr-xr-x  3 1002 1002 4096 May  3 02:05 ..
drwxr-xr-x  2 1002 1002 4096 Dec  2 13:05 Access
[...]

And of course rm was now working, too:

# rm -rfv /backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth
removed directory: ‘/backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth/Access’
[...]
removed directory: ‘/backup/rsnapshot/daily.14/p01/mnt/containerdata/admin/[...]/Auth’

TL;DR: When rsnapshot fails with such an error, it's worth to check out the file system's health.

Add a comment

Comments (newest first)

No comments yet.