check_disk on openSUSE 13.2: Permission denied on Samba mount

Written by - 0 comments

Published on - last updated on April 24th 2021 - Listed in Monitoring Nagios Icinga Linux Samba


I hit a very strange problem today. After a server running openSUSE 13.2 (yes, a server, don't ask...) was patched and rebooted in the night, our Icinga monitoring started sending alerts for a special mountpoint:

DISK CRITICAL - /special is not accessible: Permission denied

Special in this case, because this path (/special) is a cifs/samba mount:

opensuse:~ # mount | grep /special
//sambaserver.example.local/special on /special type cifs (rw,...)

The permissions on /special were correct (0755), so the "nagios" user has the correct permissions to access the mountpoint. But even as root the check_disk failed:

opensuse:~ # /usr/lib/nagios/plugins/check_disk -w 10% -c 5% -p /special ; date
DISK CRITICAL - /special is not accessible: Permission denied
Thu Dec 10 08:14:04 CET 2015

In systemd's journal I found the following errors:

opensuse:~ # journalctl --system | grep "Dec 10 08:14:04"
Dec 10 08:14:04 opensuse kernel: CIFS VFS: Error -13 sending data on socket to server

At first I thought the problems arose because of the installed patches and a bug was installed. But I was able to run the same check_disk just fine out of /tmp:

opensuse:~ # cp /usr/lib/nagios/plugins/check_disk /tmp/
opensuse:~ # /tmp/check_disk -w 15% -c 5% -p /special
DISK OK - free space: /special 64882412 MB (49% inode=-);| /special=66640360MB;111794356;124946633;0;131522772

I tried several other directories to launch check_disk from and all worked - except the important one: /usr/lib/nagios/plugins.

By grepping for "plugins" in /etc/*, I came across the following result:

/etc/apparmor.d/usr.lib.nagios.plugins.check_disk:2:/usr/lib/nagios/plugins/check_disk {
/etc/apparmor.d/usr.lib.nagios.plugins.check_disk:7:   /usr/lib/nagios/plugins/check_disk rm,

Apparmor!

The file name contains the full path to the executable and looks like this:

opensuse:~ # cat /etc/apparmor.d/usr.lib.nagios.plugins.check_disk
#include
/usr/lib/nagios/plugins/check_disk {
   #include
   #include
   /etc/mtab                        r,
   @{PROC}/[0-9]*/mounts            r,
   /usr/lib/nagios/plugins/check_disk rm,
}

In this apparmor rule, the full path of the executable check_disk is given. So now it made sense, why check_disk didn't work in /usr/lib/nagios/plugins but anywhere else.
For a quick resolution, I disabled this rule by moving it into the disabled folder followed by a restart of apparmor:

opensuse:~ # mv /etc/apparmor.d/usr.lib.nagios.plugins.check_disk /etc/apparmor.d/disable/
opensuse:~ # systemctl restart apparmor

Afterwards check_disk was working again:

opensuse:~ # /usr/lib/nagios/plugins/check_disk -w 10% -c 5% -p /special
DISK OK - free space: /special 64881742 MB (49% inode=-);| /special=66641030MB;118370494;124946633;0;131522772

But why did it work until the system was rebooted?

When the monitoring-plugins package was installed (on December 8th), the apparmor rule files came with it:

opensuse:~ # stat /etc/apparmor.d/usr.lib.nagios.plugins.check_* | egrep "File|Change"
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_dhcp’
Change: 2015-12-08 12:43:54.558360191 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_icmp’
Change: 2015-12-08 12:43:54.837370296 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_ide_smart’
Change: 2015-12-08 12:43:54.928373593 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_load’
Change: 2015-12-08 12:43:54.962374824 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_ntp_time’
Change: 2015-12-08 12:43:55.152381708 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_ping’
Change: 2015-12-08 12:43:55.266385838 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_procs’
Change: 2015-12-08 12:43:55.296386925 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_ssh’
Change: 2015-12-08 12:43:55.388390258 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_swap’
Change: 2015-12-08 12:43:55.439392106 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_users’
Change: 2015-12-08 12:43:55.571396888 +0100

2015-12-08 12:43 is exactly the time, the monitoring-plugins package was installed with zypper. However apparmor was not restarted as this can be seen with journalctl:

opensuse:~ # journalctl -u apparmor.service
[...]
-- Reboot --
Oct 26 09:56:17 opensuse boot.apparmor[798]: Starting AppArmor ..done
Oct 26 10:16:32 opensuse boot.apparmor[2839]: Unloading AppArmor profiles  ..done
-- Reboot --
Oct 26 10:16:50 opensuse boot.apparmor[901]: Starting AppArmor ..done
Dec 09 20:09:19 opensuse boot.apparmor[22683]: Unloading AppArmor profiles  ..done
-- Reboot --
Dec 09 20:12:16 opensuse boot.apparmor[1019]: Starting AppArmor ..done
Dec 10 10:32:22 opensuse boot.apparmor[3016]: Unloading AppArmor profiles  ..done
Dec 10 10:32:23 opensuse boot.apparmor[3032]: Starting AppArmor ..done

Only when the server booted up on December 9th 20:12, apparmor was loaded with the new rules. And since then the check_disk plugin stopped working on /special. 

Instead of just ranting here, I will contact the maintainer of the monitoring-plugins (I installed from zypper repo server:monitoring) and ask for a correction of the apparmor rule. Information about building and maintaining this package (in this case monitoring-plugins-disk) can be found here: Monitoring Plug-Ins on openSUSE Build. According to Request 341451, the apparmor rules were added on August 1 2015:

+-------------------------------------------------------------------
+Sat Aug  1 19:09:11 UTC 2015 - lars@linux-schulserver.de
+
+- add apparmor profiles for the following checks:
+  + check_disk
+  + check_load
+  + check_procs
+  + check_swap
+  + check_users



Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder