I hit a very strange problem today. After a server running openSUSE 13.2 (yes, a server, don't ask...) was patched and rebooted in the night, our Icinga monitoring started sending alerts for a special mountpoint:

DISK CRITICAL - /special is not accessible: Permission denied

Special in this case, because this path (/special) is a cifs/samba mount:

opensuse:~ # mount | grep /special
//sambaserver.example.local/special on /special type cifs (rw,...)

The permissions on /special were correct (0755), so the "nagios" user has the correct permissions to access the mountpoint. But even as root the check_disk failed:

opensuse:~ # /usr/lib/nagios/plugins/check_disk -w 10% -c 5% -p /special ; date
DISK CRITICAL - /special is not accessible: Permission denied
Thu Dec 10 08:14:04 CET 2015

In systemd's journal I found the following errors:

opensuse:~ # journalctl --system | grep "Dec 10 08:14:04"
Dec 10 08:14:04 opensuse kernel: CIFS VFS: Error -13 sending data on socket to server

At first I thought the problems arose because of the installed patches and a bug was installed. But I was able to run the same check_disk just fine out of /tmp:

opensuse:~ # cp /usr/lib/nagios/plugins/check_disk /tmp/
opensuse:~ # /tmp/check_disk -w 15% -c 5% -p /special
DISK OK - free space: /special 64882412 MB (49% inode=-);| /special=66640360MB;111794356;124946633;0;131522772

I tried several other directoris to launch check_disk from and all worked - except the important one: /usr/lib/nagios/plugins.

By grepping for "plugins" in /etc/*, I came across the following result:

/etc/apparmor.d/usr.lib.nagios.plugins.check_disk:2:/usr/lib/nagios/plugins/check_disk {
/etc/apparmor.d/usr.lib.nagios.plugins.check_disk:7:   /usr/lib/nagios/plugins/check_disk rm,

Apparmor!

The file name contains the full path to the executable and looks like this:

opensuse:~ # cat /etc/apparmor.d/usr.lib.nagios.plugins.check_disk
#include
/usr/lib/nagios/plugins/check_disk {
   #include
   #include
   /etc/mtab                        r,
   @{PROC}/[0-9]*/mounts            r,
   /usr/lib/nagios/plugins/check_disk rm,
}

In this apparmor rule, the full path of the executable check_disk is given. So now it made sense, why check_disk didn't work in /usr/lib/nagios/plugins but anywhere else.
For a quick resolution, I disabled this rule by moving it into the disabled folder followed by a restart of apparmor:

opensuse:~ # mv /etc/apparmor.d/usr.lib.nagios.plugins.check_disk /etc/apparmor.d/disable/
opensuse:~ # systemctl restart apparmor

Afterwards check_disk was working again:

opensuse:~ # /usr/lib/nagios/plugins/check_disk -w 10% -c 5% -p /special
DISK OK - free space: /special 64881742 MB (49% inode=-);| /special=66641030MB;118370494;124946633;0;131522772

But why did it work until the system was rebooted?

When the monitoring-plugins package was installed (on December 8th), the apparmor rule files came with it:

opensuse:~ # stat /etc/apparmor.d/usr.lib.nagios.plugins.check_* | egrep "File|Change"
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_dhcp’
Change: 2015-12-08 12:43:54.558360191 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_icmp’
Change: 2015-12-08 12:43:54.837370296 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_ide_smart’
Change: 2015-12-08 12:43:54.928373593 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_load’
Change: 2015-12-08 12:43:54.962374824 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_ntp_time’
Change: 2015-12-08 12:43:55.152381708 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_ping’
Change: 2015-12-08 12:43:55.266385838 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_procs’
Change: 2015-12-08 12:43:55.296386925 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_ssh’
Change: 2015-12-08 12:43:55.388390258 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_swap’
Change: 2015-12-08 12:43:55.439392106 +0100
  File: ‘/etc/apparmor.d/usr.lib.nagios.plugins.check_users’
Change: 2015-12-08 12:43:55.571396888 +0100

2015-12-08 12:43 is exactly the time, the monitoring-plugins package was installed with zypper. However apparmor was not restarted as this can be seen with journalctl:

opensuse:~ # journalctl -u apparmor.service
[...]
-- Reboot --
Oct 26 09:56:17 opensuse boot.apparmor[798]: Starting AppArmor ..done
Oct 26 10:16:32 opensuse boot.apparmor[2839]: Unloading AppArmor profiles  ..done
-- Reboot --
Oct 26 10:16:50 opensuse boot.apparmor[901]: Starting AppArmor ..done
Dec 09 20:09:19 opensuse boot.apparmor[22683]: Unloading AppArmor profiles  ..done
-- Reboot --
Dec 09 20:12:16 opensuse boot.apparmor[1019]: Starting AppArmor ..done
Dec 10 10:32:22 opensuse boot.apparmor[3016]: Unloading AppArmor profiles  ..done
Dec 10 10:32:23 opensuse boot.apparmor[3032]: Starting AppArmor ..done

Only when the server booted up on December 9th 20:12, apparmor was loaded with the new rules. And since then the check_disk plugin stopped working on /special. 

Instead of just ranting here, I will contact the maintainer of the monitoring-plugins (I installed from zypper repo http://download.opensuse.org/repositories/server:/monitoring/openSUSE_13.2/) and ask for a correction of the apparmor rule. Information about building and maintaining this package (in this case monitoring-plugins-disk) can be found here: https://build.opensuse.org/package/show?project=server%3Amonitoring&package=monitoring-plugins.  According to Request 341451, the apparmor rules were added on August 1 2015:

+-------------------------------------------------------------------
+Sat Aug  1 19:09:11 UTC 2015 - lars@linux-schulserver.de
+
+- add apparmor profiles for the following checks:
+  + check_disk
+  + check_load
+  + check_procs
+  + check_swap
+  + check_users