MySQL InnoDB: Operating system error number 13 in a file operation

Written by - 0 comments

Published on - Listed in Linux Database MySQL


Got an interesting MySQL error today, after I updated an Ubuntu 14.04 system with apt-get upgrade:

[...]
libwbclient0:amd64 (2:4.3.11+dfsg-0ubuntu0.14.04.4) wird eingerichtet ...
libdbd-mysql-perl (4.025-1ubuntu0.1) wird eingerichtet ...
mysql-client-core-5.5 (5.5.54-0ubuntu0.14.04.1) wird eingerichtet ...
mysql-client-5.5 (5.5.54-0ubuntu0.14.04.1) wird eingerichtet ...
mysql-server-core-5.5 (5.5.54-0ubuntu0.14.04.1) wird eingerichtet ...
mysql-server-5.5 (5.5.54-0ubuntu0.14.04.1) wird eingerichtet ...

Konfigurationsdatei »/etc/apparmor.d/usr.sbin.mysqld«
 ==> Geändert (von Ihnen oder von einem Skript) seit der Installation.
 ==> Paketverteiler hat eine aktualisierte Version herausgegeben.
   Wie möchten Sie vorgehen? Ihre Wahlmöglichkeiten sind:
    Y oder I : Die Version des Paket-Betreuers installieren
    N oder O : Die momentan installierte Version beibehalten
       D     : Die Unterschiede zwischen den Versionen anzeigen
       Z     : Eine Shell starten, um die Situation zu begutachten
 Der Standardweg ist das Beibehalten der momentanen Version.
*** usr.sbin.mysqld (Y/I/N/O/D/Z) [Vorgabe=N] ? D

Konfigurationsdatei »/etc/apparmor.d/usr.sbin.mysqld«
 ==> Geändert (von Ihnen oder von einem Skript) seit der Installation.
 ==> Paketverteiler hat eine aktualisierte Version herausgegeben.
   Wie möchten Sie vorgehen? Ihre Wahlmöglichkeiten sind:
    Y oder I : Die Version des Paket-Betreuers installieren
    N oder O : Die momentan installierte Version beibehalten
       D     : Die Unterschiede zwischen den Versionen anzeigen
       Z     : Eine Shell starten, um die Situation zu begutachten
 Der Standardweg ist das Beibehalten der momentanen Version.
*** usr.sbin.mysqld (Y/I/N/O/D/Z) [Vorgabe=N] ? I
Neue Version der Konfigurationsdatei /etc/apparmor.d/usr.sbin.mysqld wird installiert ...
start: Job failed to start
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: Fehler beim Bearbeiten des Paketes mysql-server-5.5 (--configure):
 Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
dpkg: Abhängigkeitsprobleme verhindern Konfiguration von mysql-server:
 mysql-server hängt ab von mysql-server-5.5; aber:
  Paket mysql-server-5.5 ist noch nicht konfiguriert.

dpkg: Fehler beim Bearbeiten des Paketes mysql-server (--configure):
 Abhängigkeitsprobleme - verbleibt unkonfiguriert
mysql-client (5.5.54-0ubuntu0.14.04.1) wird eingerichtet ...
Es wurde kein Apport-Bericht verfasst, da die Fehlermeldung darauf hindeutet, dass dies lediglich ein Folgefehler eines vorherigen Problems ist.
[...]

So MySQL didn't start, that's why apt-get upgrade didn't finish with success. In /var/log/mysql/error I saw the following errors:

170221 14:24:14 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
170221 14:24:14 [Note] Plugin 'FEDERATED' is disabled.
/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13)
170221 14:24:14 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
170221 14:24:14 InnoDB: The InnoDB memory heap is disabled
170221 14:24:14 InnoDB: Mutexes and rw_locks use GCC atomic builtins
170221 14:24:14 InnoDB: Compressed tables use zlib 1.2.8
170221 14:24:14 InnoDB: Using Linux native AIO
170221 14:24:14 InnoDB: Initializing buffer pool, size = 128.0M
170221 14:24:14 InnoDB: Completed initialization of buffer pool
170221 14:24:14  InnoDB: Operating system error number 13 in a file operation.
InnoDB: The error means mysqld does not have the access rights to
InnoDB: the directory.
InnoDB: File name ./ibdata1
InnoDB: File operation call: 'open'.
InnoDB: Cannot continue operation.

dmesg revealed some more information:

[  634.595214] init: mysql respawning too fast, stopped
[  790.457596] audit_printk_skb: 69 callbacks suppressed
[  790.457599] audit: type=1400 audit(1487683611.187:119): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/db/mysql/app003.lower-test" pid=32765 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[  790.457620] audit: type=1400 audit(1487683611.187:120): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/db/mysql/app003.lower-test" pid=32765 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[  790.459437] audit: type=1400 audit(1487683611.187:121): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/db/mysql/mysql/db.frm" pid=302 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=105 ouid=105
[  790.472090] audit: type=1400 audit(1487683611.199:122): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/db/mysql/app003.lower-test" pid=315 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[  790.472117] audit: type=1400 audit(1487683611.199:123): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/db/mysql/app003.lower-test" pid=315 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[  790.486818] audit: type=1400 audit(1487683611.215:124): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/db/mysql/ibdata1" pid=315 comm="mysqld" requested_mask="wr" denied_mask="wr" fsuid=105 ouid=105
[  790.610780] audit: type=1400 audit(1487683611.339:125): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/db/mysql/app003.lower-test" pid=333 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[  790.610820] audit: type=1400 audit(1487683611.339:126): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/db/mysql/app003.lower-test" pid=333 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[  790.624533] audit: type=1400 audit(1487683611.351:127): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/db/mysql/ibdata1" pid=333 comm="mysqld" requested_mask="wr" denied_mask="wr" fsuid=105 ouid=105
[  790.740241] audit: type=1400 audit(1487683611.467:128): apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld" name="/db/mysql/app003.lower-test" pid=349 comm="mysqld" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
[  791.175361] init: mysql main process (430) terminated with status 1
[  791.175369] init: mysql main process ended, respawning
[  792.060469] init: mysql post-start process (431) terminated with status 1
[  792.203859] init: mysql main process (475) terminated with status 1
[  792.203867] init: mysql main process ended, respawning
[  793.085085] init: mysql post-start process (476) terminated with status 1
[  793.237776] init: mysql main process (521) terminated with status 1
[  793.237793] init: mysql respawning too fast, stopped

Combine the errors from mysql ("InnoDB: Operating system error number 13 in a file operation.") and dmesg ("apparmor="DENIED" operation="mknod" profile="/usr/sbin/mysqld") and you know where the problem is. During the system patching, a new version of the apparmor rules file was also installed. In most cases this is wanted, as the rule files themselves get patched and fixed from previous bugs.  But on this particular server this was not a wise choice, because - for an unknown reason to me - the admin who installed this system (with German as system language, ugh!) chose /db as path for the MySQL datadir. The Debian/Ubuntu default is of course /var/lib/mysql. Instead of setting /db as datadir and making a new partition on /db, it would have been easier to keep the system defaults and just mount /var/lib/mysql as a separate partition. This would have kept the MySQL installation compatible to future updates.

Because I didn't want apparmor to be running anyway, I stopped and removed apparmor (alternative would have been to edit the apparmor rules file for the MySQL binary). But even after that, MySQL was not able to start. I needed to reboot the server completely in order to get rid of the apparmor security settings. But after the reboot MySQL started correctly again.


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder