Weird case: MySQL causing high disk IO when ZFS filesystem shrinks

Written by - 1 comments

Published on - Listed in BSD Database Shell MySQL ZFS Rant

In the past days I was working on a FreeBSD server to figure out why the performance went down sharply.
The disks, used by the zfs pool used by MySQL for data, logs and backup, had an increasing I/O. With every passing day, the I/O's, especially the writes, jumped higher.

With methods like mytop I tried to find the responsible database causing so much I/O but nothing would point to a single database or single user causing all the trouble.

By accident I also stumbled across a filling file system. Due to a mistake, the daily created database dumps weren't deleted for several months and were logically eating more and more disk space of the zfs pool. I corrected the dump script, which would run as a daily cron job the coming night.

The next day the surprise was perfect: The disk I/Os of the database fell sharply down, as soon as the available/free disk space increased (due to the deletion of old dumps).
The following graph shows it clearly:

MySQL high IO due to low zfs file system

Usually I write in my blog about cases, where I found the reason for the problem. But this time I can't explain it.
I've heard of problems with zfs when free disk space gets lower than 10%, but as one can see, the 10% were still far away.
Also, is this a zfs, FreeBSD or MySQL issue? What would be causing much more write I/O's just because the available space shrinks?

Any hint is welcome! :-)

System infos: FreeBSD 8.3, MySQL 5.1.45, zfs v28

Add a comment

Show form to leave a comment

Comments (newest first)

foo from wrote on Dec 21st, 2012:

Meine Vermutung geht in die Richtung journaling filesystem ...

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder