Weird case: MySQL causing high disk IO when ZFS filesystem shrinks

Written by Claudio Kuenzler - 1 comments

Published on December 18th 2012 - Listed in BSD Database Shell MySQL ZFS Rant

In the past days I was working on a FreeBSD server to figure out why the performance went down sharply.
The disks, used by the zfs pool used by MySQL for data, logs and backup, had an increasing I/O. With every passing day, the I/O's, especially the writes, jumped higher.

With methods like mytop I tried to find the responsible database causing so much I/O but nothing would point to a single database or single user causing all the trouble.

By accident I also stumbled across a filling file system. Due to a mistake, the daily created database dumps weren't deleted for several months and were logically eating more and more disk space of the zfs pool. I corrected the dump script, which would run as a daily cron job the coming night.

The next day the surprise was perfect: The disk I/Os of the database fell sharply down, as soon as the available/free disk space increased (due to the deletion of old dumps).
The following graph shows it clearly:

Usually I write in my blog about cases, where I found the reason for the problem. But this time I can't explain it.
I've heard of problems with zfs when free disk space gets lower than 10%, but as one can see, the 10% were still far away.
Also, is this a zfs, FreeBSD or MySQL issue? What would be causing much more write I/O's just because the available space shrinks?

Any hint is welcome! :-)

System infos: FreeBSD 8.3, MySQL 5.1.45, zfs v28

Add a comment

Comments (newest first)

foo from wrote on Dec 21st, 2012:

Meine Vermutung geht in die Richtung journaling filesystem ...