In the past days I was working on a FreeBSD server to figure out why the performance went down sharply.
The disks, used by the zfs pool used by MySQL for data, logs and backup, had an increasing I/O. With every passing day, the I/O's, especially the writes, jumped higher.
With methods like mytop I tried to find the responsible database causing so much I/O but nothing would point to a single database or single user causing all the trouble.
By accident I also stumbled across a filling file system. Due to a mistake, the daily created database dumps weren't deleted for several months and were logically eating more and more disk space of the zfs pool. I corrected the dump script, which would run as a daily cron job the coming night.
The next day the surprise was perfect: The disk I/Os of the database fell sharply down, as soon as the available/free disk space increased (due to the deletion of old dumps).
The following graph shows it clearly:
Usually I write in my blog about cases, where I found the reason for the problem. But this time I can't explain it.
I've heard of problems with zfs when free disk space gets lower than 10%, but as one can see, the 10% were still far away.
Also, is this a zfs, FreeBSD or MySQL issue? What would be causing much more write I/O's just because the available space shrinks?
Any hint is welcome! :-)
System infos: FreeBSD 8.3, MySQL 5.1.45, zfs v28