Header RSS Feed
 
If you only want to see the articles of a certain category, please click on the desired category below:
ALL Android Backup BSD Database Hacks Hardware Internet Linux Mail MySQL Monitoring Network Personal PHP Proxy Shell Solaris Unix Virtualization VMware Windows Wyse

Weird case: MySQL causing high disk IO when filesystem shrinks
Tuesday - Dec 18th 2012 - by - (1 comments)

In the past days I was working on a FreeBSD server to figure out why the performance went down sharply.
The disks, used by the zfs pool used by MySQL for data, logs and backup, had an increasing I/O. With every passing day, the I/O's, especially the writes, jumped higher.

With methods like mytop I tried to find the responsible database causing so much I/O but nothing would point to a single database or single user causing all the trouble.

By accident I also stumbled across a filling file system. Due to a mistake, the daily created database dumps weren't deleted for several months and were logically eating more and more disk space of the zfs pool. I corrected the dump script, which would run as a daily cron job the coming night.

The next day the surprise was perfect: The disk I/Os of the database fell sharply down, as soon as the available/free disk space increased (due to the deletion of old dumps).
The following graph shows it clearly:

MySQL high IO due to low zfs file system

Usually I write in my blog about cases, where I found the reason for the problem. But this time I can't explain it.
I've heard of problems with zfs when free disk space gets lower than 10%, but as one can see, the 10% were still far away.
Also, is this a zfs, FreeBSD or MySQL issue? What would be causing much more write I/O's just because the available space shrinks?

Any hint is welcome! :-)

System infos: FreeBSD 8.3, MySQL 5.1.45, zfs v28

 

Add a comment

Show form to leave a comment

Comments (newest first):

foo wrote on Dec 21st, 2012:
Meine Vermutung geht in die Richtung journaling filesystem ...


Go to Homepage home
Linux Howtos how to's
Nagios Plugins nagios plugins
Links links

Valid HTML 4.01 Transitional
Valid CSS!
[Valid RSS]

7633 Days
until Death of Computers
Why?