In the past days I was working on a FreeBSD server to figure out why the performance went down sharply.
The disks, used by the zfs pool used by MySQL for data, logs and backup, had an increasing I/O. With every passing day, the I/O's, especially the writes, jumped higher.
With methods like mytop I tried to find the responsible database causing so much I/O but nothing would point to a single database or single user causing all the trouble.
By accident I also stumbled across a filling file system. Due to a mistake, the daily created database dumps weren't deleted for several months and were logically eating more and more disk space of the zfs pool. I corrected the dump script, which would run as a daily cron job the coming night.
The next day the surprise was perfect: The disk I/Os of the database fell sharply down, as soon as the available/free disk space increased (due to the deletion of old dumps).
The following graph shows it clearly:
Usually I write in my blog about cases, where I found the reason for the problem. But this time I can't explain it.
I've heard of problems with zfs when free disk space gets lower than 10%, but as one can see, the 10% were still far away.
Also, is this a zfs, FreeBSD or MySQL issue? What would be causing much more write I/O's just because the available space shrinks?
Any hint is welcome! :-)
System infos: FreeBSD 8.3, MySQL 5.1.45, zfs v28
foo from wrote on Dec 21st, 2012:
Meine Vermutung geht in die Richtung journaling filesystem ...
Personal Internet VMware PHP Linux Shell Bluecoat Proxy Windows Hardware Virtualization Nagios MySQL DB Monitoring Mail Android Network Wyse Hacks Tomcat Postgres Apple Mac Backup BSD ZFS Solaris SmartOS Unix Multimedia Perl Database MongoDB CMS OTRS FreeBSD Wordpress LXC Nginx Proxmox DNS Graphics GlusterFS Security Chef HAProxy Icinga Ansible HTML MariaDB Containers Rancher Docker AWS ELK Kibana Logstash Filebeat Varnish PGSQL PostgreSQL ElasticSearch CouchDB Bash Macintosh Container Minio Grafana InfluxDB Databases NFS OSSEC SystemD Java Zoneminder Surveillance Elasticsearch SSL TLS Icingaweb2 Cloud Wireless Kubernetes Ubuntu