For quite some time now I've been using rsnapshot to create local backups on some web servers. However I lately saw increasing loads in the night during the rsnapshot backup which causes some services (like HTTP) to time out. That, of course, is not acceptable.
To reduce the load during the backup, I already used nice to decrease the CPU priority on the backup processes. However this caused no visible change in the load because nice handles CPU requests but not IO requests. And the timeouts of the services occurred due to disk latency, not CPU wait.
So to handle this scenario, ionice can be used. There are three classed ionice can be used with. For this purpose (backup) I decided to use class 3 (idle) - meaning that the rsnapshot process will only access the disk when the disk is idle. I was a bit afraid that the backup process would now take much longer but that wasn't actually the case.
Here is a graphic showing the system load before (red) and after (dark red) ionice was added.
As one can clearly see from the graphs, the CPU load decreased a lot!
I also checked the logs to see how much longer the backup process now takes and I was surprised that it wasn't such a big difference:
June 15: 42 minutes
June 16: 46 minutes
June 17: 42 minutes
June 18: 58 minutes
June 19: 48 minutes
---------------------------------
Average: 47.2 minutes (without ionice)
On June 19th during the day I adapted the cronjob and added ionice (see above):
ionice -c 3 /usr/bin/rsnapshot daily
June 20: 69 minutes
June 21: 51 minutes
June 22: 54 minutes
June 23: 55 minutes
June 24: 73 minutes
---------------------------------
Average: 60.4 minutes (with ionice)
That's an average of 13 minutes longer. I've expected much worse numbers and can greatly live with a backup process which takes 13mins longer.
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb Icingaweb2 Influx Internet Java KVM Kibana Kodi Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder