One of the reasons why ZFS is a fast file system is the use of a file system caching called ARC (Adaptive Replacement Cache), which runs in memory.
The current size of ARC can be seen with the following command:
kstat -p zfs:0:arcstats:size
At the time of this output, ARC was using 11.4GB.
Although ARC should be intelligent enough to not occupy RAM which is needed by zones or applications, I have seen cases where ARC uses 50% of the total available memory and the other 50% were fully occupied by zones and applications, causing applications (like MySQL) to swap. That's a performance nightmare.But there's a way to limit the ARC size by setting the value of zfs_arc_max. At the end of /etc/system I added the following lines:
* Limit ZFS Arc
set zfs:zfs_arc_max = 21474836480
This means that I have set a maximum limit for the zfs arc to 20GB (21474836480 Bytes).
After a reboot (and RAM-Upgrade at the same time), the memory graph clearly shows that the ARC size (represented by CACHES in the graph) has hit 20GB, but never went above it:
ck from Switzerland wrote on May 27th, 2020:
Hi Jodi, your current action plan wants to set a minimum of 6GB and a maximum of 10GB for the ARC cache. Given the total memory capacity (256GB ?), I'd go for for a maximum of 50GB for the ARC cache. I havent worked with Solaris in the last couple of years but I kind of remember that the ARC cache should never be lower than 20% of the whole memory capacity - or you will get into file system performance problems. 50GB is around that 20% level. So you give Solaris the possibility to increase the ARC cache up to 50GB / 20% of the total system memory. Maybe this even needs to be set higher but you can start trying with that value. I would set these values in /etc/system:
set zfs:zfs_arc_max = 53687091200
set zfs:zfs_arc_min = 10737418240
Jodi from wrote on May 26th, 2020:
Thanks for sharing this article
i have memory performance issue in Ldom where ARC uses 60% of total memory and the other 10% were used by zone and oracle db,
with the following output grep by command ( zfs used 145Gb of ram and apps only 71G )
# echo "::memstat" | mdb -kw
Usage Type/Subtype Pages Bytes %Tot %Tot/%Subt
---------------------------- ---------------- -------- ----- -----------
ZFS 19061406 145.4g 55.0% --> zfs cache
User/Anon 9322025 71.1g 26.9% --> Application
also current memory setting as below :
- Ldom : ram=290g,swap=290g
- zone : ram=256g,swap=256g ( shared )
do the action plan, i adjusted the /etc/system file to add the followeing parameter and reboot os :
set zfs:zfs_arc_max = 10737418240
set zfs:zfs_arc_min = 6442450944
kindly let me know if the above action plan is correct ?
Thank you and really appreciated for attantion and response
Jodi | unix support
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Container Containers CouchDB DB DNS Database Docker ELK Elasticsearch Filebeat FreeBSD Galera GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb2 InfluxDB Internet Java KVM Kibana Kodi Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Redis SSL Samba Seafile Security Shell SmartOS Solaris Surveillance SystemD Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder