A new version of the monitoring plugin check_es_system, to monitor Elasticsearch, is available!
Version 1.5 features a new check type called "readonly" (-t readonly) and checks if any indexes are set to read-only. By default the plugin checks all indexes ("_all") for their settings and will alert if any read-only indexes are found. In the background the plugin checks, if it can find the value index.blocks.read_only_allow_delete set to true.
To allow a check on specific indexes, version 1.5 also introduces a new parameter -i. This parameter is used to overwrite the default list of indexes to be checked. The following example shows how to check all filebeat and logstash indexes:
$ ./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t readonly -i "filebeat-* logstash-*"
CRITICAL - Elasticsearch Index filebeat-* is read-only (found 53
index(es) set to read-only) Elasticsearch Index logstash-* is read-only
(found 125 index(es) set to read-only)
Such a read-only situation might happen if your Elasticsearch cluster (temporarily) ran below the disk watermark thresholds. Elasticsearch then retries a couple of times to re-apply the write operation (default: 5 times) until it gives up and marks the index as read-only.
Only a manual overwrite of the index.blocks.read_only_allow_delete setting (per index or on "_all") will remove the read-only flag. More on this topic in article "Elasticsearch ignored disk watermark settings and enforced read only index".
Once the issue is fixed and the indexes allow write-operations again, the plugin will return OK:
$ ./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t readonly -i "filebeat-* logstash-*"
ES SYSTEM OK - Elasticsearch Indexes (filebeat-* logstash-*) are writeable
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder