Version 1.5 of check_es_system adds monitoring for read-only indexes in Elasticsearch

Written by - 0 comments

Published on - Listed in Monitoring Elasticsearch Icinga Nagios ELK


A new version of the monitoring plugin check_es_system, to monitor Elasticsearch, is available!

Version 1.5 features a new check type called "readonly" (-t readonly) and checks if any indexes are set to read-only. By default the plugin checks all indexes ("_all") for their settings and will alert if any read-only indexes are found. In the background the plugin checks, if it can find the value index.blocks.read_only_allow_delete set to true.

To allow a check on specific indexes, version 1.5 also introduces a new parameter -i. This parameter is used to overwrite the default list of indexes to be checked. The following example shows how to check all filebeat and logstash indexes:

$ ./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t readonly -i "filebeat-* logstash-*"
CRITICAL - Elasticsearch Index filebeat-* is read-only (found 53 index(es) set to read-only) Elasticsearch Index logstash-* is read-only (found 125 index(es) set to read-only)

Such a read-only situation might happen if your Elasticsearch cluster (temporarily) ran below the disk watermark thresholds. Elasticsearch then retries a couple of times to re-apply the write operation (default: 5 times) until it gives up and marks the index as read-only.

Only a manual overwrite of the index.blocks.read_only_allow_delete setting (per index or on "_all") will remove the read-only flag. More on this topic in article "Elasticsearch ignored disk watermark settings and enforced read only index".

Once the issue is fixed and the indexes allow write-operations again, the plugin will return OK:

$ ./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t readonly -i "filebeat-* logstash-*"
ES SYSTEM OK - Elasticsearch Indexes (filebeat-* logstash-*) are writeable



Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder