Version 1.5 of check_es_system adds monitoring for read-only indexes in Elasticsearch

Written by Claudio Kuenzler - 0 comments

Published on June 26th 2019 - Listed in Monitoring Elasticsearch Icinga Nagios ELK

A new version of the monitoring plugin check_es_system, to monitor Elasticsearch, is available!

Version 1.5 features a new check type called "readonly" (-t readonly) and checks if any indexes are set to read-only. By default the plugin checks all indexes ("_all") for their settings and will alert if any read-only indexes are found. In the background the plugin checks, if it can find the value index.blocks.read_only_allow_delete set to true.

To allow a check on specific indexes, version 1.5 also introduces a new parameter -i. This parameter is used to overwrite the default list of indexes to be checked. The following example shows how to check all filebeat and logstash indexes:

$ ./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t readonly -i "filebeat-* logstash-*"
CRITICAL - Elasticsearch Index filebeat-* is read-only (found 53 index(es) set to read-only) Elasticsearch Index logstash-* is read-only (found 125 index(es) set to read-only)

Such a read-only situation might happen if your Elasticsearch cluster (temporarily) ran below the disk watermark thresholds. Elasticsearch then retries a couple of times to re-apply the write operation (default: 5 times) until it gives up and marks the index as read-only.

Only a manual overwrite of the index.blocks.read_only_allow_delete setting (per index or on "_all") will remove the read-only flag. More on this topic in article "Elasticsearch ignored disk watermark settings and enforced read only index".

Once the issue is fixed and the indexes allow write-operations again, the plugin will return OK:

$ ./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t readonly -i "filebeat-* logstash-*"
ES SYSTEM OK - Elasticsearch Indexes (filebeat-* logstash-*) are writeable

Add a comment

Comments (newest first)

No comments yet.