A new version of check_es_system, a monitoring plugin to monitor Elasticsearch clusters and nodes, is available!
Version 1.11.0 is an enhancement release (requested in issue #30). The readonly check type now shows discovered index(es) marked as read_only or read_only_allow_delete when using jq as json parser in the background:
./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t readonly
ES SYSTEM CRITICAL - 1 index(es) found read-only claudiotest - 116
index(es) found read-only (allow delete) logstash-2020.09.21
logstash-2020.09.04 filebeat-2020.09.04 .monitoring-es-6-2020.09.21 idx
filebeat-2020.09.08 filebeat-2020.09.07 filebeat-2020.09.01 claudiotest2
.management-beats logstash-2020.09.05 filebeat-2020.09.10
logstash-2020.09.06 .triggered_watches-6 logstash-2020.09.02
filebeat-2020.09.21 logstash-2020.09.03 .monitoring-kibana-6-2020.09.21
filebeat-2020.09.03 logstash-2020.09.01 logstash-2020.09.08
kibana_sample_data_logs .kibana_7 .kibana_task_manager
filebeat-2020.09.05 .security-6 .watches filebeat-2020.09.11
filebeat-2020.09.09 .kibana-6 filebeat-2020.09.02 logstash-2020.09.07
filebeat-2020.09.06
As you can see from the output, read_only and read_only_allow_delete indexes are shown differently.
Note: In case you are unaware, when Elasticsearch runs into problems (e.g. disk full), it marks the index(es) as read_only_allow_delete and new data cannot be written into these indexes anymore. See article Elasticsearch ignored disk watermark settings and enforced read only index for more information. It is therefore a very handy check to quickly identify if your Elasticsearch went into read only mode.
To be able to parse all the read only indexes from Elasticsearch's output (in json), a special function of jq is needed. From the source code:
roindexes=$(echo $settings | jq -r '.[].settings.index |select(.blocks.read_only == "true").provided_name')
The select function is used as a filter and is looking for all the keys read_only with a value of true. If it finds such a key, the name of this relevant index (provided_name) is displayed.
Unfortunately the other supported JSON parser, jshon, does not offer such a filter or search function. Therefore the output of the discovered read only index(es) only show up when using jq as parser.
Due to this fact, the default JSON parser from this version on is jq. Both parsers remain (as for now) supported.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Container Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb2 InfluxDB Internet Java KVM Kibana Kodi Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher SSL Seafile Security Shell SmartOS Solaris Surveillance SystemD TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder