How to quickly change the number of replicas setting in an Elasticsearch Index

Written by - 0 comments

Published on - Listed in Elasticsearch ELK Monitoring


Depending on the environment where an Elasticsearch cluster is running, the (default) index settings might need to be changed. Especially if the cluster only exists with a one-node setup (which is nothing abnormal) indexes with a replica setting of 1 or more will report unassigned shards.

This can be verified with the check_es_system Elasticsearch monitoring plugin:

root@elasticsearch:~# /usr/lib/nagios/plugins/check_es_system.sh -H localhost -t status
ES SYSTEM WARNING - Elasticsearch Cluster "mycluster" is yellow (1 nodes, 1 data nodes, 3 shards, 0 relocating shards, 0 initializing shards, 3 unassigned shards, 10934 docs)

Unassigned shards and replica settings

The reason can be spotted when listing all the available shards:

root@elasticsearch:~# curl -q -s "http://localhost:9200/_cat/shards"
magento2_product_1_v204 0 p STARTED    3639 6.2mb 127.0.0.1 elasticsearch
magento2_product_1_v204 0 r UNASSIGNED                      
magento2_product_2_v204 0 p STARTED    3639 7.1mb 127.0.0.1 elasticsearch
magento2_product_2_v204 0 r UNASSIGNED                      
magento2_product_3_v204 0 p STARTED    3656 6.2mb 127.0.0.1 elasticsearch
magento2_product_3_v204 0 r UNASSIGNED  

All the unassigned shards have "r" as second flag - which stands for replicated shard. As this is a single node cluster, Elasticsearch does not find any other nodes where to replicate this shard to -> UNASSIGNED.

By looking at the settings of these three indexes, especially the number_of_replicas setting, all indexes have a (default) setting of 1 replica defined:

root@elasticsearch:~# curl -q -s "http://localhost:9200/magento2_product_1_v204/_settings" | jq -r '.magento2_product_1_v204.settings.index.number_of_replicas'
1

root@elasticsearch:~# curl -q -s "http://localhost:9200/magento2_product_2_v204/_settings" | jq -r '.magento2_product_2_v204.settings.index.number_of_replicas'
1

root@elasticsearch:~# curl -q -s "http://localhost:9200/magento2_product_3_v204/_settings" | jq -r '.magento2_product_2_v204.settings.index.number_of_replicas'
1

Changing the number_of_replicas setting

Fortunately this setting can be changed without downtime. With a PUT request the index settings can be updated:

root@elasticsearch:~# curl -X PUT -H "Content-Type: application/json" "http://localhost:9200/magento2_product_1_v204/_settings" -d '{ "number_of_replicas": 0 }'
{"acknowledged":true}

Verification of the current setting value:

root@elasticsearch:~# curl -q -s "http://localhost:9200/magento2_product_1_v204/_settings" | jq -r '.magento2_product_1_v204.settings.index.number_of_replicas'
0

Yes, it worked!

Now looking at the shards again, we can see there are only two unassigned shards left:

root@elasticsearch:~# curl -q -s "http://localhost:9200/_cat/shards"
magento2_product_1_v204 0 p STARTED    3639 6.2mb 127.0.0.1 elasticsearch
magento2_product_2_v204 0 p STARTED    3639 7.1mb 127.0.0.1 elasticsearch
magento2_product_2_v204 0 r UNASSIGNED                      
magento2_product_3_v204 0 p STARTED    3656 6.2mb 127.0.0.1 elasticsearch
magento2_product_3_v204 0 r UNASSIGNED  

The number_of_replicas setting is now also changed on the two remaining indexes:

root@elasticsearch:~# curl -X PUT -H "Content-Type: application/json" "http://localhost:9200/magento2_product_2_v204/_settings" -d '{ "number_of_replicas": 0 }'
{"acknowledged":true}

root@elasticsearch:~# curl -X PUT -H "Content-Type: application/json" "http://localhost:9200/magento2_product_3_v204/_settings" -d '{ "number_of_replicas": 0 }'
{"acknowledged":true}

And voilĂ , no more unassigned shards are shown:

root@elasticsearch:~# curl -q -s "http://localhost:9200/_cat/shards"
magento2_product_1_v204 0 p STARTED 3639 6.2mb 127.0.0.1 elasticsearch
magento2_product_2_v204 0 p STARTED 3639 7.1mb 127.0.0.1 elasticsearch
magento2_product_3_v204 0 p STARTED 3656 6.2mb 127.0.0.1 elasticsearch

Elasticsearch Status change

Now with all unassigned shards gone, the Elasticsearch status immediately changed to green:

root@elasticsearch:~# /usr/lib/nagios/plugins/check_es_system.sh -H localhost -t status
ES SYSTEM OK - Elasticsearch Cluster "mycluster" is green (1 nodes, 1 data nodes, 3 shards, 10934 docs)|total_nodes=1;;;; data_nodes=1;;;; total_shards=3;;;; relocating_shards=0;;;; initializing_shards=0;;;; unassigned_shards=0;;;; docs=10934;;;;



Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder