Header RSS Feed
 
If you only want to see the articles of a certain category, please click on the desired category below:
ALL Android Backup BSD Database Hacks Hardware Internet Linux Mail MySQL Monitoring Network Personal PHP Proxy Shell Solaris Unix Virtualization VMware Windows Wyse

Reduce the number of shards of an Elasticsearch index
Thursday - Dec 27th 2018 - by - (0 comments)

When you run a lot of indexes, this can create quite a large sum of shards in your ELK stack. As the documentation states, the default creates 5 shards per index:

index.number_of_shards
    The number of primary shards that an index should have. Defaults to 5. This setting can only be set at index creation time. It cannot be changed on a closed index. Note: the number of shards are limited to 1024 per index. This limitation is a safety limit to prevent accidental creation of indices that can destabilize a cluster due to resource allocation. The limit can be modified by specifying export ES_JAVA_OPTS="-Des.index.max_number_of_shards=128" system property on every node that is part of the cluster.

Another interesting default setting is the number of replicas of each shard:

 index.number_of_replicas
    The number of replicas each primary shard has. Defaults to 1.

Once a new index was created, the number of shards is fixed. Only by creating a new index the number of shards can be defined; either during the creation of the index itself or by defining the settings in the template. 

Note: Yes, it is possible to change the number of shards on already created indexes, but that means you must re-index that index again, possibly causing downtime.

In my setup, a classical ELK stack, there are a couple of indexes (logstash, filebeat, haproxy, ...)  created every day, typically with the date in the index name (logstash-2018.12.27). By adjusting the shard settings in the templates "logstash" and "filebeat", the indexes created from tomorrow on and later will have a reduced number of shards.

First let's take a backup:

# elk=localhost:9200
# curl $elk/_template/logstash?pretty -u elastic -p > /root/template-logstash.backup

Now create a new file, e.g. /tmp/logstash, based on the backup flie. Add the "number_of_shards" and "number_of_replicas" into the settings key.

Also make sure, that you remove the "logstash" main key itself. So the file looks like this:

# cat /tmp/logstash
{
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "number_of_shards" : 2,
      "number_of_replicas" : 1,
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
[...]
        ],
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "keyword"
          },
          "geoip" : {
            "dynamic" : true,
            "properties" : {
              "ip" : {
                "type" : "ip"
              },
              "location" : {
                "type" : "geo_point"
              },
              "latitude" : {
                "type" : "half_float"
              },
              "longitude" : {
                "type" : "half_float"
              }
            }
          }
        }
      }
    },
    "aliases" : { }
}

And now this file can be "PUT" into Elasticsearch templates:

# curl -H "Content-Type: application/json" -XPUT $elk/_template/logstash -d "@/tmp/logstash" -u elastic -p
Enter host password for user 'elastic':
{"acknowledged":true}

By checking out the template again, our adjusted shard settings are now showing:

# curl $elk/_template/logstash?pretty -u elastic -p
{
  "logstash" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "2",
        "number_of_replicas" : "1",
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {


 

Add a comment

Show form to leave a comment

Comments (newest first):

No comments yet.

Go to Homepage home
Linux Howtos how to's
Monitoring Plugins monitoring plugins
Links links

Valid HTML 4.01 Transitional
Valid CSS!
[Valid RSS]

6937 Days
until Death of Computers
Why?