Header RSS Feed
 
If you only want to see the articles of a certain category, please click on the desired category below:
ALL Android Backup BSD Database Hacks Hardware Internet Linux Mail MySQL Monitoring Network Personal PHP Proxy Shell Solaris Unix Virtualization VMware Windows Wyse

Increase JVM Heap Size Memory for ElasticSearch
Thursday - Sep 14th 2017 - by - (0 comments)

After increasing the usage of the ELK stack in the last few days, I noticed more and more lags and even timeouts in searches. I found the bottleneck in Elasticsearch itself: The available memory for ElasticSearch was limited to 2GB (probably the default).

Elasticsearch memory

To increase the JVM Heap Size for ElasticSearch (remember, ES is a Java application), the preferred way to define the heap sizes is to use the jvm.options file. This file is located in ELASTICSEARCH_CONF_DIR, in a standard installation this means in /etc/elasticsearch/:

root@elkes01:~# cat /etc/elasticsearch/jvm.options |egrep "(Xms|Xmx)"
## -Xms4g
## -Xmx4g
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms12g
-Xmx12g

 Important: Both initial (Xms) and maximum (Xmx) sizes must be the same value, or you get an error like this and ES won't start:

[1]: initial heap size [4294967296] not equal to maximum heap size [12884901888]; this can cause resize pauses and prevents mlockall from locking the entire heap

After adapting the Heap Size values, ElasticSearch must be restarted and if you follow the log file, it will confirm the new values:

[2017-09-14T09:28:00,021][INFO ][o.e.n.Node               ] [elkes01] initializing ...
[2017-09-14T09:28:00,118][INFO ][o.e.e.NodeEnvironment    ] [elkes01] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/mapper/vges-lves)]], net usable_space [819.5gb], net total_space [1007.8gb], spins? [possibly], types [ext4]
[2017-09-14T09:28:00,118][INFO ][o.e.e.NodeEnvironment    ] [elkes01] heap size [11.9gb], compressed ordinary object pointers [true]
[2017-09-14T09:28:00,531][INFO ][o.e.n.Node               ] [elkes01] node name [elkes01], node ID [t3GAvhY1SS2xZkt4U389jw]
[2017-09-14T09:28:00,531][INFO ][o.e.n.Node               ] [elkes01] version[5.6.0], pid[9390], build[781a835/2017-09-07T03:09:58.087Z], OS[Linux/4.4.0-83-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_144/25.144-b01]
[2017-09-14T09:28:00,531][INFO ][o.e.n.Node               ] [elkes01] JVM arguments [-Xms4g, -Xmx12g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-09-14T09:28:01,212][INFO ][o.e.p.PluginsService     ] [elkes01] loaded module [aggs-matrix-stats]
[2017-09-14T09:28:01,212][INFO ][o.e.p.PluginsService     ] [elkes01] loaded module [ingest-common]

Searches and also visual graphics (which at the end use seaches, too) are now much faster in Kibana!

 

Add a comment

Show form to leave a comment

Comments (newest first):

No comments yet.

Go to Homepage home
Linux Howtos how to's
Monitoring Plugins monitoring plugins
Links links

Valid HTML 4.01 Transitional
Valid CSS!
[Valid RSS]

7393 Days
until Death of Computers
Why?