Kibana throws error X of Y shards failed on visualizations and dashboards

Written by - 0 comments

Published on - Listed in ELK Elasticsearch Filebeat

Kibana might at some point throw an error on dashboards or visualizations: X of Y shards failed.

Kibana error X of Y shards failed

The problem? This error message is not helpful at all.

Shards failed?

Once familiar with the Elastic Stack (previously known as ELK Stack) one knows that Elasticsearch's indexes are split into shards onto one or more nodes. Kibana's error message clearly indicates failed shards, so something must have gone bad with shards in Elasticsearch, right?

Wrong! Although Kibana mentions shards failed, all shards are working correctly as they should. This can be verified in Elasticsearch itself:

root@es:~# curl -s http://localhost:9200/_cat/shards?pretty | egrep -v STARTED; echo "result"

Meaning: All shards are correctly started.

But why would Kibana still mention to failed shards?

Browser console to the help. No, really!

As the error shows up in Kibana and therefore in the browser, opening the browser console (F12) helps to show additional information. Inside the json-formatted response , failures are quite nicely listed for each shard. And more importantly: The reason why the shard(s) failed appears, too!

Kibana Browser Console Shards Failed in JSON Response

One important hint to find the reason for the error message was retrieved:

Fielddata is disabled on text fields by default. Set fielddata=true on [response] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

Finally some useful error message.

What data does the visualization want to show?

This might be a (helpful) spoiler, but the error itself comes from the data visualizations. Or dashboards, as a dashboard is just a collection of visualizations. To figure out why a certain visualization returns this error, the data itself must be checked; what data or field is the visualization supposed to show?

In this particular example the filebeat-* index is analyzed and the text field "clientip" is supposed to show up as visualized data. But the graph fails to show data and, yet again, the error message X of Y shards failed shows up:

Kibana Visualization Shards Failed

The research of the previous error message (fielddata is disabled on text fields) leads to the official Elasticsearch reference of fielddata and something important is written there:

Before you enable fielddata, consider why you are using a text field for aggregations, sorting, or in a script. It usually doesn’t make sense to do so. [...]
Instead, you should have a text field for full text searches, and an unanalyzed keyword field with doc_values enabled for aggregations

Update Kibana Index Patterns

All the fields of the filebeat index can be verified in Kibana's management of "Index Patterns".

Kibana Index Patterns

A total of 163 fields were discovered in the filebeat-* index patterns. The "clientip" text field was listed, too, however no clientip.keyword field. After a click on the "refresh field list" icon and a few seconds later 190 fields were shown. One of the new ones: clientip.keyword.

Adjust the Visualization

Back editing the visualization, the new field "clientip.keyword" was used instead of the "original text field" clientip:

Kibana visualization with text field keyword

And finally the data is shown (again)!

What broke the data visualization?

The visualizations were created a while ago and were always working in the dashboard(s). But all of a sudden this error X of Y shards failed showed up in Kibana. But why would this all of a sudden happen?

Two recent modifications happened in this environment.

1: There was a recent update of Elasticsearch from 6.5 to 6.8.
2: The filebeat index template was updated with a new one, probably missing previously created mappings.

One of these modifications or possibly even both of them combined caused the keyword fields to be gone and the visualizations to break.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder