Elasticsearch failed on parsing mappings on index creation ([_default_] mappings are not allowed)

Written by - 1 comments

Published on - Listed in Elasticsearch ELK


And here's one more article based on upgrading Elasticsearch from 6.8.6 to 7.15.0, after the previous articles:

This time the errors were only spotted a day after the Elasticsearch upgrade, when Elasticsearch (being part of an ELK stack) wanted to create new indices based on the current date:

[2021-10-01T14:07:24,982][INFO ][o.e.c.m.MetadataCreateIndexService] [elk01] failed on parsing mappings on index creation [rancher-vamp-2021.10.01]
java.lang.IllegalArgumentException: [_default_] mappings are not allowed on new indices and should no longer be used. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#default-mapping-not-allowed] for more information.

[2021-10-01T14:07:24,983][INFO ][o.e.c.m.MetadataCreateIndexService] [elk01] failed on parsing mappings on index creation [filebeat-2021.10.01]
java.lang.IllegalArgumentException: [_default_] mappings are not allowed on new indices and should no longer be used. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#default-mapping-not-allowed] for more information.

The same error was logged thousands of times - every time Elasticsearch wanted to create a new index from the incoming logs (sent by Logstash). The error mentions that a [_default_] mapping is not allowed anymore and therefore the index cannot be created.

_default_ mapping?

Based on the error on above on the filebeat-2021.10.01 index, let's take a closer look at the filebeat index template:

root@elk01:~# curl -X GET "localhost:9200/_template/filebeat?pretty"
{
  "filebeat" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "retention-10d"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

There's nothing at all defined inside the "mappings" context! So why does Elasticsearch think, there's a _default_ mapping defined somewhere?

Actually the Elastic 7.0 breaking changes documentation mentions something interesting concerning the mappings:

Note that in 7.x, the get template API does not show the _default_ mapping by default, even when it is defined in the mapping. To see all mappings in the template, the include_type_name parameter must be supplied:
GET /_template/my_template?include_type_name

Alright, let's add this parameter then:

root@elk01:~# curl -X GET "localhost:9200/_template/filebeat?pretty&include_type_name"
{
  "filebeat" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "retention-10d"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [
          {
            "message_field" : {
              "path_match" : "message",
              "mapping" : {
                "norms" : false,
                "type" : "text"
              },
              "match_mapping_type" : "string"
            }
          },
          {
            "string_fields" : {
              "mapping" : {
                "norms" : false,
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              },
              "match_mapping_type" : "string",
              "match" : "*"
            }
          }
        ],
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "geoip" : {
            "dynamic" : true,
            "properties" : {
              "ip" : {
                "type" : "ip"
              },
              "latitude" : {
                "type" : "half_float"
              },
              "location" : {
                "type" : "geo_point"
              },
              "longitude" : {
                "type" : "half_float"
              }
            }
          },
          "@version" : {
            "type" : "keyword"
          }
        }
      }
    },

    "aliases" : { }
  }
}

Whoa, there's much more data coming back now! And indeed, the _default_ mapping is set in this filebeat template. The solution, according to Elastic's documentation, is to remove the mapping from the index template.

Adjusting the template

Changing an Elasticsearch template basically means the template is first "downloaded" from the Elasticsearch API using a GET request, modified in a local file and the final file with the modifications is uploaded back into the Elasticsearch API using the PUT method. An alternative is of course to not use a temporary file but push everything in one curl command with a very large -d "data" part - but this is much more complicated and prone to (human) errors.

First download the template, with the additional include_type_name parameter:

root@elk01:~# curl -X GET "localhost:9200/_template/filebeat?pretty&include_type_name" > /tmp/template-filebeat-20211001

Now modify the file (/tmp/template-filebeat-20211001) with your favorite editor (vim!) and within mappings {} remove the whole _default_ context. Also make sure you remove the top-level context with the template's name (filebeat {}). The file should then look something like this:

root@elk01:~# cat /tmp/template-filebeat-20211001
{
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "retention-10d"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : { },
    "aliases" : { }
}

Note: In this case this Elasticsearch is part of a TEST ELK setup, with only one Elasticsearch node. Hence the limited number_of_shards and number_of_replicas values.

Now this modified file can be uploaded back into the Elasticsearch API, to replace the existing filebeat template:

root@elk01:~# curl -H "Content-Type: application/json" -X PUT localhost:9200/_template/filebeat -d "@/tmp/template-filebeat-20211001"
{"acknowledged":true}

Verify that the new template doesn't contain the _default_ mapping anymore:

root@elk01:~# curl -X GET "localhost:9200/_template/filebeat?pretty&include_type_name"
{
  "filebeat" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "retention-10d"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : {
      "_doc" : { }
    },

    "aliases" : { }
  }
}

Done - this looks good!

Are we there yet? Does it work?

So what happens now in Elasticsearch? Is the new filebeat-2021-10-01 index created?

root@elk01:~# curl -s http://localhost:9200/_cat/indices?pretty | grep filebeat
green  open filebeat-2021.10.01               TuQ_qNCUQuO_WBXyvHF8ug 1 0    16878      0    3.5mb    3.5mb
green  open filebeat-2021.09.16               wGkmLqK5SJGT5rI-WfvGyQ 1 0    77037      0   14.6mb   14.6mb
green  open filebeat-2021.09.17               i_v0Wr3YRIOLIeWu3gppLw 1 0      141      0  118.3kb  118.3kb
green  open filebeat-2021.09.18               SnbjHKCES72IcKo2w-7uug 1 0       86      0     70kb     70kb
green  open filebeat-2021.09.19               ztA6QXi0R9SVYYrrHEDwtQ 1 0       84      0   68.6kb   68.6kb
green  open filebeat-2021.09.15               h6QwsW-qSz6WNYw5jcIJWA 1 0   615768      0  126.2mb  126.2mb
green  open filebeat-2021.09.30               LKiXjStIQOCrAuat69vncQ 1 0 16745885      0    2.2gb    2.2gb
green  open filebeat-2021.09.27               UVppic3QQWOIIyM7BlBYqw 1 0 16791098      0    2.7gb    2.7gb
green  open filebeat-2021.09.28               PulI3NlmTw-WvtgBTKnWmA 1 0 37654204      0    8.1gb    8.1gb
green  open filebeat-2021.09.29               p2-0aSsOSrWRpGALe2Lg4A 1 0 16755970      0    2.7gb    2.7gb
green  open filebeat-2021.09.23               5unlKkY6T4KVUU4zD16A6Q 1 0       85      0     69kb     69kb
green  open filebeat-2021.09.24               nwu0DlxVQNiDxbSoZTislQ 1 0  1659948      0  858.5mb  858.5mb
green  open filebeat-2021.09.25               QotuSLyZT5Cp6DaDF6hXkw 1 0 39875298      0    5.8gb    5.8gb
green  open filebeat-2021.09.26               r6uczxqjRISiIKHNpCqWJw 1 0 16723026      0    2.7gb    2.7gb
green  open filebeat-2021.09.20               wAwV8gFdR6G0Gz_7c2vs1g 1 0      195      0  148.5kb  148.5kb
green  open filebeat-2021.09.21               5ae0Z_0SR0KEqPcwCqkpnQ 1 0       90      0   72.5kb   72.5kb
green  open filebeat-2021.09.22               6pRjthCnSJSMyQK_SonNGA 1 0       86      0   88.8kb   88.8kb

Yep, it's here! The index was finally created, the index creation errors disappeared from Elasticsearch logs and data is filling up in this index (since the template was adjusted):

Obviously this template adjustment needs to be done on all the other index templates as well.


Add a comment

Show form to leave a comment

Comments (newest first)

Julian E from Colombia wrote on Jan 6th, 2022:

Thanks, you save my day with this article, cheers.


RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder