Elasticsearch failed on parsing mappings on index creation ([_default_] mappings are not allowed)

Written by Claudio Kuenzler - 1 comments

Published on October 5th 2021 - Listed in Elasticsearch ELK

---

And here's one more article based on upgrading Elasticsearch from 6.8.6 to 7.15.0, after the previous articles:

• Elasticsearch X-Pack error after upgrade: Incorrect authc realm settings found
• Elasticsearch error: The index was created with version [x.x.x] but the minimum compatible version is [x.x.x]

This time the errors were only spotted a day after the Elasticsearch upgrade, when Elasticsearch (being part of an ELK stack) wanted to create new indices based on the current date:

[2021-10-01T14:07:24,982][INFO ][o.e.c.m.MetadataCreateIndexService] [elk01] failed on parsing mappings on index creation [rancher-vamp-2021.10.01]
java.lang.IllegalArgumentException: [_default_] mappings are not allowed on new indices and should no longer be used. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#default-mapping-not-allowed] for more information.

[2021-10-01T14:07:24,983][INFO ][o.e.c.m.MetadataCreateIndexService] [elk01] failed on parsing mappings on index creation [filebeat-2021.10.01]
java.lang.IllegalArgumentException: [_default_] mappings are not allowed on new indices and should no longer be used. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#default-mapping-not-allowed] for more information.

The same error was logged thousands of times - every time Elasticsearch wanted to create a new index from the incoming logs (sent by Logstash). The error mentions that a [_default_] mapping is not allowed anymore and therefore the index cannot be created.

_default_ mapping?

Based on the error on above on the filebeat-2021.10.01 index, let's take a closer look at the filebeat index template:

root@elk01:~# curl -X GET "localhost:9200/_template/filebeat?pretty"
{
  "filebeat" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "retention-10d"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

There's nothing at all defined inside the "mappings" context! So why does Elasticsearch think, there's a _default_ mapping defined somewhere?

Actually the Elastic 7.0 breaking changes documentation mentions something interesting concerning the mappings:

Note that in 7.x, the get template API does not show the _default_ mapping by default, even when it is defined in the mapping. To see all mappings in the template, the include_type_name parameter must be supplied:
GET /_template/my_template?include_type_name

Alright, let's add this parameter then:

root@elk01:~# curl -X GET "localhost:9200/_template/filebeat?pretty&include_type_name"
{
  "filebeat" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "retention-10d"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [
          {
            "message_field" : {
              "path_match" : "message",
              "mapping" : {
                "norms" : false,
                "type" : "text"
              },
              "match_mapping_type" : "string"
            }
          },
          {
            "string_fields" : {
              "mapping" : {
                "norms" : false,
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              },
              "match_mapping_type" : "string",
              "match" : "*"
            }
          }
        ],
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "geoip" : {
            "dynamic" : true,
            "properties" : {
              "ip" : {
                "type" : "ip"
              },
              "latitude" : {
                "type" : "half_float"
              },
              "location" : {
                "type" : "geo_point"
              },
              "longitude" : {
                "type" : "half_float"
              }
            }
          },
          "@version" : {
            "type" : "keyword"
          }
        }
      }
    },
    "aliases" : { }
  }
}

Whoa, there's much more data coming back now! And indeed, the _default_ mapping is set in this filebeat template. The solution, according to Elastic's documentation, is to remove the mapping from the index template.

Adjusting the template

Changing an Elasticsearch template basically means the template is first "downloaded" from the Elasticsearch API using a GET request, modified in a local file and the final file with the modifications is uploaded back into the Elasticsearch API using the PUT method. An alternative is of course to not use a temporary file but push everything in one curl command with a very large -d "data" part - but this is much more complicated and prone to (human) errors.

First download the template, with the additional include_type_name parameter:

root@elk01:~# curl -X GET "localhost:9200/_template/filebeat?pretty&include_type_name" > /tmp/template-filebeat-20211001

Now modify the file (/tmp/template-filebeat-20211001) with your favorite editor (vim!) and within mappings {} remove the whole _default_ context. Also make sure you remove the top-level context with the template's name (filebeat {}). The file should then look something like this:

root@elk01:~# cat /tmp/template-filebeat-20211001
{
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "retention-10d"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : { },
    "aliases" : { }
}

Note: In this case this Elasticsearch is part of a TEST ELK setup, with only one Elasticsearch node. Hence the limited number_of_shards and number_of_replicas values.

Now this modified file can be uploaded back into the Elasticsearch API, to replace the existing filebeat template:

root@elk01:~# curl -H "Content-Type: application/json" -X PUT localhost:9200/_template/filebeat -d "@/tmp/template-filebeat-20211001"
{"acknowledged":true}

Verify that the new template doesn't contain the _default_ mapping anymore:

root@elk01:~# curl -X GET "localhost:9200/_template/filebeat?pretty&include_type_name"
{
  "filebeat" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "retention-10d"
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "number_of_replicas" : "0"
      }
    },
    "mappings" : {
      "_doc" : { }
    },
    "aliases" : { }
  }
}

Done - this looks good!

Are we there yet? Does it work?

So what happens now in Elasticsearch? Is the new filebeat-2021-10-01 index created?

root@elk01:~# curl -s http://localhost:9200/_cat/indices?pretty | grep filebeat
green  open filebeat-2021.10.01               TuQ_qNCUQuO_WBXyvHF8ug 1 0    16878      0    3.5mb    3.5mb
green  open filebeat-2021.09.16               wGkmLqK5SJGT5rI-WfvGyQ 1 0    77037      0   14.6mb   14.6mb
green  open filebeat-2021.09.17               i_v0Wr3YRIOLIeWu3gppLw 1 0      141      0  118.3kb  118.3kb
green  open filebeat-2021.09.18               SnbjHKCES72IcKo2w-7uug 1 0       86      0     70kb     70kb
green  open filebeat-2021.09.19               ztA6QXi0R9SVYYrrHEDwtQ 1 0       84      0   68.6kb   68.6kb
green  open filebeat-2021.09.15               h6QwsW-qSz6WNYw5jcIJWA 1 0   615768      0  126.2mb  126.2mb
green  open filebeat-2021.09.30               LKiXjStIQOCrAuat69vncQ 1 0 16745885      0    2.2gb    2.2gb
green  open filebeat-2021.09.27               UVppic3QQWOIIyM7BlBYqw 1 0 16791098      0    2.7gb    2.7gb
green  open filebeat-2021.09.28               PulI3NlmTw-WvtgBTKnWmA 1 0 37654204      0    8.1gb    8.1gb
green  open filebeat-2021.09.29               p2-0aSsOSrWRpGALe2Lg4A 1 0 16755970      0    2.7gb    2.7gb
green  open filebeat-2021.09.23               5unlKkY6T4KVUU4zD16A6Q 1 0       85      0     69kb     69kb
green  open filebeat-2021.09.24               nwu0DlxVQNiDxbSoZTislQ 1 0  1659948      0  858.5mb  858.5mb
green  open filebeat-2021.09.25               QotuSLyZT5Cp6DaDF6hXkw 1 0 39875298      0    5.8gb    5.8gb
green  open filebeat-2021.09.26               r6uczxqjRISiIKHNpCqWJw 1 0 16723026      0    2.7gb    2.7gb
green  open filebeat-2021.09.20               wAwV8gFdR6G0Gz_7c2vs1g 1 0      195      0  148.5kb  148.5kb
green  open filebeat-2021.09.21               5ae0Z_0SR0KEqPcwCqkpnQ 1 0       90      0   72.5kb   72.5kb
green  open filebeat-2021.09.22               6pRjthCnSJSMyQK_SonNGA 1 0       86      0   88.8kb   88.8kb

Yep, it's here! The index was finally created, the index creation errors disappeared from Elasticsearch logs and data is filling up in this index (since the template was adjusted):

Obviously this template adjustment needs to be done on all the other index templates as well.

Add a comment

Show form to leave a comment

Comments (newest first)

Julian E from Colombia wrote on Jan 6th, 2022:

Thanks, you save my day with this article, cheers.

---