Elasticsearch: unknown setting [xpack notification slack account my-watcher url]

Written by - 0 comments

Published on - Listed in Elasticsearch ELK

After upgrading Elasticsearch from 6.8.x to 7.15.x, xpack.notification settings in elasticsearch.yml resulted in an error during start of Elasticsearch:

java.lang.IllegalArgumentException: unknown setting [xpack.notification.slack.account.my-watcher.url] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

Slack notifications with X-Pack

X-Pack allows to configure notifications sent by Elasticsearch directly to Slack. In the past this could be configured by defining an xpack.notification.slack snippet in /etc/elasticsearch/elasticsearch.yml:

# This worked fine with Elasticsearch 6.x
      url: https://hooks.slack.com/services/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
        from: watcher

But this very same config now prevents Elasticsearch from starting.

Breaking change!

Once more, I ran into a breaking change with the newest Elasticsearch 7.15 release, after seeing a couple of them in the past few days during the TEST upgrade:

But in this situation, concerning the xpack notification, the breaking changes documentation is actually more confusing than straightforward:

By reading this, it actually sounds pretty easy; just replace <url> with <secure_url>, right? Ha! No!

The catch is that secure_url is not allowed to be defined in elasticsearch.yml!

# Do not use this, secure_url is not allowed in elasticsearch.yml!
      secure_url: https://hooks.slack.com/services/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
        from: watcher

If you do, the following error will show up during Elasticsearch start (and prevents ES from starting):

[2021-10-12T15:30:55,760][ERROR][o.e.b.Bootstrap          ] [elastic01] Exception
java.lang.IllegalArgumentException: Setting [xpack.notification.slack.account.my-watcher.secure_url] is a secure setting and must be stored inside the Elasticsearch keystore, but was found inside elasticsearch.yml

In order to continue, disable the xpack.notification.slack settings from elasticsearch.yml and restart Elasticsearch

Save secure_url in Elasticsearch's keystore

To prevent having some clear text secure information in the config file, certain settings (such as passwords) have been moved into Elasticsearch itself. Whether this is more secure or not is questionable, but that's the case. The "keystore" can be executed on the command line, the binary should be located under /usr/share/elasticsearch/bin/elasticsearch-keystore (at least in deb packages).

The elasticsearch-keystore command is executed with an additional input - setting the "key name", such as it would be set in elasticsearch.yml:

root@elastic01:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.notification.slack.account.my-watcher.secure_url
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Enter value for xpack.notification.slack.account.my-watcher.secure_url: [paste url here]

The command asks for an additional input as a prompt. Copy the previous "url" and paste it here. Note that the input here is not shown (not even with asterisks), as it is considered a secure input (as a password).

Now you can enable the xpack.notification.slack configuration in /etc/elasticsearch/elasticsearch.yml again and do another Elasticsearch restart:

        from: watcher

root@elastic01:~# systemctl restart elasticsearch.service

This time, Elasticsearch should start up correctly and read the relevant "secure_url" settings from the keystore.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder