After upgrading Elasticsearch from 6.8.x to 7.15.x, xpack.notification settings in elasticsearch.yml resulted in an error during start of Elasticsearch:
java.lang.IllegalArgumentException: unknown setting [xpack.notification.slack.account.my-watcher.url] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
X-Pack allows to configure notifications sent by Elasticsearch directly to Slack. In the past this could be configured by defining an xpack.notification.slack snippet in /etc/elasticsearch/elasticsearch.yml:
# This worked fine with Elasticsearch 6.x
But this very same config now prevents Elasticsearch from starting.
Once more, I ran into a breaking change with the newest Elasticsearch 7.15 release, after seeing a couple of them in the past few days during the TEST upgrade:
But in this situation, concerning the xpack notification, the breaking changes documentation is actually more confusing than straightforward:
By reading this, it actually sounds pretty easy; just replace <url> with <secure_url>, right? Ha! No!
The catch is that secure_url is not allowed to be defined in elasticsearch.yml!
# Do not use this, secure_url is not allowed in elasticsearch.yml!
If you do, the following error will show up during Elasticsearch start (and prevents ES from starting):
[2021-10-12T15:30:55,760][ERROR][o.e.b.Bootstrap ] [elastic01] Exception
java.lang.IllegalArgumentException: Setting [xpack.notification.slack.account.my-watcher.secure_url] is a secure setting and must be stored inside the Elasticsearch keystore, but was found inside elasticsearch.yml
In order to continue, disable the xpack.notification.slack settings from elasticsearch.yml and restart Elasticsearch.
To prevent having some clear text secure information in the config file, certain settings (such as passwords) have been moved into Elasticsearch itself. Whether this is more secure or not is questionable, but that's the case. The "keystore" can be executed on the command line, the binary should be located under /usr/share/elasticsearch/bin/elasticsearch-keystore (at least in deb packages).
The elasticsearch-keystore command is executed with an additional input - setting the "key name", such as it would be set in elasticsearch.yml:
root@elastic01:~# /usr/share/elasticsearch/bin/elasticsearch-keystore add xpack.notification.slack.account.my-watcher.secure_url
warning: usage of JAVA_HOME is deprecated, use ES_JAVA_HOME
Enter value for xpack.notification.slack.account.my-watcher.secure_url: [paste url here]
The command asks for an additional input as a prompt. Copy the previous "url" and paste it here. Note that the input here is not shown (not even with asterisks), as it is considered a secure input (as a password).
Now you can enable the xpack.notification.slack configuration in /etc/elasticsearch/elasticsearch.yml again and do another Elasticsearch restart:
root@elastic01:~# systemctl restart elasticsearch.service
This time, Elasticsearch should start up correctly and read the relevant "secure_url" settings from the keystore.
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Container Containers CouchDB DB DNS Database Docker ELK Elasticsearch Filebeat FreeBSD Galera GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb2 InfluxDB Internet Java KVM Kibana Kodi Kubernetes LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Redis SSL Samba Seafile Security Shell SmartOS Solaris Surveillance SystemD Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder