After a monitoring (satellite) server was upgraded from Debian 9 (Stretch) to 10 (Buster), the hardware monitoring of HP servers (using ILO3) stopped working.
The plugin used for monitoring the HP hardware health is check_ilo2_health, an excellent monitoring plugin created and maintained since 2007 by Alexander Greiner-Baer, which uses the ILO (Integrated Lights Out) management interface of HP servers.
After the Debian distribution upgrade, the ILO hardware checks stopped working. On the command line the error could be reproduced:
root@buster:~# /usr/lib/nagios/plugins/check_ilo2_health.pl -H iloip -3 -u monitoring -p secret -a -c -d -o -W -t 60
ILO2_HEALTH UNKNOWN - ERROR: Failed to establish SSL connection with iloip:443 SSL connect attempt failed error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol.
The Debian upgrade from Stretch to Buster includes a major change in the OpenSSL version and its settings:
Besides this, the OpenSSL configuration (/etc/ssl/openssl.cnf) sets a minimum protocol of TLSv1.2 since the upgrade:
root@buster:~# tail /etc/ssl/openssl.cnf
# identifier (optional, default: sha1)
[default_conf]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2
The problem with this? ILO3 is too old and does not support TLSv1.2. And HP refused to create a new ILO3 firmware which would enable a newer TLS protocol:
HPE does not plan for support to be added to iLO 3 because the HPE ProLiant G7-series servers are out of the product support timeframe.
Use a web browser that still supports TLSv1.0 and TLSv1.1 or use the Remote Insight Board Command Language (RIBCL) and Command Line Interface (CLI) SSH to access iLO 3.
Welcome in the land of legacy.
The OpenSSL settings were adjusted to use a lower minimum protocol (TLSv1.0):
root@buster:~# tail /etc/ssl/openssl.cnf
# identifier (optional, default: sha1)
[default_conf]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
MinProtocol = TLSv1.0
CipherString = DEFAULT@SECLEVEL=2
Right after this change, the hardware checks with check_ilo2_health started working again:
root@buster:~# /usr/lib/nagios/plugins/check_ilo2_health.pl -H iloip -3 -u monitoring -p secret -a -c -d -o -W -t 60
ILO2_HEALTH OK - (Board-Version: ILO>=3) Power Usage: 218 Watts, Temperatures: Temp_1 (OK): 28, Temp_2 (OK): 40, Temp_3 (OK): 40, Temp_4 (OK): 44, Temp_5 (OK): 43, Temp_6 (OK): 52, Temp_7 (OK): 51, Temp_8 (OK): 52, Temp_9 (OK): 47, Temp_10 (OK): 53, Temp_11 (OK): 45, Temp_12 (OK): 54, Temp_19 (OK): 31, Temp_20 (OK): 39, Temp_21 (OK): 39, Temp_22 (OK): 37, Temp_23 (OK): 46, Temp_24 (OK): 43, Temp_25 (OK): 44, Temp_26 (OK): 46, Temp_30 (OK): 77 | power=218;; Temp_1=28;41;45 Temp_2=40;82;83 Temp_3=40;82;83 Temp_4=44;87;92 Temp_5=43;87;92 Temp_6=52;87;92 Temp_7=51;87;92 Temp_8=52;90;95 Temp_9=47;65;70 Temp_10=53;90;95 Temp_11=45;70;75 Temp_12=54;90;95 Temp_19=31;70;75 Temp_20=39;70;75 Temp_21=39;80;85 Temp_22=37;80;85 Temp_23=46;77;82 Temp_24=43;70;75 Temp_25=44;70;75 Temp_26=46;70;75 Temp_30=77;110;115
The fix mentioned above works fine for Debian 10 (Buster) and 11 (Bullseye) but does not work for Debian 12 (Bookworm), which now has OpenSSL 3.0.x installed. Another workaround needs to be applied here.
First you can verify that the check_ilo2_health.pl can't establish a SSL connection:
root@bookworm:~# /usr/lib/nagios/plugins/check_ilo2_health.pl -H iloip -3 -u monitoring -p secret -a -c -d -o -W -t 60
ILO2_HEALTH UNKNOWN - ERROR: Failed to establish SSL connection with 192.168.11.11:443 SSL connect attempt failed error:0A000152:SSL routines::unsafe legacy renegotiation disabled.
Create a separate openssl.cnf configuration file (I'm using /usr/lib/nagios/plugins/openssl.cnf) with the following contents:
root@bookworm:~# cat /usr/lib/nagios/plugins/openssl.cnf
openssl_conf = openssl_init
[openssl_init]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
MinProtocol = TLSv1.0
Options = UnsafeLegacyRenegotiation
CipherString = DEFAULT@SECLEVEL=0
Tell the systen to use this SSL config in your Shell session by setting the OPENSSL_CONF environment variable:
root@bookworm:~# export OPENSSL_CONF=/usr/lib/nagios/plugins/openssl.cnf
From the OpenSSL documentation:
OPENSSL_CONF
The path to the config file. Ignored in set-user-ID and set-group-ID programs.
Now the plugin works again:
root@bookworm:~# /usr/lib/nagios/plugins/check_ilo2_health.pl -H iloip -3 -u monitoring -p secret -a -c -d -o -W -t 60
ILO2_HEALTH OK - (Board-Version: ILO>=3) Power Usage: 214 Watts, Temperatures: Temp_1 (OK): 24, Temp_2 (OK): 40, Temp_3 (OK): 40, Temp_4 (OK): 40, Temp_5 (OK): 43, Temp_6 (OK): 47, Temp_7 (OK): 48, Temp_8 (OK): 53, Temp_9 (OK): 46, Temp_10 (OK): 52, Temp_11 (OK): 43, Temp_12 (OK): 52, Temp_19 (OK): 29, Temp_20 (OK): 37, Temp_21 (OK): 37, Temp_22 (OK): 35, Temp_23 (OK): 46, Temp_24 (OK): 44, Temp_25 (OK): 42, Temp_26 (OK): 44, Temp_30 (OK): 77 | power=214;; Temp_1=24;41;45 Temp_2=40;82;83 Temp_3=40;82;83 Temp_4=40;87;92 Temp_5=43;87;92 Temp_6=47;87;92 Temp_7=48;87;92 Temp_8=53;90;95 Temp_9=46;65;70 Temp_10=52;90;95 Temp_11=43;70;75 Temp_12=52;90;95 Temp_19=29;70;75 Temp_20=37;70;75 Temp_21=37;80;85 Temp_22=35;80;85 Temp_23=46;77;82 Temp_24=44;70;75 Temp_25=42;70;75 Temp_26=44;70;75 Temp_30=77;110;115
After another upgrade to Debian 13 (Trixie) with OpenSSL 3.5, there's yet another fix that's needed to make check_ilo2_health.pl run against ILO3.
Even with the previous custom openssl.cnf, the monitoring plugin would throw the following error:
cktrixie:~$ /usr/lib/nagios/plugins/check_ilo2_health.pl -H iloip -3 -u monitoring -p secret -a -c -d -o -W -t 60
ILO2_HEALTH UNKNOWN - ERROR: Failed to establish SSL connection with iloip:443 SSL connect attempt failed error:0A000438:SSL routines::tlsv1 alert internal error.
One reason is that the previously used Options = UnsafeLegacyRenegotiation in the custom OpenSSL configuration has been replaced by UnsafeLegacyServerConnect. As mentioned in this AskUbuntu question:
For SSL v3.0.2 and older Options = UnsafeLegacyRenegotiation
For SSL v3.0.4+ Options = UnsafeLegacyServerConnect
That means the custom openssl.cnf needs to be adjusted accordingly for the newer OpenSSL version:
cktrixie:~$ cat /usr/lib/nagios/plugins/openssl.cnf
openssl_conf = openssl_init
[openssl_init]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
Options = UnsafeLegacyServerConnect
MinProtocol = TLSv1.0
CipherString = DEFAULT@SECLEVEL=0
But that alone is not enough. On Trixie, the check_ilo2_health.pl plugin needs to be executed with additional ssl options, using the plugin's already prepared --sslopts parameter:
cktrixie:~$ export OPENSSL_CONF=/usr/lib/nagios/plugins/openssl.cnf
cktrixie:~$ /usr/lib/nagios/plugins/check_ilo2_health.pl -H iloip -3 -u monitoring -p secret -a -c -d -o -W -t 60 --sslopts 'SSL_verify_mode => SSL_VERIFY_NONE, SSL_version => "TLSv1_1"'
ILO2_HEALTH OK - (Board-Version: ILO>=3) Power Usage: 228 Watts, Temperatures: Temp_1 (OK): 24, Temp_2 (OK): 40, Temp_3 (OK): 40, Temp_4 (OK): 41, Temp_5 (OK): 44, Temp_6 (OK): 46, Temp_7 (OK): 47, Temp_8 (OK): 52, Temp_9 (OK): 45, Temp_10 (OK): 52, Temp_11 (OK): 43, Temp_12 (OK): 52, Temp_19 (OK): 29, Temp_20 (OK): 36, Temp_21 (OK): 37, Temp_22 (OK): 35, Temp_23 (OK): 46, Temp_24 (OK): 44, Temp_25 (OK): 41, Temp_26 (OK): 43, Temp_30 (OK): 77 | power=228;; Temp_1=24;41;45 Temp_2=40;82;83 Temp_3=40;82;83 Temp_4=41;87;92 Temp_5=44;87;92 Temp_6=46;87;92 Temp_7=47;87;92 Temp_8=52;90;95 Temp_9=45;65;70 Temp_10=52;90;95 Temp_11=43;70;75 Temp_12=52;90;95 Temp_19=29;70;75 Temp_20=36;70;75 Temp_21=37;80;85 Temp_22=35;80;85 Temp_23=46;77;82 Temp_24=44;70;75 Temp_25=41;70;75 Temp_26=43;70;75 Temp_30=77;110;115
And eureka, the plugin works again!
If you are using Icinga, you can add the custom OPENSSL_CONF environment variable right into the CheckCommand definition of the plugin:
root@icinga2:~# cat /etc/icinga2/zones.d/global-templates/commands/check_ilo2_health.conf
# check_ilo2_health definition added 20190731 by Claudio Kuenzler
# 20240711 added OPENSSL_CONF env (https://www.claudiokuenzler.com/blog/1199/ilo3-hp-monitoring-stopped-working-debian-upgrade-tls-protocol)
object CheckCommand "check_ilo2_health" {
import "plugin-check-command"
command = [ PluginDir + "/check_ilo2_health.pl" ]
arguments = {
"-H" = "$ilo_address$"
"-u" = "$ilo_user$"
"-p" = "$ilo_password$"
"-t" = "$ilo_timeout$"
"-a" = { set_if = "$ilo_fan_redundancy$" }
"-c" = { set_if = "$ilo_check_drives$" }
"-d" = { set_if = "$ilo_perfdata$" }
"-3" = { set_if = "$ilo_ilo3$" }
"-o" = { set_if = "$ilo_power_redundancy$" }
"-W" = { set_if = "$ilo_power_usage$" }
"-l" = { set_if = "$ilo_check_eventlog$" }
"--sslopts" = "$ilo_sslopts$"
}
vars.ilo_address = "$host.vars.iloaddress$"
vars.ilo_user = "monitoring"
vars.ilo_password = "secret"
vars.ilo_fan_redundancy = true
vars.ilo_check_drives = true
vars.ilo_perfdata = true
vars.ilo_ilo3 = true
vars.ilo_power_redundancy = true
vars.ilo_power_usage = true
vars.ilo_check_eventlog = false
vars.ilo_timeout = "60"
vars.ilo_sslopts = "SSL_verify_mode => SSL_VERIFY_NONE, SSL_version => \"TLSv1_1\""
env.OPENSSL_CONF = "/usr/lib/nagios/plugins/openssl.cnf"
}
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Observability Office OpenSearch PHP Perl Personal PostgreSQL PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder