check_http and OpenSSL 1.1.1 - an evil combo due to backward incompatibility with older OpenSSL versions?

Written by - 0 comments

Published on - last updated on March 23rd 2022 - Listed in TLS SSL Security Monitoring Linux


Most articles I publish here are about technical issues and how to solve them. Sometimes with a real solution, sometimes with a workaround. But I have a feeling that this one will remain a head-scratcher for the years to come...

Monitoring SSL sites with check_http

Of course monitoring HTTPS sites using the monitoring plugin check_http is nothing new and nothing extraordinary. By appending the parameters -S and the optional --sni, a SSL website can be checked:

$ /usr/lib/nagios/plugins/check_http -H www.claudiokuenzler.com -S --sni
HTTP OK: HTTP/1.1 200 OK - 18964 bytes in 0.035 second response time |time=0.035075s;;;0.000000;10.000000 size=18964B;;;0

This also allows to monitor SSL certificates with check_http, but check_ssl_cert is a much better monitoring plugin for this purpose.

check_http 2.2 and OpenSSL 1.1.1: SSL alert number 47

However a major problem can occur, when remote sites are running on a server with an older OpenSSL version.

In the following example, check_http from the monitoring-plugins 2.2 package is compiled against OpenSSL 1.1:

ck@monitoring:~$ /usr/lib/nagios/plugins/check_http -V
check_http v2.2 (monitoring-plugins 2.2)

admck@monitoring:~$ ldd /usr/lib/nagios/plugins/check_http
        linux-vdso.so.1 (0x00007ffc909ed000)
        libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fc8c56bc000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fc8c53e6000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc8c53c3000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc8c51d1000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc8c51cb000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc8c5781000)

A HTTP check on a remote server with a lower OpenSSL version (TBV!) then results in the following error:

ck@monitoring:~$ /usr/lib/nagios/plugins/check_http -I 192.168.33.51 -p 7004 -S
CRITICAL - Cannot make SSL connection.
140634064565568:error:14094417:SSL routines:ssl3_read_bytes:sslv3 alert illegal parameter:../ssl/record/rec_layer_s3.c:1543:SSL alert number 47

This error basically means that a SSL handshake could not be established.

Note: In July 2021 I wrote a related article which also handled about the SSL alert number 47 but on a Nginx reverse proxy setup. But back then the problem was related to disabled ciphers - which could be enabled again in /etc/ssl/openssl.cnf. Unfortunately the CipherString workaround does not work in this case.

Verification with testssl

The first guess from reading the error message above would lead to an assumption that sslv3 was used, right? But even though SSLv3 is indeed enabled by the remote server, so is TLS 1.2 - which should be the preferred protocol. This can be verified with SSL scanners, such as testssl.sh:

ck@monitoring:~/testssl.sh-3.0.7$ ./testssl.sh https://192.168.33.51:7004

###########################################################
    testssl.sh       3.0.7 from https://testssl.sh/

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

Using "OpenSSL 1.0.2-chacha (1.0.2k-dev)" [~183 ciphers]
 on monitoring:./bin/openssl.Linux.x86_64
 (built: "Jan 18 17:12:17 2019", platform: "linux-x86_64")


 Start 2022-03-17 11:25:18        -->> 192.168.33.51:7004 <<--

 rDNS (192.168.33.51):   --
 Service detected:       HTTP


 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)

 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    not offered and downgraded to a weaker protocol
 NPN/SPDY   not offered
 ALPN/HTTP2 not offered
[...]

From the output we can see that testssl is using its own internal OpenSSL version (1.0.2k) to establish a connection to the remote server. The script also detects that three protocols are supported by the remote server: SSLv2, SSLv3 and TLS 1.2.

At the very bottom of the testssl output, a couple of communication simulations are happening. Most interestingly here is the fact, that a connection with OpenSSL 1.1.1d is simulated - and fails:

[...]
Java 8u161                   TLSv1.2 ECDHE-RSA-AES256-SHA384, 256 bit ECDH (P-256)
 Java 11.0.2 (OpenJDK)        TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Java 12.0.1 (OpenJDK)        TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 OpenSSL 1.0.2e               TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 OpenSSL 1.1.0l (Debian)      TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)

 OpenSSL 1.1.1d (Debian)      No connection
 Thunderbird (68.3)           TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)

With that in mind, we can also use testssl to use the system's openssl command, which is 1.1.1f on this Ubuntu 20.04 machine:

ck@monitoring:~/testssl.sh-3.0.7$ ./testssl.sh --openssl=/usr/bin/openssl https://192.168.33.51:7004

No engine or GOST support via engine with your /usr/bin/openssl

###########################################################
    testssl.sh       3.0.7 from https://testssl.sh/

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~79 ciphers]
 on monitoring:/usr/bin/openssl

 (built: "Mar  9 12:12:45 2022", platform: "debian-amd64")


 Start 2022-03-17 11:34:50        -->> 192.168.33.51:7004 <<--

 rDNS (192.168.33.51):   --
 Your OpenSSL cannot connect to 192.168.33.51:7004
 The results might look ok but they could be nonsense. Really proceed ? ("yes" to continue) --> yes
 Service detected:       Couldn't determine what's running on port 7004, assuming no HTTP service => skipping all HTTP checks

testssl now uses OpenSSL 1.1.1f with much less ciphers than before. And it is unable to connect to the remote server.

It does indeed look as if OpenSSL 1.1.1 is incompatible with lower OpenSSL versions. That's very bad news, especially for a monitoring server!

What about an older check_http?

If you've been in the monitoring field as long as I am (17 years meanwhile), you probably have some of the older plugin versions laying around. In this case, I still have an older check_http from the monitoring-plugins package 2.1.2 around:

ck@monitoring:~$ /usr/lib/nagios/plugins/check_http.2.1.2 -V
check_http v2.1.2 (monitoring-plugins 2.1.2)

This older check_http was compiled against OpenSSL 1.0.x:

ck@monitoring:~$ ldd /usr/lib/nagios/plugins/check_http.2.1.2
        linux-vdso.so.1 (0x00007ffc39d48000)
        libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f6712364000)
        libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f6711f20000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6711efd000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6711d0b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6711d05000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f67125d7000)

Note: If you've upgraded your Ubuntu from previous versions, you might be in luck and libssl.so.1.0.0 is still on the system. If not, you can take a look at this question on askubuntu and install the older ssl library from a downloaded deb package.

As this check_http version uses the OpenSSL library in version 1.0.x (and not 1.1.x), how does the HTTPS check look like? Let's find out:

ck@monitoring:~$ /usr/lib/nagios/plugins/check_http.2.1.2 -I 192.168.33.51 -p 7004 -S
HTTP WARNING: HTTP/1.1 404 Not Found - 1308 bytes in 0.051 second response time |time=0.051490s;;;0.000000 size=1308B;;;0

It works! Well, with a 404 return code, but the connection could be established!

It's not only check_http

Affected by this problem, what seems to be an OpenSSL incompatibility, is not only check_http. Basically everything which uses the OpenSSL library in the background, will eventually run into this problem.

This can also be tested with check_ssl_cert. This monitoring plugin uses by default the system's openssl command in the background to obtain and verify certificates. Run against an older web server with OpenSSL 1.0.1f, this results in the following error:

ck@monitoring:~$ /usr/lib/nagios/plugins/check_ssl_cert -H 192.168.33.164 --sni www.example.com
SSL_CERT CRITICAL www.example.com: SSL error: 140715112686912:error:1425F102:SSL routines:ssl_choose_client_version:unsupported protocol:../ssl/statem/statem_lib.c:1941:

But check_ssl_cert has a clever workaround: It allows to specify another openssl command by using the --openssl parameter. Remember testssl from before? This included an openssl binary with version 1.0.2k:

ck@monitoring:~$ /usr/lib/nagios/plugins/check_ssl_cert -H 192.168.33.164 --sni www.example.com --openssl /home/ck/testssl.sh-3.0.7/bin/openssl.Linux.x86_64
SSL_CERT OK - x509 certificate '*.example.com' from 'Gandi Standard SSL CA 2' valid until Feb  4 23:59:59 2023 GMT (expires in 324 days)|days_chain_elem1=324;20;15;; days_chain_elem2=909;20;15;; days_chain_elem3=5786;20;15;;

And - oh wonder - this check works again. Magic!

OpenSSL backward compatibility gone?

The verifications with testssl.sh before would indicate that communication with an older OpenSSL (library) version would work:

OpenSSL 1.0.x -> 1.1.1 = works

But if the client runs OpenSSL 1.1.1 and wants to connect to a server with a lower OpenSSL version, this seems to break now:

OpenSSL 1.1.1 -> 1.0.x = no connection

According to the OpenSSL SecurityLevel documentation, the compatibility with older OpenSSL versions can be established by setting the security level to 0:

Level 0
Everything is permitted. This retains compatibility with previous versions of OpenSSL.

But unfortunately this doesn't work.

Right now I'm left pretty baffled and negatively surprised that something so important (openssl backward compatibility) seems to be gone. In an environment where you need to monitor a lot of servers, including legacy systems (which nobody likes, but they do exist) you want to make sure you can continue to monitor your servers. Doesn't matter which OpenSSL they run.

Hence my question to you, dear reader: What did you do in your monitoring environment to tackle this problem? Please leave a comment.

Shake it off and focus!

Alright. Meanwhile, a couple of hours after I let out this rant of possible OpenSSL backward incompatibility, focus returned to my mind. Let's get to the bottom of these things. 

check_ssl_cert reason: Outdated protocols

First, the certificate check (using check_ssl_cert) which failed on a particular host (192.168.33.164). By using testssl.sh I was able to see that this web server only provided SSLv2, SSLv3 and TLSv1 protocols to communicate with:

ck@monitoring:~/testssl.sh-3.0.7# ./testssl.sh https://192.168.33.164
[...]
Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      offered (NOT ok)
 TLS 1      offered (deprecated)

 TLS 1.1    not offered
 TLS 1.2    not offered and downgraded to a weaker protocol
 TLS 1.3    not offered and downgraded to a weaker protocol
 NPN/SPDY   http/1.1 (advertised)
 ALPN/HTTP2 not offered
[...]

The problem with OpenSSL 1.1.1? These protocols are disabled in openssl itself and can't just be enabled in a config file.

After enabling TLS 1.1 and 1.2 on this legacy web server, testssl can now see these protocols, too:

ck@monitoring:~/testssl.sh-3.0.7# ./testssl.sh https://192.168.33.164
[...]
Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      offered (NOT ok)
 TLS 1      offered (deprecated)
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    not offered and downgraded to a weaker protocol
 NPN/SPDY   http/1.1 (advertised)
 ALPN/HTTP2 not offered
[...]

The check_ssl_cert command now works with the default openssl command again (without having to specify an older openssl path):

ck@monitoring:~# /usr/lib/nagios/plugins/check_ssl_cert -H 192.168.33.164 --sni www.example.com
SSL_CERT OK - x509 certificate '*.example.com' from 'Gandi Standard SSL CA 2' valid until Feb  4 23:59:59 2023 GMT (expires in 324 days)|days_chain_elem1=324;20;15;; days_chain_elem2=909;20;15;; days_chain_elem3=5786;20;15;;

check_http and the SSL alert number 47

Back to the initial problem: The check_http failing against a web server, even with TLS 1.2 protocol. 

Interestingly, there are two machines of that application. Both are supposed to be set up in the exact same way. Yet the HTTPS checks using check_http 2.2 with OpenSSL 1.1.1 continued to work on the second machine. Why would the check stop working on machine 1 but work on machine 2?

ck@monitoring:~# /usr/lib/nagios/plugins/check_http -I 192.168.33.51 -p 7004 -S
CRITICAL - Cannot make SSL connection.
139827283326272:error:14094417:SSL routines:ssl3_read_bytes:sslv3 alert illegal parameter:../ssl/record/rec_layer_s3.c:1543:SSL alert number 47

ck@monitoring:~# /usr/lib/nagios/plugins/check_http -I 192.168.33.52 -p 7004 -S
HTTP WARNING: HTTP/1.1 404 Not Found - 1308 bytes in 0.049 second response time |time=0.049114s;;;0.000000;10.000000 size=1308B;;;0

We decided to run a cipher scan using nmap on both application servers.

ck@monitoring:~# nmap --script ssl-enum-ciphers -p 7004 192.168.33.51
ck@monitoring:~# nmap --script ssl-enum-ciphers -p 7004 192.168.33.52

Both results showed the same ciphers, however with one difference. The first server with IP ending 51 (with the OpenSSL communication problem) showed a SHA1 signature:

|     warnings:
|       Key exchange (secp192r1) of lower strength than certificate key
|       Weak certificate signature: SHA1

testssl.sh was also used again, this time with a focus on this SHA1 signature. And this SHA1 key can indeed be found in the signature algorithm:

ck@monitoring:~/testssl.sh-3.0.7# ./testssl.sh https://192.168.33.51:7004
[...]
 Testing server defaults (Server Hello)

 TLS extensions (standard)    "EC point formats/#11" "renegotiation info/#65281" "max fragment length/#1"
 Session Ticket RFC 5077 hint no -- no lifetime advertised
 SSL Session ID support       yes
 Session Resumption           Tickets no, ID: no
 TLS clock skew               -4 sec from localtime
 Signature Algorithm          SHA1 with RSA -- besides: users will receive a strong browser WARNING
 Server key size              RSA 2048 bits
 Server key usage             --
 Server extended key usage    --
[...]

Using testssl.sh on the second application server reveals a different signature algorithm with a much newer SHA256 key:

ck@monitoring:~/testssl.sh-3.0.7# ./testssl.sh https://192.168.33.52:7004
[...]
Testing server defaults (Server Hello)

 TLS extensions (standard)    "EC point formats/#11" "renegotiation info/#65281" "max fragment length/#1"
 Session Ticket RFC 5077 hint no -- no lifetime advertised
 SSL Session ID support       yes
 Session Resumption           Tickets no, ID: no
 TLS clock skew               -4 sec from localtime
 Signature Algorithm          SHA256 with RSA
 Server key size              RSA 2048 bits
 Server key usage             --
 Server extended key usage    --
[...]

SHA1 is considered weak and insecure for many years. Could this be the reason why a communication would not work with OpenSSL 1.1.1?

Once the key was replaced by a newer SHA256 key, testssl.sh now showed this newer key in the signature algorithm:

 Testing server defaults (Server Hello)

 TLS extensions (standard)    "EC point formats/#11" "renegotiation info/#65281" "max fragment length/#1"
 Session Ticket RFC 5077 hint no -- no lifetime advertised
 SSL Session ID support       yes
 Session Resumption           Tickets no, ID: no
 TLS clock skew               -4 sec from localtime
 Signature Algorithm          SHA256 with RSA
 Server key size              RSA 2048 bits
 Server key usage             Digital Signature, Key Encipherment
 Server extended key usage    TLS Web Client Authentication, TLS Web Server Authentication

What about check_http? Does the check work now?

ck@monitoring:~$ /usr/lib/nagios/plugins/check_http -I 192.168.33.51 -p 7004 -S
HTTP WARNING: HTTP/1.1 404 Not Found - 1308 bytes in 0.047 second response time |time=0.046706s;;;0.000000;10.000000 size=1308B;;;0

The HTTP check with check_http now works again!

TL;DR

Both communication problems using the newer check_http with OpenSSL 1.1.1 could be resolved by modifying the target servers (to be monitored) to use newer TLS protocols and by replacing old SHA1 keys.


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder