Fixing Perl script using LWP unable to connect to TLS 1.2 URL; failing with sslv3 alert handshake failure

Written by - 0 comments

Published on November 2nd 2020 - Listed in Perl Coding Security TLS


To monitor Citrix from a user endpoint view (=simulating a real user login), we are using Simon Lauger's check_netscaler_gateway monitoring plugin. This has been working very well for the last couple of years - until the plugin started to return a failure one day:

ckadm@mintp ~ $ ./check_netscaler_gateway.pl -H citrix.example.com -u user -p secret -S STORE_ID
NetScaler Gateway CRITICAL - request to https://citrix.example.com/Citrix/STORE_IDWeb/cgi/login failed with HTTP 500

By using the additional -d parameter (for debug), the real reason why the plugin fails is showing:

ckadm@mintp ~ $ ./check_netscaler_gateway.pl -H citrix.example.com -u user -p secret -S STORE_ID -d
$VAR1 = bless( {
                 '_headers' => bless( {
                                        'client-warning' => 'Internal response',
                                        'content-type' => 'text/plain',
                                        'client-date' => 'Mon, 02 Nov 2020 08:44:09 GMT',
                                        '::std_case' => {
                                                          'client-date' => 'Client-Date',
                                                          'client-warning' => 'Client-Warning',
                                                          'set-cookie2' => 'Set-Cookie2',
                                                          'set-cookie' => 'Set-Cookie'
                                                        }
                                      }, 'HTTP::Headers' ),
                 '_rc' => 500,
                 '_request' => bless( {
                                        '_method' => 'POST',
                                        '_content' => 'login=user&passwd=secret',
                                        '_uri' => bless( do{\(my $o = 'https://citrix.example.com/cgi/login')}, 'URI::https' ),
                                        '_uri_canonical' => $VAR1->{'_request'}{'_uri'},
                                        '_headers' => bless( {
                                                               'referer' => 'https://citrix.example.com/vpn/index.html',
                                                               'accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
                                                               'content-length' => 33,
                                                               'content-type' => 'application/x-www-form-urlencoded',
                                                               'user-agent' => 'libwww-perl/6.15'
                                                             }, 'HTTP::Headers' )
                                      }, 'HTTP::Request' ),
                 '_content' => 'Can\'t connect to citrix.example.com:443

SSL connect attempt failed because of handshake problems error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure at /usr/share/perl5/LWP/Protocol/http.pm line 47.
',
                 '_msg' => 'Can\'t connect to citrix.example.com:443'
               }, 'HTTP::Response' );
NetScaler Gateway CRITICAL - request to https://citrix.example.com/Citrix/STORE_IDWeb/cgi/login failed with HTTP 500

The relevant part, highlighted in bold, shows a protocol problem (handshake problems error). 

But why would a working code suddenly stop working from one day to another? It turns out that on that particular day the plugin started to report CRITICAL, the SSL/TLS versions on the Citrix Netscaler were adjusted. The old SSLv3, TLS1.0 and TLS1.1 protocols were disabled and only TLS1.2 is now accepted. From a security point of view this makes total sense of course - but how can the plugin be fixed?

Comparing different OS

As the error message hints to a problem in Perl's LWP module, the first guess was that an outdated version of LWP was being used. But when the very same monitoring plugin was executed on a slightly newer OS (Debian Stretch), the plugin ran just fine. A quick comparison between the different package versions revealed that it cannot be a problem of LWP:

 Non-working  Working
 Operating System
 Ubuntu 16.04 (xenial)
 Debian 9 (stretch)
 Perl version
 5.22.1  5.24.1
 Perl LWP version
 6.06-2  6.06-2
 Perl SSLeay version
 1.72
 1.80
 OpenSSL version
 1.0.2g
 1.1.0l

Clearly LWP is the exact same version, however OpenSSL and the Perl module SSLeay (Perl extension for using OpenSSL) differ a lot!

It's not the Perl script, it's SSLeay (openssl)

When the plugin / Perl script is executed, SSLeay (basically Openssl) is used in the background. This can be seen with strace, here on Ubuntu xenial:

stat("/etc/perl/Net/SSLeay.pmc", 0x7fff6be9cd90) = -1 ENOENT (No such file or directory)
stat("/etc/perl/Net/SSLeay.pm", 0x7fff6be9ccc0) = -1 ENOENT (No such file or directory)
stat("/usr/local/lib/x86_64-linux-gnu/perl/5.22.1/Net/SSLeay.pmc", 0x7fff6be9cd90) = -1 ENOENT (No such file or directory)
stat("/usr/local/lib/x86_64-linux-gnu/perl/5.22.1/Net/SSLeay.pm", 0x7fff6be9ccc0) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/perl/5.22.1/Net/SSLeay.pmc", 0x7fff6be9cd90) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/perl/5.22.1/Net/SSLeay.pm", 0x7fff6be9ccc0) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/perl5/5.22/Net/SSLeay.pmc", 0x7fff6be9cd90) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/perl5/5.22/Net/SSLeay.pm", {st_mode=S_IFREG|0644, st_size=52995, ...}) = 0
open("/usr/lib/x86_64-linux-gnu/perl5/5.22/Net/SSLeay.pm", O_RDONLY) = 4
ioctl(4, TCGETS, 0x7fff6be9ca70)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR)                   = 0
read(4, "# Net::SSLeay.pm - Perl module f"..., 8192) = 8192

... and later:

read(4, "# Copyright (c) 1997-2007 Graham"..., 8192) = 1324
lseek(4, 1323, SEEK_SET)                = 1323
lseek(4, 0, SEEK_CUR)                   = 1323
close(4)                                = 0
getuid()                                = 901
geteuid()                               = 901
getgid()                                = 901
getegid()                               = 901
read(3, "rust path by default, RT#104759\n"..., 8192) = 8192
brk(0x27b3000)                          = 0x27b3000
getuid()                                = 901
geteuid()                               = 901
getgid()                                = 901
getegid()                               = 901
read(3, "::INET works.  All configuration"..., 8192) = 8192
brk(0x27d4000)                          = 0x27d4000
read(3, "opened'} = -1;\n\t\t$DEBUG>=1 && DE"..., 8192) = 8192
brk(0x27f5000)                          = 0x27f5000
brk(0x2816000)                          = 0x2816000
read(3, "al_ssl_error();\n\t}\n    }\n\n    $D"..., 8192) = 8192
brk(0x2837000)                          = 0x2837000
read(3, "ningful\n\t\t    last;\n\t\t}\n\n\t\t# ini"..., 8192) = 8192
brk(0x285b000)                          = 0x285b000
read(3, "         => 1,\n\t},\n    );\n\n    f"..., 8192) = 8192
brk(0x287e000)                          = 0x287e000
read(3, "ertificate failed because\n# host"..., 8192) = 8192

brk(0x28a0000)                          = 0x28a0000
brk(0x28c1000)                          = 0x28c1000

Comparing this to the strace outpout on Debian stretch:

stat("/etc/perl/Net/SSLeay.pmc", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/etc/perl/Net/SSLeay.pm", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/local/lib/x86_64-linux-gnu/perl/5.24.1/Net/SSLeay.pmc", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/local/lib/x86_64-linux-gnu/perl/5.24.1/Net/SSLeay.pm", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/perl/5.24.1/Net/SSLeay.pmc", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/local/share/perl/5.24.1/Net/SSLeay.pm", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/perl5/5.24/Net/SSLeay.pmc", 0x7ffdbbb7d080) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64-linux-gnu/perl5/5.24/Net/SSLeay.pm", {st_mode=S_IFREG|0644, st_size=52995, ...}) = 0
open("/usr/lib/x86_64-linux-gnu/perl5/5.24/Net/SSLeay.pm", O_RDONLY) = 4
ioctl(4, TCGETS, 0x7ffdbbb7ce50)        = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR)                   = 0
read(4, "# Net::SSLeay.pm - Perl module f"..., 8192) = 8192

And later:

read(4, "# Copyright (c) 1997-2007 Graham"..., 8192) = 1410
lseek(4, 1409, SEEK_SET)                = 1409
lseek(4, 0, SEEK_CUR)                   = 1409
close(4)                                = 0
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
read(3, "all_digests();\n\tNet::SSLeay::ran"..., 8192) = 8192
brk(0x55c3c5c37000)                     = 0x55c3c5c37000
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
read(3, "NSSL_DIR. Unfortunately it is no"..., 8192) = 8192
brk(0x55c3c5c58000)                     = 0x55c3c5c58000
brk(0x55c3c5c57000)                     = 0x55c3c5c57000
read(3, "->{PeerAddr} || $arg_hash->{Peer"..., 8192) = 8192
brk(0x55c3c5c7a000)                     = 0x55c3c5c7a000
read(3, "\n\t$SSL_OBJECT{$ssl} = [$socket,1"..., 8192) = 8192
brk(0x55c3c5c9b000)                     = 0x55c3c5c9b000
read(3, "et::SSLeay::peek($ssl,1);\n\tif ( "..., 8192) = 8192
brk(0x55c3c5cbc000)                     = 0x55c3c5cbc000
brk(0x55c3c5cdd000)                     = 0x55c3c5cdd000
read(3, "} else {\n    *peer_certificates "..., 8192) = 8192
brk(0x55c3c5cfe000)                     = 0x55c3c5cfe000
read(3, "my ($self,$algo,$cert,$key_only)"..., 8192) = 8192
brk(0x55c3c5d1f000)                     = 0x55c3c5d1f000
brk(0x55c3c5d40000)                     = 0x55c3c5d40000
brk(0x55c3c5d3f000)                     = 0x55c3c5d3f000
read(3, "eaken($handle);\n    bless \\$hand"..., 8192) = 8192
brk(0x55c3c5d60000)                     = 0x55c3c5d60000
read(3, "least one\n\t# buffer was written "..., 8192) = 8192
brk(0x55c3c5d82000)                     = 0x55c3c5d82000
read(3, "eds Net::SSLeay>=1.56 and OpenSS"..., 8192) = 8192
brk(0x55c3c5da3000)                     = 0x55c3c5da3000
brk(0x55c3c5da2000)                     = 0x55c3c5da2000
brk(0x55c3c5dc4000)                     = 0x55c3c5dc4000
read(3, "lsext_status_cb($ctx,undef);\n\t  "..., 8192) = 8192
brk(0x55c3c5de5000)                     = 0x55c3c5de5000
brk(0x55c3c5de4000)                     = 0x55c3c5de4000
read(3, "=2 && DEBUG(\"$uri just answered "..., 8192) = 1671
brk(0x55c3c5e05000)                     = 0x55c3c5e05000
close(3)                                = 0

On Debian 9 no al_ssl_error occurred.

How to solve this now?

Obviously the cleanest way is to upgrade the already outdated Operating System (Ubuntu 16.04/xenial). This release will be end of life in April 2021 and should be updated. This way the newer OpenSSL and SSLeay versions will be installed which include compatibility changes for updated protocols and ciphers.

If the OS cannot be upgraded (for whatever reason), the Net::SSLeay Perl module and the openssl libraries should be manually upgraded. See the steps below.

Compiling and installing newer OpenSSL

Installing a new version of the Net::SSLeay Perl module using cpan is not enough as it compiles against the already installed OpenSSL libraries (1.0.2g in this case). The same SSL protocol/handshake error would still happen. So first, a new OpenSSL version needs to be downloaded and compiled. It doesn't have to replace the existing openssl version on that machine but the libraries are required.

First download a recent version of OpenSSL (here 1.1.1h):

ckadm@mintp ~/build $ wget https://www.openssl.org/source/openssl-1.1.1h.tar.gz
ckadm@mintp ~/build $ tar -xzf openssl-1.1.1h.tar.gz
ckadm@mintp ~/build $ cd openssl-1.1.1h/

Then create a target directory for the new OpenSSL libraries (default target directory would be /usr/local/lib):

ckadm@mintp ~/build/openssl-1.1.1h $ mkdir ~/build/openssl

This path (/home/ckadm/build/openssl) can now be used as prefix:

ckadm@mintp ~/build/openssl-1.1.1h $ ./config --prefix=/home/ckadm/build/openssl
ckadm@mintp ~/build/openssl-1.1.1h $ make -j24
ckadm@mintp ~/build/openssl-1.1.1h $ make install

Usually this all should work fine and you should have the following directories the target path:

ckadm@mintp ~/build/openssl-1.1.1h $ ll /home/ckadm/build/openssl
total 28
drwxrwxr-x 7 ckadm ckadm 4096 Nov  2 13:27 ./
drwxrwxr-x 4 ckadm ckadm 4096 Nov  2 13:25 ../
drwxrwxr-x 2 ckadm ckadm 4096 Nov  2 13:27 bin/
drwxrwxr-x 3 ckadm ckadm 4096 Nov  2 13:27 include/
drwxrwxr-x 4 ckadm ckadm 4096 Nov  2 13:27 lib/
drwxrwxr-x 4 ckadm ckadm 4096 Nov  2 13:28 share/
drwxrwxr-x 5 ckadm ckadm 4096 Nov  2 13:27 ssl/

Compiling and installing newer Net::SSLeay module

The current version of Net::SSLeay (1.88 as of this writing) can easily be downloaded using cpan:

cpan[1]> get Net::SSLeay
cpan[2]> quit

This should download and extract the current release in a .cpan directory.

Note: As an alternative, the release can also be manually downloaded from meta::cpan.

After changing into the source code directory created by cpan, the magic key is to set an environment variable OPENSSL_PREFIX which points to the newer OpenSSL version, followed by perl Makefile.PL.

mintp ~ # cd .cpan/build/Net-SSLeay-1.88-c7vzSa/
mintp Net-SSLeay-1.88-c7vzSa # OPENSSL_PREFIX=/home/ckadm/build/openssl perl Makefile.PL
Do you want to run external tests?
These tests *will* *fail* if you do not have network connectivity. [n]
*** Found OpenSSL-1.1.1h installed in /home/ckadm/build/openssl
*** Be sure to use the same compiler and options to compile your OpenSSL, perl,
    and Net::SSLeay. Mixing and matching compilers is not supported.
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Net::SSLeay
Writing MYMETA.yml and MYMETA.json

The output already mentions it: OpenSSL 1.1.1h was found.

Now the module can be built and installed (as root - or run sudo make install):

mintp Net-SSLeay-1.88-c7vzSa # make
mintp Net-SSLeay-1.88-c7vzSa # make install
Running Mkbootstrap for Net::SSLeay ()
chmod 644 "SSLeay.bs"
Manifying 2 pod documents
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /usr/local/lib/x86_64-linux-gnu/perl/5.22.1/auto/Net/SSLeay/SSLeay.so
Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5.22.1/perllocal.pod

Testing and running the plugin again

Now that the newer Net::SSLeay Perl module was installed (with root privileges) into /usr/local... (which should be part of $PATH), Perl should automatically detect the newer module:

ckadm@mintp ~ $ perl -MNet::SSLeay -le 'print $Net::SSLeay::VERSION'
1.88

So far so good. What about the Perl script, the monitoring plugin check_netscaler_gateway?

ckadm@mintp ~ $ ./check_netscaler_gateway.pl -H citrix.example.com -u user -p secret -S STORE_ID
NetScaler Gateway OK - DESKTOP; Desktop MCP; Desktop Media; Desktop-MCP;

Great! It works again!

TL;DR

Handling https and its protocols has changed a lot in recent years. This is not always obvious. But it becomes a problem when older OpenSSL libraries are used in programs and scripts. A distribution upgrade is often the easiest way to make sure newer ssl/tls protocols and ciphers are accepted. However it is also possible to manually compile and upgrade programs with the newer libraries.














Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.