Lets Encrypt Root CA expiry (server certificate verification failed): Make sure to remove DST Root CA X3!

Written by - 8 comments

Published on - Listed in Internet TLS Security SSL Linux


When we wanted to work on one of our code repositories with git, we ran into an interesting error on our own Gitlab server:

cka@mintp ~/Git/infiniroot/ansible $ git pull
fatal: unable to access 'https://gitlab.infiniroot.net/infiniroot/ansible.git/': server certificate verification failed. CAfile: none CRLfile: none

Server certificate verification failed? Did the automatic Let's Encrypt renewal not work? But a quick check on the server and also using a browser to access Gitlab revealed that the certificate was OK and recently renewed (on September 25th):

root@server ~ # ls -l /etc/letsencrypt/live/gitlab.infiniroot.net/fullchain.pem
lrwxrwxrwx 1 root root 50 Sep 25 13:14 /etc/letsencrypt/live/gitlab.infiniroot.net/fullchain.pem -> ../../archive/gitlab.infiniroot.net/fullchain2.pem

root@server ~ # ls -l /etc/letsencrypt/archive/gitlab.infiniroot.net/fullchain2.pem
-rw-r--r-- 1 root root 5.5K Sep 25 13:14 /etc/letsencrypt/archive/gitlab.infiniroot.net/fullchain2.pem

What's going on?

Chasing the rabbit (following the chain)

Let's take a closer look at the full chain certificate file, which contains three certificates:

  1. The domain validated certificate (gitlab.infiniroot.net), issued by R3
  2. The intermediate certificate ICA (R3), issued by ISRG Root X1
  3. The root certificate Root CA (ISRG Root X1), issued by yet another Root CA: DST Root CA X3

Looking closer at the last certificate in this chain, CN = ISRG Root X1, reveals that it is issued by DST Root CA X3:

ck@mintp ~ $ openssl x509 -text -in /tmp/ISRG_Root_X1.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Subject Public Key Info:
[...]

The last step of certificate validation always happens on the local machine (the client) where the last issuer (DST Root CA X3 in this case) needs to be installed locally. Does this "DST Root CA X3" exist on the local machine?

ck@mintp /etc/ssl/certs $ ll|grep -i DST
lrwxrwxrwx 1 root root     18 Feb  4  2021 12d55845.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root     18 Feb  4  2021 2e5ac55d.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root     53 May 26  2018 DST_Root_CA_X3.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt

Yes, it exists. If it was missing, this could have been the reason why the chain was not trusted. But let's take a closer look at this one, too:

ck@mintp /etc/ssl/certs $ openssl x509 -text -in DST_Root_CA_X3.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
        Subject: O = Digital Signature Trust Co., CN = DST Root CA X3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
[...]

This Root CA certificate has expired (Validity Not After: Sep 30 14:01:15 2021 GMT)!

DST Root CA X3 has expired

While researching information on this particular Root CA, a blog post from Let's Encrypt, last updated on September 30th 2021, turned up:

Let’s Encrypt has a “root certificate” called ISRG Root X1. Modern browsers and devices trust the Let’s Encrypt certificate installed on your website because they include ISRG Root X1 in their list of root certificates. To make sure the certificates we issue are trusted on older devices, we also have a “cross-signature” from an older root certificate: DST Root CA X3.
[...]
DST Root CA X3 will expire on September 30, 2021. That means those older devices that don’t trust ISRG Root X1 will start getting certificate warnings when visiting sites that use Let’s Encrypt certificates.
[...]
If you provide an API or have to support IoT devices, you’ll need to make sure of two things: (1) all clients of your API must trust ISRG Root X1 (not just DST Root CA X3), and (2) if clients of your API are using OpenSSL, they must use version 1.1.0 or later.

To sum that up: The ISRG Root X1 CA certificate needs to be installed on the system. But it turns out, that this is already the case:

ck@mintp /etc/ssl/certs $ ll|grep -i ISRG
lrwxrwxrwx 1 root root     16 Feb  4  2021 4042bcee.0 -> ISRG_Root_X1.pem
lrwxrwxrwx 1 root root     16 Feb  4  2021 6187b673.0 -> ISRG_Root_X1.pem
lrwxrwxrwx 1 root root     51 May 26  2018 ISRG_Root_X1.pem -> /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

Taking a closer look at how the trust chain looks like (using SSLLabs), there are two certificate paths detected:

As the ISRG Root X1 still contains the information this it was issued by the now expired DST Root CA X3, this trust path is checked (until the end of the chain) and then fails. So the fault is really that the chain was not updated during the automatic renewal of the Let's Encrypt certificates. This is also the case for certificates still renewed in September and October. In our case, the Let's Encrypt certificate of gitlab.infiniroot.net was automatically renewed on September 25th 2021 - yet it still contains the chain leading up to DST Root CA X3, just a few days before this Root CA would expire.

Update the local certificates using system updates

Luckily the browsers automatically detect and handle the different chain paths, but how can this be solved now for CLI commands, such as git?  According to the ca-certificates package, deleting the "DST Root CA X3" certificate should be enough. This can be seen in the changelog of the latest ca-certificates package update:

After updating the package, the machine was restarted (a session logout should also do the job) and git pull now worked again, without any certificate warning:

ck@mintp ~/Git/infiniroot/ansible $ git pull
Username for 'https://gitlab.infiniroot.net': 

Manually updating the local certificates

Older Operating Systems or Linux distributions have not yet received an update of the ca-certificates - or never will. On Debian Stretch the current ca-certificates package hasn't been updated in a while:

root@stretch ~ # dpkg -l|grep ca-cert
ii  ca-certificates     20200601~deb9u2       all          Common CA certificates

On these systems, the Let's Encrypt problem can be reproduced by using curl on a known domain using Let's Encrypt:

root@stretch ~ # curl https://gitlab.infiniroot.net
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Looking at the ca-certificates, the expired DST Root CA X3 is still on the system:

root@stretch /etc/ssl/certs # ll|grep -i DST
lrwxrwxrwx 1 root root     18 Jun 10  2020 12d55845.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root     18 Jun 10  2020 2e5ac55d.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root     53 Mar 15  2016 DST_Root_CA_X3.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt

This certificate is also inside the ca-certificates.crt file, which is actually the one the system (openssl) reads:

root@stretch ~ # awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep DST
subject=O = Digital Signature Trust Co., CN = DST Root CA X3

The configuration of ca-certificates is located at /etc/ca-certificates.conf and holds this information as well:

root@stretch ~ # grep DST /etc/ca-certificates.conf
!mozilla/DST_ACES_CA_X6.crt
mozilla/DST_Root_CA_X3.crt

To manually remove the certificate, append a "!" in front of mozilla/DST_Root_CA_X3.crt, followed by running the update-ca-certificates command:

root@stretch ~ # vi /etc/ca-certificates.conf

root@stretch ~ # grep DST /etc/ca-certificates.conf
!mozilla/DST_ACES_CA_X6.crt
!mozilla/DST_Root_CA_X3.crt

root@stretch ~ # update-ca-certificates
Updating certificates in /etc/ssl/certs...
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.bundle.crt.20200602
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.bundle.crt.20200602
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.crt
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.crt
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

The output says it: 1 certificate was removed. And the ca-certificates.crt is updated:

root@stretch ~ # ll /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 197216 Oct  7 09:15 /etc/ssl/certs/ca-certificates.crt

Let's use the awk command from above to verify that the DST Root CA X3 is really gone:

root@stretch ~ # awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep DST
root@stretch ~ #

Yep, it's gone. Does the curl command work now?

root@stretch ~ # curl https://gitlab.infiniroot.net
<html><body>You are being <a href="http://gitlab.infiniroot.net/users/sign_in">redirected</a>.</body></html>

And yes, curl on our gitlab.infiniroot.net server works again!

A global problem

There are two major problems with this Root CA expiration from Let's Encrypt:

  1. In order to allow older Android devices to still work with Let's Encrypt certificates, LE still continues with the "alternative" trust chain, referring to the expired DST Root CA X3 as issuer. This alternative chain is seen (mainly) in command line tools and libraries and (may) leads to the server certificate verification failure.

  2. Due to problem 1, every affected client (and there are a lot!) will have to manually update and remove the DST Root CA X3 certificate from its own list of certificates. Either by updating the system packages (ca-certificates in Linux) or by manually removing the DST Root CA X3 certificate (see above). As this is something to do/fix on the client side, this has a global effect and server administrators can nothing do to fix this - it needs to be fixed on the client side.


Add a comment

Show form to leave a comment

Comments (newest first)

tudou from wrote on Mar 10th, 2022:

Thank you!


Attila from wrote on Feb 1st, 2022:

It was very helpful, thank you!


J Williamson from Dublin wrote on Jan 11th, 2022:

This was SUCH a help. Learnt a lot going through this. Really appreciate the well laid out, easy to follow and well explained instructions and WHY we were doing each step.

THANK YOU.


grgalex from wrote on Nov 29th, 2021:

A clear and concise explanation. Excellent!


Kakoen from wrote on Nov 13th, 2021:

Thank you so much! This solved my issue perfectly while explaining the background.


RobertT from Cluj-Napoca wrote on Nov 12th, 2021:

Nice detailed work. Really Helpful.


Henri from wrote on Nov 1st, 2021:

This not only solved my issues when trying to update my Trisquel Linux, it also explained the background very well, which I really like. Thanks a lot.


Peter from wrote on Oct 18th, 2021:

Great detailed explanation of both the problem and the mitigation, thank you very much for this useful post!


RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder