When we wanted to work on one of our code repositories with git, we ran into an interesting error on our own Gitlab server:
cka@mintp ~/Git/infiniroot/ansible $ git pull
fatal: unable to access 'https://gitlab.infiniroot.net/infiniroot/ansible.git/': server certificate verification failed. CAfile: none CRLfile: none
Server certificate verification failed? Did the automatic Let's Encrypt renewal not work? But a quick check on the server and also using a browser to access Gitlab revealed that the certificate was OK and recently renewed (on September 25th):
root@server ~ # ls -l /etc/letsencrypt/live/gitlab.infiniroot.net/fullchain.pem
lrwxrwxrwx 1 root root 50 Sep 25 13:14 /etc/letsencrypt/live/gitlab.infiniroot.net/fullchain.pem -> ../../archive/gitlab.infiniroot.net/fullchain2.pem
root@server ~ # ls -l /etc/letsencrypt/archive/gitlab.infiniroot.net/fullchain2.pem
-rw-r--r-- 1 root root 5.5K Sep 25 13:14 /etc/letsencrypt/archive/gitlab.infiniroot.net/fullchain2.pem
What's going on?
Let's take a closer look at the full chain certificate file, which contains three certificates:
Looking closer at the last certificate in this chain, CN = ISRG Root X1, reveals that it is issued by DST Root CA X3:
ck@mintp ~ $ openssl x509 -text -in /tmp/ISRG_Root_X1.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Validity
Not Before: Jan 20 19:14:03 2021 GMT
Not After : Sep 30 18:14:03 2024 GMT
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Subject Public Key Info:
[...]
The last step of certificate validation always happens on the local machine (the client) where the last issuer (DST Root CA X3 in this case) needs to be installed locally. Does this "DST Root CA X3" exist on the local machine?
ck@mintp /etc/ssl/certs $ ll|grep -i DST
lrwxrwxrwx 1 root root 18 Feb 4 2021 12d55845.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root 18 Feb 4 2021 2e5ac55d.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root 53 May 26 2018 DST_Root_CA_X3.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
Yes, it exists. If it was missing, this could have been the reason why the chain was not trusted. But let's take a closer look at this one, too:
ck@mintp /etc/ssl/certs $ openssl x509 -text -in DST_Root_CA_X3.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
Signature Algorithm: sha1WithRSAEncryption
Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Validity
Not Before: Sep 30 21:12:19 2000 GMT
Not After : Sep 30 14:01:15 2021 GMT
Subject: O = Digital Signature Trust Co., CN = DST Root CA X3
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
[...]
This Root CA certificate has expired (Validity Not After: Sep 30 14:01:15 2021 GMT)!
While researching information on this particular Root CA, a blog post from Let's Encrypt, last updated on September 30th 2021, turned up:
Let’s Encrypt has a “root certificate” called ISRG Root X1. Modern browsers and devices trust the Let’s Encrypt certificate installed on your website because they include ISRG Root X1 in their list of root certificates. To make sure the certificates we issue are trusted on older devices, we also have a “cross-signature” from an older root certificate: DST Root CA X3.
[...]
DST Root CA X3 will expire on September 30, 2021. That means those older devices that don’t trust ISRG Root X1 will start getting certificate warnings when visiting sites that use Let’s Encrypt certificates.
[...]
If you provide an API or have to support IoT devices, you’ll need to make sure of two things: (1) all clients of your API must trust ISRG Root X1 (not just DST Root CA X3), and (2) if clients of your API are using OpenSSL, they must use version 1.1.0 or later.
To sum that up: The ISRG Root X1 CA certificate needs to be installed on the system. But it turns out, that this is already the case:
ck@mintp /etc/ssl/certs $ ll|grep -i ISRG
lrwxrwxrwx 1 root root 16 Feb 4 2021 4042bcee.0 -> ISRG_Root_X1.pem
lrwxrwxrwx 1 root root 16 Feb 4 2021 6187b673.0 -> ISRG_Root_X1.pem
lrwxrwxrwx 1 root root 51 May 26 2018 ISRG_Root_X1.pem -> /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
Taking a closer look at how the trust chain looks like (using SSLLabs), there are two certificate paths detected:
As the ISRG Root X1 still contains the information this it was issued by the now expired DST Root CA X3, this trust path is checked (until the end of the chain) and then fails. So the fault is really that the chain was not updated during the automatic renewal of the Let's Encrypt certificates. This is also the case for certificates still renewed in September and October. In our case, the Let's Encrypt certificate of gitlab.infiniroot.net was automatically renewed on September 25th 2021 - yet it still contains the chain leading up to DST Root CA X3, just a few days before this Root CA would expire.
Luckily the browsers automatically detect and handle the different chain paths, but how can this be solved now for CLI commands, such as git? According to the ca-certificates package, deleting the "DST Root CA X3" certificate should be enough. This can be seen in the changelog of the latest ca-certificates package update:
After updating the package, the machine was restarted (a session logout should also do the job) and git pull now worked again, without any certificate warning:
ck@mintp ~/Git/infiniroot/ansible $ git pull
Username for 'https://gitlab.infiniroot.net':
Older Operating Systems or Linux distributions have not yet received an update of the ca-certificates - or never will. On Debian Stretch the current ca-certificates package hasn't been updated in a while:
root@stretch ~ # dpkg -l|grep ca-cert
ii ca-certificates 20200601~deb9u2 all Common CA certificates
On these systems, the Let's Encrypt problem can be reproduced by using curl on a known domain using Let's Encrypt:
root@stretch ~ # curl https://gitlab.infiniroot.net
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
Looking at the ca-certificates, the expired DST Root CA X3 is still on the system:
root@stretch /etc/ssl/certs # ll|grep -i DST
lrwxrwxrwx 1 root root 18 Jun 10 2020 12d55845.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root 18 Jun 10 2020 2e5ac55d.0 -> DST_Root_CA_X3.pem
lrwxrwxrwx 1 root root 53 Mar 15 2016 DST_Root_CA_X3.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
This certificate is also inside the ca-certificates.crt file, which is actually the one the system (openssl) reads:
root@stretch ~ # awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep DST
subject=O = Digital Signature Trust Co., CN = DST Root CA X3
The configuration of ca-certificates is located at /etc/ca-certificates.conf and holds this information as well:
root@stretch ~ # grep DST /etc/ca-certificates.conf
!mozilla/DST_ACES_CA_X6.crt
mozilla/DST_Root_CA_X3.crt
To manually remove the certificate, append a "!" in front of mozilla/DST_Root_CA_X3.crt, followed by running the update-ca-certificates command:
root@stretch ~ # vi /etc/ca-certificates.conf
root@stretch ~ # grep DST /etc/ca-certificates.conf
!mozilla/DST_ACES_CA_X6.crt
!mozilla/DST_Root_CA_X3.crt
root@stretch ~ # update-ca-certificates
Updating certificates in /etc/ssl/certs...
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.bundle.crt.20200602
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.bundle.crt.20200602
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.crt
WARNING: Skipping duplicate certificate wildcard.novahosting.ch.crt
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
The output says it: 1 certificate was removed. And the ca-certificates.crt is updated:
root@stretch ~ # ll /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 197216 Oct 7 09:15 /etc/ssl/certs/ca-certificates.crt
Let's use the awk command from above to verify that the DST Root CA X3 is really gone:
root@stretch ~ # awk -v cmd='openssl x509 -noout -subject' '/BEGIN/{close(cmd)};{print | cmd}' < /etc/ssl/certs/ca-certificates.crt | grep DST
root@stretch ~ #
Yep, it's gone. Does the curl command work now?
root@stretch ~ # curl https://gitlab.infiniroot.net
<html><body>You are being <a href="http://gitlab.infiniroot.net/users/sign_in">redirected</a>.</body></html>
And yes, curl on our gitlab.infiniroot.net server works again!
There are two major problems with this Root CA expiration from Let's Encrypt:
tudou from wrote on Mar 10th, 2022:
Thank you!
Attila from wrote on Feb 1st, 2022:
It was very helpful, thank you!
J Williamson from Dublin wrote on Jan 11th, 2022:
This was SUCH a help. Learnt a lot going through this. Really appreciate the well laid out, easy to follow and well explained instructions and WHY we were doing each step.
THANK YOU.
grgalex from wrote on Nov 29th, 2021:
A clear and concise explanation. Excellent!
Kakoen from wrote on Nov 13th, 2021:
Thank you so much! This solved my issue perfectly while explaining the background.
RobertT from Cluj-Napoca wrote on Nov 12th, 2021:
Nice detailed work. Really Helpful.
Henri from wrote on Nov 1st, 2021:
This not only solved my issues when trying to update my Trisquel Linux, it also explained the background very well, which I really like. Thanks a lot.
Peter from wrote on Oct 18th, 2021:
Great detailed explanation of both the problem and the mitigation, thank you very much for this useful post!
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder