An interesting case happened a few days ago on a central mail server, running Postfix as MTA: All (!) incoming e-mails were rejected by one of the DNSBLs (Spamhaus). Did the whole Internet become blacklisted? Was Spamhaus having a technical problem? Let's find out!
Let's start with the basics. What is Spamhaus and what does it have to do with e-mails?
Spamhaus is a so-called DNSBL (DNS Block List). A DNSBL is basically a database holding data about the reputation of an IP address. If a mail is received from an IP address with a bad reputation (= there were spams sent from this IP) the e-mail gets rejected. This leads to an error message to the sender which looks like this:
554 5.7.1 Client host 122.161.53.188 blocked using ZEN - see https://www.spamhaus.org/query/ip/122.161.53.188 for details
Postfix can be configured to run (incoming) mails through one ore several DNSBL services:
smtpd_client_restrictions = permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination,
reject_rbl_client zen.spamhaus.org
Spamhaus, being one of the oldest and most reliable DNSBL, is often used in such a setup.
Note: An article from Geeker's Digest describes how a DNS Block List works (on a technical level).
With that knowledge in mind we know that incoming mails are rejected when the sender server's IP address is listed in the Spamhaus database. But as mentioned before, all of a sudden all incoming e-mails were rejected - even from well known and widely used mail servers with a known good reputation.
The small difference can be spotted in the mail logs:
Aug 3 10:15:40 mail01 postfix/smtpd[23645]: NOQUEUE: reject: RCPT from sender.example.com[xx.xx.xx.xx]: 554 5.7.1 Service unavailable; Client host [xx.xx.xx.xx] blocked using zen.spamhaus.org; Error: open resolver; https://www.spamhaus.org/returnc/pub/162.158.148.77; from=<sender@example.com> to=<recipient@example.com> proto=ESMTP helo=<icinga2.infiniroot.net>
The log message contains a small but important hint: Error: open resolver.
The error message is not obvious and requires some research. However after a couple of minutes the reason was found. The error shown above is due to a change in Spamhaus' DNSBL service which was announced in January 2022:
Are you currently using the Spamhaus Project's DNS Blocklists (DNSBLs)? Do you access them via the Public Mirrors, for example, query "sbl.spamhaus.org"? Do you use Cloudflare's DNS? If you've answered "yes" to all three of those questions, you need to make some changes to your email infrastructure. These changes are quick and easy to make, but if you fail to make them, you could find that at some point in 2022, all or none of your email is blocked!
All these facts were true, looking at this Postfix setup:
We've switched to the Cloudflare DNS resolvers a couple of years ago because they showed a much better performance and lower caching times than our old resolvers from the ISP. Unfortunately this now leads to a problem with the Spamhaus DNSBL.
Note: This change affects multiple blocklists, such as zen.spamhaus.org, sbl.spamhaus.org, pbl.spamhaus.org, cbl.abuseat.org and potentially more.
Yes, that's the question. Should we switch the DNS resolvers to something else? But an alternative public DNS server could also be added to Spamhaus' public DNS policy and we would face the same problem later again. Or should we build (and maintain) our own resolvers? And what does Spamhaus suggest we do?
After contacting the Spamhaus support, they suggested to switch to their free DQS (Data Query Service). The difference here is that you are using a unique key (called a DQS key) to use the Spamhaus DNSBL. Instead of querying zen.spamhaus.org as before, you would use dqs_key.zen.dq.spamhaus.net as DNSBL URL in the Postfix config:
smtpd_client_restrictions = permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination,
reject_rbl_client dqs_key.zen.dq.spamhaus.net=127.0.0.[2..11]
Obviously dqs_key needs to be replaced with your unique key. A technical description can be found in the Spamhaus docs.
After registering a DQS account, I was told (by support) to request a quote. Due to a very low number of mailboxes on this Postfix setup (around 30 mailboxes) the number of DNSBL lookups are obviously not that high. This should (again, according to Spamhaus support) be eligible to continue the Spamhaus DNSBL for free.
However I received a quote for $450 after applying for it. After some additional back and forth with the Spamhaus support, the account was seen as "commercial" and not "free" account - even though I registered on the free data query service account sign-up page.
Finally, after responding to the sales rep and confirming that this is indeed a small setup and we're not a large ISP, I was told that we can continue to use the DQS for free. We'll see if that's still true a year from now ;-).
The change in Spamhaus' DNS response when using public resolvers does not only cause a problem on Postfix. If you are using the check_rbl monitoring plugin to monitor if your mail server(s) are listed on any RBL/DNSBL, you could get alerts that your server is listed on Spamhaus:
root@icinga2:~# /usr/lib/nagios/plugins/check_rbl -H 212.103.71.210 -s zen.spamhaus.org
CHECK_RBL CRITICAL - 212.103.71.210 BLACKLISTED on 1 server of 1 (zen.spamhaus.org (127.255.255.254)) | servers=1;0;0 time=0s;;
As you can see from the DNS response (127.255.255.254), this is the Spamhaus answer that a public DNS resolver was used.
To solve this, a different DNS resolver needs to be used for the check_rbl monitoring plugin. I am currently working on a pull request and will provide this solution in the upstream project - and write about it in the next blog post.
Ashok from India wrote on Nov 24th, 2022:
Thanks a lot Claudio. This really helped me. Happy that my inbound email is back up now! This is the exact problem I had. I have temporarily disabled the reject_rbl_client while I wait for the DQS Key from spamhaus.
Richard from Great Britain wrote on Aug 30th, 2022:
Thanks for this, I had the same issue. Now seems to be resolved.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb Icingaweb2 Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder