A few years ago we've introduced PowerDNS as our new public DNS name server solution, after a couple of different DNS solutions were evaluated. To ease the management of DNS zones, we've installed Opera's DNS-UI alongside PowerDNS. DNS-UI uses PowerDNS' API in the background.
Today I came across an error, when a specific zone would not load in DNS-UI anymore, showing a "Oops! Something went wrong!" error. Other zones continued to show up correctly in DNS-UI.
Loading the zone in question (example.com) in DNS-UI resulted in the error seen in the screenshot above. But what about the web server's error logs? A stack trace of the error was actually logged and showed the following entries:
[Tue Oct 31 11:07:54.986629 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: Pest_ServerError: {"error": "Internal Server Error"} in /var/dns-ui/Pest.php:334, referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986673 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: Stack trace:, referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986680 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #0 /var/dns-ui/Pest.php(268): Pest->checkLastResponseForError(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986685 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #1 /var/dns-ui/Pest.php(154): Pest->doRequest(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986689 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #2 /var/dns-ui/powerdns.php(30): Pest->get(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986694 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #3 /var/dns-ui/model/zone.php(224): PowerDNS->get(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986699 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #4 /var/dns-ui/views/zone.php(32): Zone->list_resource_record_sets(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986704 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #5 /var/dns-ui/requesthandler.php(62): require('/var/dns-ui/vie...'), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986709 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #6 /var/dns-ui/public_html/init.php(18): require('/var/dns-ui/req...'), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986713 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #7 {main}, referer: https://1.1.1.1/zones
Note: Obviously the IP 1.1.1.1 is used to obfuscate the real IP address of this PowerDNS server.
Although the stack trace is not making much sense here, the first line gives an important hint: An Internal Server Error was detected.
As I mentioned at the beginning, DNS-UI uses the PowerDNS API in the background to read (and write) zones and records. Let's reproduce the error using a direct API request:
root@powerdns:~# curl -H "X-API-Key: secret" -H "Accept: application/json" http://localhost:8081/api/v1/servers/localhost/zones/example.com
{"error": "Internal Server Error"}
The PowerDNS API returns the exact same error message already reported in the DNS-UI stack trace: Internal Server Error. Unfortunately no additional information is presented by the API.
The PowerDNS service (usually) logs right into syslog. And this is also where the following error on the "webserver" (which represents the API) was found:
Oct 31 11:16:03 powerdns pdns_server[2258670]: [webserver] 6c4876d5-0e71-4ab9-b623-61f132782a5e HTTP ISE for "/api/v1/servers/localhost/zones/example.com": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es'
The log entry shows two hints:
Let's first verify the zone as suggested:
root@powerdns:~# pdnsutil check-zone example.com
Checked 9 records of 'example.com', 0 errors, 0 warnings.
However no errors were found...
Let's list all the records from that zone:
root@powerdns:~# pdnsutil list-zone example.com
$ORIGIN .
example.com 300 IN A 172.217.168.46
example.com 300 IN NS ns1.dnstechprovider.net.
example.com 300 IN NS ns2.dnstechprovider.net.
example.com 300 IN NS ns3.dnstechprovider.net.
example.com 300 IN SOA ns1.dnstechprovider.net. dnsadmin.dnstechprovider.net. 2021031001 86400 1800 2419200 86400
example.com 300 IN TXT "google-site-verification=-J0EfMRaH3M7lef3p0Ms09jO-G7ukKNRLpNBUhlZeK8"
www.example.com 300 IN A 172.217.168.46
_acme-challenge.example.com 300 IN TXT YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es
_acme-challenge.www.example.com 300 IN TXT wqNhrlqvUTe0cjOAo3b2OlHfzrjjvK0PlJnpNayN56U
Interesting. There are two TXT records at the bottom of the zone where the value is not placed within quotes. Another TXT record for the google-site-verification also exists and there the value is placed inside quotes.
The first "unquoted" TXT value is YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es - the same value mentioned in the error logged in syslog.
The easiest thing would be to delete the unquoted TXT records and create them properly again. However pdnsutil does not offer a "delete-record" function (I've been missing that for years...). The only way to delete or edit a record is to open the zone in an editor (e.g. vim) and modify/fix the records. To edit a zone, the command pdnsutil edit-zone can be used:
root@powerdns:~# pdnsutil edit-zone example.com
Error: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es'
But nope! We've just ran into the same error again. The zone cannot even be edited anymore because of the unquoted TXT record error!
But luckily PowerDNS is always using a backend where the zones and records are stored. In our DNS architecture this is a MySQL database. The domains/zones can be found in the table "domains":
mysql> select * from domains where name = 'example.com';
+-----+--------------+--------+------------+--------+-----------------+----------+
| id | name | master | last_check | type | notified_serial | account |
+-----+--------------+--------+------------+--------+-----------------+----------+
| 607 | example.com | | NULL | MASTER | 2021031001 | CH Media |
+-----+--------------+--------+------------+--------+-----------------+----------+
1 row in set (0.00 sec)
Note the ID 607 of that domain. This is our domain identifier in the "records" table:
mysql> select * from records where domain_id = 607;
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
| id | domain_id | name | type | content | ttl | prio | change_date | disabled | ordername | auth |
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
| 13024 | 607 | example.com | A | 172.217.168.46 | 300 | 0 | NULL | 0 | NULL | 1 |
| 13025 | 607 | www.example.com | A | 172.217.168.46 | 300 | 0 | NULL | 0 | NULL | 1 |
| 13026 | 607 | example.com | TXT | "google-site-verification=-J0EfMRaH3M7lef3p0Ms09jO-G7ukKNRLpNBUhlZeK8" | 300 | 0 | NULL | 0 | NULL | 1 |
| 13027 | 607 | _acme-challenge.example.com | TXT | YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es | 300 | 0 | NULL | 0 | NULL | 1 |
| 13028 | 607 | _acme-challenge.www.example.com | TXT | wqNhrlqvUTe0cjOAo3b2OlHfzrjjvK0PlJnpNayN56U | 300 | 0 | NULL | 0 | NULL | 1 |
| 13029 | 607 | example.com | NS | ns1.dnstechprovider.net | 300 | 0 | NULL | 0 | NULL | 1 |
| 13030 | 607 | example.com | NS | ns2.dnstechprovider.net | 300 | 0 | NULL | 0 | NULL | 1 |
| 13031 | 607 | example.com | NS | ns3.dnstechprovider.net | 300 | 0 | NULL | 0 | NULL | 1 |
| 13032 | 607 | example.com | SOA | ns1.dnstechprovider.net. dnsadmin.dnstechprovider.net. 2021031001 86400 1800 2419200 86400 | 300 | 0 | NULL | 0 | NULL | 1 |
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
9 rows in set (0.01 sec)
The records look familiar, don't they?
To fix the TXT records, we can simply update the "content" field using the correct IDs and put the content into quotes:
mysql> UPDATE records SET content = '"YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es"' WHERE id = 13027;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> UPDATE records SET content = '"wqNhrlqvUTe0cjOAo3b2OlHfzrjjvK0PlJnpNayN56U"' WHERE id = 13028;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
Let's verify the updated records in the database:
mysql> select * from records where domain_id = 607 and type = 'TXT';
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
| id | domain_id | name | type | content | ttl | prio | change_date | disabled | ordername | auth |
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
| 13026 | 607 | example.com | TXT | "google-site-verification=-J0EfMRaH3M7lef3p0Ms09jO-G7ukKNRLpNBUhlZeK8" | 300 | 0 | NULL | 0 | NULL | 1 |
| 13027 | 607 | _acme-challenge.example.com | TXT | "YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es" | 300 | 0 | NULL | 0 | NULL | 1 |
| 13028 | 607 | _acme-challenge.www.example.com | TXT | "wqNhrlqvUTe0cjOAo3b2OlHfzrjjvK0PlJnpNayN56U" | 300 | 0 | NULL | 0 | NULL | 1 |
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
3 rows in set (0.00 sec)
All the TXT record values are now placed within quotes.
How does that change show in DNS-UI now?
The zone shows up and can be edited in DNS-UI again!
No comments yet.
AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Icingaweb Icingaweb2 Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder