Analyzing and fixing Internal Server Error in PowerDNS API and Opera DNS UI

Written by - 0 comments

Published on - Listed in DNS PowerDNS MySQL MariaDB Linux


A few years ago we've introduced PowerDNS as our new public DNS name server solution, after a couple of different DNS solutions were evaluated. To ease the management of DNS zones, we've installed Opera's DNS-UI alongside PowerDNS. DNS-UI uses PowerDNS' API in the background.

Today I came across an error, when a specific zone would not load in DNS-UI anymore, showing a "Oops! Something went wrong!" error. Other zones continued to show up correctly in DNS-UI.

Oops! Something went wrong! in Opera DNS-UI for PowerDNS.

Internal Server Error from API

Loading the zone in question (example.com) in DNS-UI resulted in the error seen in the screenshot above. But what about the web server's error logs? A stack trace of the error was actually logged and showed the following entries:

[Tue Oct 31 11:07:54.986629 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: Pest_ServerError: {"error": "Internal Server Error"} in /var/dns-ui/Pest.php:334, referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986673 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: Stack trace:, referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986680 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #0 /var/dns-ui/Pest.php(268): Pest->checkLastResponseForError(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986685 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #1 /var/dns-ui/Pest.php(154): Pest->doRequest(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986689 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #2 /var/dns-ui/powerdns.php(30): Pest->get(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986694 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #3 /var/dns-ui/model/zone.php(224): PowerDNS->get(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986699 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #4 /var/dns-ui/views/zone.php(32): Zone->list_resource_record_sets(), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986704 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #5 /var/dns-ui/requesthandler.php(62): require('/var/dns-ui/vie...'), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986709 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #6 /var/dns-ui/public_html/init.php(18): require('/var/dns-ui/req...'), referer: https://1.1.1.1/zones
[Tue Oct 31 11:07:54.986713 2023] [php7:notice] [pid 2271140] [client 192.168.200.43:40256] 1698746874: #7 {main}, referer: https://1.1.1.1/zones

Note: Obviously the IP 1.1.1.1 is used to obfuscate the real IP address of this PowerDNS server.

Although the stack trace is not making much sense here, the first line gives an important hint: An Internal Server Error was detected.

As I mentioned at the beginning, DNS-UI uses the PowerDNS API in the background to read (and write) zones and records. Let's reproduce the error using a direct API request:

root@powerdns:~# curl -H "X-API-Key: secret" -H "Accept: application/json" http://localhost:8081/api/v1/servers/localhost/zones/example.com
{"error": "Internal Server Error"}

The PowerDNS API returns the exact same error message already reported in the DNS-UI stack trace: Internal Server Error. Unfortunately no additional information is presented by the API.

Missing quotes in TXT record

The PowerDNS service (usually) logs right into syslog. And this is also where the following error on the "webserver" (which represents the API) was found:

Oct 31 11:16:03 powerdns pdns_server[2258670]: [webserver] 6c4876d5-0e71-4ab9-b623-61f132782a5e HTTP ISE for "/api/v1/servers/localhost/zones/example.com": STL Exception: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es'

The log entry shows two hints:

  • try 'pdnsutil check-zone': This command can be used to find errors in a zone file.
  • Data field in DNS should start with quote ("): Obviously there is a data field (value) inside this zone which should be within quotes, but it is not.

Let's first verify the zone as suggested:

root@powerdns:~# pdnsutil check-zone example.com
Checked 9 records of 'example.com', 0 errors, 0 warnings.

However no errors were found...

Let's list all the records from that zone:

root@powerdns:~# pdnsutil list-zone example.com
$ORIGIN .
example.com    300    IN    A    172.217.168.46
example.com    300    IN    NS    ns1.dnstechprovider.net.
example.com    300    IN    NS    ns2.dnstechprovider.net.
example.com    300    IN    NS    ns3.dnstechprovider.net.
example.com    300    IN    SOA    ns1.dnstechprovider.net. dnsadmin.dnstechprovider.net. 2021031001 86400 1800 2419200 86400
example.com    300    IN    TXT    "google-site-verification=-J0EfMRaH3M7lef3p0Ms09jO-G7ukKNRLpNBUhlZeK8"
www.example.com    300    IN    A    172.217.168.46
_acme-challenge.example.com    300    IN    TXT    YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es
_acme-challenge.www.example.com    300    IN    TXT    wqNhrlqvUTe0cjOAo3b2OlHfzrjjvK0PlJnpNayN56U

Interesting. There are two TXT records at the bottom of the zone where the value is not placed within quotes. Another TXT record for the google-site-verification also exists and there the value is placed inside quotes. 

The first "unquoted" TXT value is YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es - the same value mentioned in the error logged in syslog.

Fixing DNS records in the backend

The easiest thing would be to delete the unquoted TXT records and create them properly again. However pdnsutil does not offer a "delete-record" function (I've been missing that for years...). The only way to delete or edit a record is to open the zone in an editor (e.g. vim) and modify/fix the records. To edit a zone, the command pdnsutil edit-zone can be used:

root@powerdns:~# pdnsutil edit-zone example.com
Error: Parsing record content (try 'pdnsutil check-zone'): Data field in DNS should start with quote (") at position 0 of 'YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es'

But nope! We've just ran into the same error again. The zone cannot even be edited anymore because of the unquoted TXT record error!

But luckily PowerDNS is always using a backend where the zones and records are stored. In our DNS architecture this is a MySQL database. The domains/zones can be found in the table "domains":

mysql> select * from domains where name = 'example.com';
+-----+--------------+--------+------------+--------+-----------------+----------+
| id  | name         | master | last_check | type   | notified_serial | account  |
+-----+--------------+--------+------------+--------+-----------------+----------+
| 607 | example.com |        |       NULL | MASTER |      2021031001 | CH Media |
+-----+--------------+--------+------------+--------+-----------------+----------+
1 row in set (0.00 sec)

Note the ID 607 of that domain. This is our domain identifier in the "records" table:

mysql> select * from records where domain_id = 607;
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
| id    | domain_id | name                             | type | content                                                                | ttl  | prio | change_date | disabled | ordername | auth |
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
| 13024 |       607 | example.com                     | A    | 172.217.168.46                                                         |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13025 |       607 | www.example.com                 | A    | 172.217.168.46                                                         |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13026 |       607 | example.com                     | TXT  | "google-site-verification=-J0EfMRaH3M7lef3p0Ms09jO-G7ukKNRLpNBUhlZeK8" |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13027 |       607 | _acme-challenge.example.com     | TXT  | YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es                            |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13028 |       607 | _acme-challenge.www.example.com | TXT  | wqNhrlqvUTe0cjOAo3b2OlHfzrjjvK0PlJnpNayN56U                            |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13029 |       607 | example.com                     | NS   | ns1.dnstechprovider.net                                                        |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13030 |       607 | example.com                     | NS   | ns2.dnstechprovider.net                                                        |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13031 |       607 | example.com                     | NS   | ns3.dnstechprovider.net                                                        |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13032 |       607 | example.com                     | SOA  | ns1.dnstechprovider.net. dnsadmin.dnstechprovider.net. 2021031001 86400 1800 2419200 86400  |  300 |    0 |        NULL |        0 | NULL      |    1 |
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
9 rows in set (0.01 sec)

The records look familiar, don't they?

To fix the TXT records, we can simply update the "content" field using the correct IDs and put the content into quotes:

mysql> UPDATE records SET content = '"YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es"' WHERE id = 13027;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> UPDATE records SET content = '"wqNhrlqvUTe0cjOAo3b2OlHfzrjjvK0PlJnpNayN56U"' WHERE id = 13028;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

Let's verify the updated records in the database:

mysql> select * from records where domain_id = 607 and type = 'TXT';
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
| id    | domain_id | name                             | type | content                                                                | ttl  | prio | change_date | disabled | ordername | auth |
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
| 13026 |       607 | example.com                     | TXT  | "google-site-verification=-J0EfMRaH3M7lef3p0Ms09jO-G7ukKNRLpNBUhlZeK8" |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13027 |       607 | _acme-challenge.example.com     | TXT  | "YUSLVPd-SZwAqrwmsLnaPdQ8HQUXBOCux-UEZRcn8Es"                          |  300 |    0 |        NULL |        0 | NULL      |    1 |
| 13028 |       607 | _acme-challenge.www.example.com | TXT  | "wqNhrlqvUTe0cjOAo3b2OlHfzrjjvK0PlJnpNayN56U"                          |  300 |    0 |        NULL |        0 | NULL      |    1 |
+-------+-----------+----------------------------------+------+------------------------------------------------------------------------+------+------+-------------+----------+-----------+------+
3 rows in set (0.00 sec)

All the TXT record values are now placed within quotes.

How does that change show in DNS-UI now?

The zone shows up and can be edited in DNS-UI again!


Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder