DNS is one of the core services of networking and even more important on the Internet. Usually it's very easy to understand and to maintain - as long as standards are used and the admins work properly.
In an old environment where DNS issues occurred, both of these basic rules were ignored:
A djbdns dnscache installation was set up and was used as resolver for the whole environment. The dnscache was using itself as resolver, causing an infinite caching with old dns records. Great.
So there are two fails:
1) djbdns isn't exactly a standard dns server.
2) The admin at that time set up dnscache pointing to itself as resolver.
The issue in this environment was, that a public nameserver of a domain couldn't be resolved anymore, because the IP address was changed and the old IP was not accessible anymore. Dnscache then kept the old IP in its cache while the new IP was already known on the Internet.
I have searched for ways to "clear dnscache cache" and all found websites said that a simple restart of dnscache would do the job. But after a restart dnscache still kept the old IP address.
I even tried to set the local resolver in /etc/resolv.conf on this machine to Google's public DNS server 18.104.22.168, but dnscache seemed to ignore that entry.
Finally I found a website with the matching title "The known problems with djbdns" which was describing the exact problem as I saw:
Each query resolution times out without receiving an answer, and so never sends an answer itself. The delegation is effectively lame, and the cycle never breaks.
Does look familiar, doesn't it?
The solution is to create a special file for the affected domain (let's call it example.com), containing an IP address where to look up dns records for this domain.
Being on a FreeBSD system, I created the domain file with the public Google DNS IP:
After another restart of dnscache, the domain example.com was no longer being cached and finally the correct and new ip address of the nameserver for that domain was found.This finally fixed the DNS issue in the whole environment.