WatchMouse Laboratories

Posted by Pieter Ennes on December 9th, 2009

Google’s announcement to offer a public DNS service similar to OpenDNS was already discussed exhaustively last week. But we noticed that a lot of people wondered how the different public DNS recursors compare performance-wise, worldwide. We therefore did a small study in this field using the WatchMouse monitoring network.

Test method
We set up monitors for the following public DNS services:

• DNSAdvantage (by Neustar)
• DNSResolvers (by EasyDNS)
• Google
• OpenDNS

Another one, ScrubIT, turned out to be DOA and was removed from the test as neither of its servers answer queries, even though their blog says “It works!“. Yet another, OpenNIC was left out since its configuration requires a different set of IP addresses in each country. Finally, there are public servers from Level3 with widely known IP’s, but an official web page about this service seems hard to find. We’ve only just started monitoring them and therefore left them out in this test.

For a period of five days, we probed each of the above DNS services for five different records: ‘A’ records for cnn.com (TTL=300), bbc.co.uk (TTL=300) and www.techcrunch.com (TTL=1200); ‘MX’ records for yahoo.com (TTL=7200), and an ‘AAAA’ record for it.ipv6.watchmouse.com (TTL=28800). Each monitor was executed every 5 minutes and was configured to send a single UDP query (with no retries), and time out after 3 seconds without a reply. This set-up was duplicated to monitor two different recursors for each service provider; the obtained results for these were averaged.

For each DNS service roughly 18.000 queries were performed over a six day period (2 servers * 6 days * 288 probes/day * 5 monitors), rotating the 42 locations in our network.

We also added two reference monitors: one utilising a local recursor running on our monitoring station, and another one, dubbed ‘Direct’, querying one of the listed name servers directly.

Results
Before going into the performance details, let’s have a look at the relative number of errors per service:

First thing to note is that a local recursor generates less failures compared to a direct queries to one of the listed name servers, probably due of its ability to cache and prefer fast and correctly functioning servers over ones that are slow or failing. Both DNSAdvantage and OpenDNS do a good job in masking name server errors and minimising lookup time-outs; their failure rate was below 5‰. DNSResolvers seems to have a common failure rate of just under 10‰. Google’s free service, on the other hand, fails to return records within the 3 second time-out in about 15‰ of the queries. That’s worse than a direct query (13‰) to one of the provider servers or through a local resolver (9‰). And it’s three times as high as the two other major competitors.

When we consider the performance of the queries that did receive a valid response, it can be seen that a query through a local resolver is a little bit slower than using a direct query, again on average, worldwide. The difference of ~20ms, however, cannot easily be explained by taking into account the (negligible) round trip time of the UDP query to the local host. The variance in performance from the local resolver is also increased, so it would be likely that other factors play a role here.

DNSResolvers either have very busy servers, or do not seem to do a good job in reducing network latency. Their performance is nearly twice as bad compared to using the local resolver. Most likely their service does not facilitate something called Anycast to route queries to a nearby data centre. The remaining providers do use anycasting, and clearly have an advantage because of this.

Discarding time-outs, Google’s public DNS (59ms on average) definitely is the best in terms of performance, with OpenDNS in a photo finish with a 80ms score. But also the services offered by DNSAdvantage display solid sub-100ms performance with 93ms.

Discussion
Most of the tested host names were either for high-volume sites or had large TTL’s, causing public caches to be easily primed and expose their qualities. OpenDNS, Google and DNSAdvantage all show that they master this and have better lookup times than a local resolver or a direct query.

To avoid influences of second order effects, the measurements were done using only a single UDP query in each probe. By doing so, we were able to separate real query performance from packet loss and server failures. In real life, however, a typical PC would retry the query after a certain time (~5 seconds). Our DNS monitors can do this, but this would cause the failures to be blended into the performance measurements, inducing an (arbitrary) bias from the chosen time-out setting.

Also, real people (or offices for that matter) are most often not in multiple places at the same time. Thus in real life one would be more interested in what the best service provider is in your area. We hope to have some time for a second blog item on this in the near future, with typical user settings, and a breakdown per area.

Measuring averaged worldwide performance the way we did now is still nice as a synthetic benchmark. So for fun, and because everyone finds DNS response times very important, we can introduce a DNS time wasting score. Based on, say, an 100 sequential lookups per day for an average user and a time-out of 3 seconds, the failures induce extra waiting time and influence the score:

(10 seconds per day adds up to roughly one hour of DNS waiting per year)

Conclusion
Three of the researched providers clearly are competitive in terms of ‘clean’ performance and offer a useful service to the public. But the number of failures shown in the first chart must be considered as an intrinsic part of the quality of service.

And this is where Google, with a 3 times higher failure rate, seems to have more problems than OpenDNS and DNSAdvantage. For 15 out of 1000 host name lookups, Google fails to respond within 3 seconds (or the packet is just lost), causing an extra lookup after a time-out and an observed lookup time of multiple seconds. It seems that the Google service is fast indeed, but not the most reliable.

Want to monitor DNS too? WatchMouse offer DNS monitoring in their packages!