WatchMouse Laboratories

Posted by Pieter Ennes on March 11th, 2010

At the same time that we were working on some improvements for Apache JMeter, Simon from BadBoy Software has kindly made available a pre-release of his software that, next to many other improvements, particularly supports exporting scripts that use Variable Setter elements to JMeter using functionality from JMeter’s similar Regular Expression Extractor. This is relevant in cases where your application exposes its session ID via the web page body, for example in a (hidden) form element instead of using a cookie. This release will make the process of recording and uploading scripts directly from BadBoy much easier, which is terrific as we recommend BadBoy as a primary tool to record our script monitors. We’ve already got one customer using this enhancement!

The preview can be downloaded from: BadboyInstaller-2.1_beta_8_pre_1_wm.exe

On a related track: We’ve added a small island on our main dashboard reminding our scripting customers that if they use BadBoy to record and upload scripts in their WatchMouse account, they are in fact entitled to a free licence for BadBoy!

Thanks for the patch Simon!

Posted by Pieter Ennes on March 10th, 2010

I’m very happy to be able to publish another patch, this time for Apache’s Jakarta JMeter that adds two new metrics (a connect time and processing time) to all output channels. To cut right to the chase, this is how it looks:

Displayed is the breakdown of a typical HTTP redirect scenario, from google.com to google.co.uk in this case, in the JMeter GUI. Shown in the right pane, are the timings of the selected parent element, summarising both the initial 302 as well as the 200 response from the server. The original Latency and Load time metrics are shown, and in yellow the stuff that I’ve added.

Connect time
This one simply measures the TCP connect time in HTTP samplers, including any host name lookups that may have to be done. So the printed time indicates the amount of time between the start of the test and the socket being ready. In case of a redirect (as above), the connect times to the individual hosts (google.com and google.co.uk) are summed into the parent element to indicate the total connect time that was involved.

Processing time
The processing time is defined as the time-to-first byte just as the latency. But in contrast to the latency, the processing time from sub-requests is summed into the parent element, just like the connect time, page load time and page size.

Having both these numbers available can allow better analysis of a performance problem. For example: If the connect time stays the same but the processing time goes up, this may indicate a CPU-related problem on the server and not a network or DNS issue. But there are many other situations where a breakdown like this may help with performance research.

Resolve time
Another performance metric that is missing in JMeter is the DNS resolve time. We are currently looking for a way to measure this, but I expect to be able to include it in the next version of the patch. Having resolve time available as well, will allow for even better analysis of a problem.

Feedback
At WatchMouse we use JMeter as a back-end for our functional monitors, and we are interested in these extra measurements to give our users a more detailed performance breakdown of their transactions. After deployment to our monitoring network we will start integration with our performance charts and Root Cause Analysis pages.

As always, we’re keen on your feedback regarding this patch! What’s your opinion on the definitions of these metrics? What else would you like to know regarding the performance of your scripts? And how would you like to see this information come back in our dashboard?

Pieter Ennes
WatchMouse

References

• Patch: jmeter-ctime-ptime.diff.txt
• JMeter: issue #48799

Posted by mark on February 11th, 2010

If you are considering a cloud infrastructure, your first questions should be: Does it exist? Does it scale? With the WatchMouse network of monitoring stations, I did some research over the past year into the computing clouds to shed some light on these basic questions.

We looked at the uptime of a number of cloud providers, such as Amazon, Akamai, and Google AppEngine. The results are in the following figure. What we see is that cloud uptime is pretty good, and it is getting better. As a matter of fact, these clouds perform better than a lot of commercial websites.

Another interesting question is: where is the cloud? We looked at the distance from the WatchMouse monitoring stations to a Google App Engine application. The following figure shows that the App Engine cloud is in fact spread out around the world!

On the vertical axis we see a number of WatchMouse monitoring stations, so that each horizontal bar represents the connect time from that station to either a regular website in the Netherlands, or an application at Google Apps Engine. From the graph it is clear that that cloud is in more than one place on the globe. Given the programming model of Google Apps Engine, this means that it is very scalable. Our research also shows that this cloud is expanding. For a more in-depth discussion of this data go to the slidecast (slides plus audio) of my presentation at the 2009 Computer Measurement Group conference.

So if you still thought that clouds are a foggy notion, this research is a wake-up call. Clouds do exist, are scalable and are getting better.

This is a guest column by Peter van Eijk, owner of Digital Infrastructures, a consultancy firm. He publishes a blog at http://petersgriddle.net and is currently setting up the Computer Measurement Group’s Dutch chapter. If you speak Dutch, follow the blog and join the NLCMG LinkedIn group.  Contact information is listed on LinkedIn and on www.digitalinfrastructures.nl/contact

Posted by Pieter Ennes on December 9th, 2009

Google’s announcement to offer a public DNS service similar to OpenDNS was already discussed exhaustively last week. But we noticed that a lot of people wondered how the different public DNS recursors compare performance-wise, worldwide. We therefore did a small study in this field using the WatchMouse monitoring network.

Test method
We set up monitors for the following public DNS services:

• DNSAdvantage (by Neustar)
• DNSResolvers (by EasyDNS)
• Google
• OpenDNS

Another one, ScrubIT, turned out to be DOA and was removed from the test as neither of its servers answer queries, even though their blog says “It works!“. Yet another, OpenNIC was left out since its configuration requires a different set of IP addresses in each country. Finally, there are public servers from Level3 with widely known IP’s, but an official web page about this service seems hard to find. We’ve only just started monitoring them and therefore left them out in this test.

For a period of five days, we probed each of the above DNS services for five different records: ‘A’ records for cnn.com (TTL=300), bbc.co.uk (TTL=300) and www.techcrunch.com (TTL=1200); ‘MX’ records for yahoo.com (TTL=7200), and an ‘AAAA’ record for it.ipv6.watchmouse.com (TTL=28800). Each monitor was executed every 5 minutes and was configured to send a single UDP query (with no retries), and time out after 3 seconds without a reply. This set-up was duplicated to monitor two different recursors for each service provider; the obtained results for these were averaged.

For each DNS service roughly 18.000 queries were performed over a six day period (2 servers * 6 days * 288 probes/day * 5 monitors), rotating the 42 locations in our network.

We also added two reference monitors: one utilising a local recursor running on our monitoring station, and another one, dubbed ‘Direct’, querying one of the listed name servers directly.

Results
Before going into the performance details, let’s have a look at the relative number of errors per service:

First thing to note is that a local recursor generates less failures compared to a direct queries to one of the listed name servers, probably due of its ability to cache and prefer fast and correctly functioning servers over ones that are slow or failing. Both DNSAdvantage and OpenDNS do a good job in masking name server errors and minimising lookup time-outs; their failure rate was below 5‰. DNSResolvers seems to have a common failure rate of just under 10‰. Google’s free service, on the other hand, fails to return records within the 3 second time-out in about 15‰ of the queries. That’s worse than a direct query (13‰) to one of the provider servers or through a local resolver (9‰). And it’s three times as high as the two other major competitors.

When we consider the performance of the queries that did receive a valid response, it can be seen that a query through a local resolver is a little bit slower than using a direct query, again on average, worldwide. The difference of ~20ms, however, cannot easily be explained by taking into account the (negligible) round trip time of the UDP query to the local host. The variance in performance from the local resolver is also increased, so it would be likely that other factors play a role here.

DNSResolvers either have very busy servers, or do not seem to do a good job in reducing network latency. Their performance is nearly twice as bad compared to using the local resolver. Most likely their service does not facilitate something called Anycast to route queries to a nearby data centre. The remaining providers do use anycasting, and clearly have an advantage because of this.

Discarding time-outs, Google’s public DNS (59ms on average) definitely is the best in terms of performance, with OpenDNS in a photo finish with a 80ms score. But also the services offered by DNSAdvantage display solid sub-100ms performance with 93ms.

Discussion
Most of the tested host names were either for high-volume sites or had large TTL’s, causing public caches to be easily primed and expose their qualities. OpenDNS, Google and DNSAdvantage all show that they master this and have better lookup times than a local resolver or a direct query.

To avoid influences of second order effects, the measurements were done using only a single UDP query in each probe. By doing so, we were able to separate real query performance from packet loss and server failures. In real life, however, a typical PC would retry the query after a certain time (~5 seconds). Our DNS monitors can do this, but this would cause the failures to be blended into the performance measurements, inducing an (arbitrary) bias from the chosen time-out setting.

Also, real people (or offices for that matter) are most often not in multiple places at the same time. Thus in real life one would be more interested in what the best service provider is in your area. We hope to have some time for a second blog item on this in the near future, with typical user settings, and a breakdown per area.

Measuring averaged worldwide performance the way we did now is still nice as a synthetic benchmark. So for fun, and because everyone finds DNS response times very important, we can introduce a DNS time wasting score. Based on, say, an 100 sequential lookups per day for an average user and a time-out of 3 seconds, the failures induce extra waiting time and influence the score:

(10 seconds per day adds up to roughly one hour of DNS waiting per year)

Conclusion
Three of the researched providers clearly are competitive in terms of ‘clean’ performance and offer a useful service to the public. But the number of failures shown in the first chart must be considered as an intrinsic part of the quality of service.

And this is where Google, with a 3 times higher failure rate, seems to have more problems than OpenDNS and DNSAdvantage. For 15 out of 1000 host name lookups, Google fails to respond within 3 seconds (or the packet is just lost), causing an extra lookup after a time-out and an observed lookup time of multiple seconds. It seems that the Google service is fast indeed, but not the most reliable.

Want to monitor DNS too? WatchMouse offer DNS monitoring in their packages!

Posted by Pieter Ennes on November 18th, 2009

Today, we received the final results from the second IPv6 questionnaire we sent out to our hosting partners.
While the results of the first poll, which was held back in 2007, revealed a total lack of interest in IPv6 at the time, the new results (two years later) show something different.

To summarize, almost 40% of our current partners offer native IPv6 already, or expect to be able to do so within the next six months. The table below shows a breakdown for different regions in the world:

Even though our statistics are coarse, the results seem to indicate a lack of interest in the North-American area. But they also show that in both Europe as well as in Asia and Australia, penetration is much better. Almost 50% of the parties in Europe can feed us with native IPv6 down the wire within six months. Go go go!

Are you a hosting provider offering quality IPv6-enabled hosting services in an area that is not currently covered by our network? Please do send us a note!

UPDATE: We’re very pleased to hear that WatchMouse were selected as one of the finalists for the Dutch IPv6 awards today!

Posted by Pieter Ennes on October 19th, 2009

Update: Post updated for v0.2…

Hi folks,

While our own migration to IPv6-enabled monitoring is progressing nicely, we are making available our version of some rudimentary MySQL User Defined Functions (UDF’s) that allow you to work with IPv6 addresses in pre-6.0 versions of MySQL.

The currently implemented functions are inet6_pton() and inet6_ntop() to convert IPv6 (and IPv4) addresses between presentation and numeric (or binary) form directly in SQL. The former function converts a readable IP address string to a binary representation of either 4 bytes (IPv4) or 16 bytes (IPv6) long. The latter function does exactly the opposite, as shown in:

mysql> select inet6_ntop(inet6_pton('2001:4860:a005::68'));
+----------------------------------------------+
| inet6_ntop(inet6_pton('2001:4860:a005::68')) |
+----------------------------------------------+
| 2001:4860:a005::68                           |
+----------------------------------------------+
1 row in set (0.00 sec)

mysql> select inet6_ntop(inet6_pton('1.2.3.4'));
+-----------------------------------+
| inet6_ntop(inet6_pton('1.2.3.4')) |
+-----------------------------------+
| 1.2.3.4                           |
+-----------------------------------+
1 row in set (0.00 sec)

mysql> select
length(inet6_pton('1.2.3.4')) as ipv4len,
length(inet6_pton('2001:4860:a005::68')) as ipv6len;
+---------+---------+
| ipv4len | ipv6len |
+---------+---------+
|       4 |      16 |
+---------+---------+
1 row in set (0.00 sec)

You can neatly store the output of inet6_pton() in a VARBINARY(16) column (using a VARCHAR(16) would not be wise because because of possible character set conversion problems).

Of course, there are many ways one could store IPv6 addresses in a database, but the method described here seems to work best for us and is unlikely to conflict with any native support for IPv6 that MySQL will implement in future versions. Furthermore, the binary notation seems to work nicely with ranges, so you can use WHERE ip BETWEEN a AND b to do something like:

mysql> select * from log
where ip between inet6_pton('2001:db8::') and inet6_pton('2001:db9::');

to get all IP addresses in the 2001:db8::/32 range.

We’ve also included two address lookup functions that are IPv6 compatible:

mysql> select inet6_lookup('www.watchmouse.com');
+------------------------------------+
| inet6_lookup('www.watchmouse.com') |
+------------------------------------+
| 2001:1938:80:73::2 |
+------------------------------------+
1 row in set (2.08 sec)

mysql> select inet6_rlookup('2a02:f8::ffff:ffe5'), inet6_rlookup('64.128.190.61');
+-------------------------------------+--------------------------------+
| inet6_rlookup('2a02:f8::ffff:ffe5') | inet6_rlookup('64.128.190.61') |
+-------------------------------------+--------------------------------+
| it.watchmouse.com | ny.watchmouse.com |
+-------------------------------------+--------------------------------+
1 row in set (0.07 sec)

As you can see, inet6_lookup takes a host name and converts it to a readable IP address. The reverse lookup function, inet6_rlookup, works on both binary as well as human readable IP addresses (since v0.2).

The inet6_pton() and inet6_ntop() functions should be quite fast, but as each call to inet6_lookup() involves a DNS lookup, the lookup functions can make your queries extremely slow. Make sure you LIMIT your result set to something sensible. In general, I would not recommend to use the lookup functions in production code as I don’t know the impact of these queries on MySQL’s performance.

Once the UDF’s are installed in the MySQL plugin directory, they can be loaded at runtime using:

mysql> CREATE FUNCTION inet6_ntop RETURNS STRING SONAME "mysql_udf_ipv6.so";
mysql> CREATE FUNCTION inet6_pton RETURNS STRING SONAME "mysql_udf_ipv6.so";
mysql> CREATE FUNCTION inet6_lookup RETURNS STRING SONAME "mysql_udf_ipv6.so";
mysql> CREATE FUNCTION inet6_rlookup RETURNS STRING SONAME "mysql_udf_ipv6.so";

These functions have helped us with our immediate needs for managing IPv6 addresses in our DBMS, and hopefully they can help you too. We have some more improvements in mind, so stay tuned if you’re interested!

The current version of the UDF’s can be downloaded from:

mysql-udf-ipv6-0.2.tar.gz

Note: Compilation was tested on Debian etch+lenny, Ubuntu jaunty+karmic, and on FreeBSD. If you get it running on other platforms, send us a patch!

Pieter Ennes
WatchMouse