Posted by Pieter Ennes on February 15th, 2011
A few weeks ago we formally launched Real Browser Monitoring after a soft introduction last December. We’re really excited about this new monitoring capability, and are working hard to add more meat to this type of monitoring. Currently we’re adding support for:
- Filling forms (POST) in a page
- Multi-step monitors with Real Browsers
- Flash (that’s actually a hard one to scale over all our customers)
In this post I’d like to share some of the implementation details, what additional improvements we’re working on, and a bit on other approaches we’re exploring.
Why Real Browser monitoring
As many website rely extensively on third party content (images, like-buttons, ads, …) and client side code (JavaScript), we see that in synthetic website monitoring too, there is a shift from basic measurements using HTTP probes, towards monitoring of complex pages including the third party content and code. And because the visitors of sites will use different browsers, there is also a need for insight into how the site loads in different browsers.
To satisfy this demand, many website monitoring companies have been competing to set up arrays of servers running Linux + Firefox, or Microsoft Windows + various flavours of Internet Explorer. For each real browser check, they need to fire up an instance of the browser (and in the case of IE, most likely including the surrounding virtual machine), load the web page, retrieve the results, and kill the instance. From an engineering point of view this scenario has room for optimisations, and from a operational point of view, with thousands of monitors that need to be executed every couple of minutes, scaling this up becomes key.
So at WatchMouse we looked at the problem from a different angle: by researching the main differences between browsers, then interviewing clients and partners which of these difference they found most important, and finally building a monitoring solution that is both satisfactory in terms of browser profiling, and has some extra features that are hard to offer using the more brute force method mentioned before. Let me try to describe some of the gory details.
Browser profiles
A lot of research on key inter-browser differences has been done by the great people involved in browserscope.org, a community-driven project for profiling web browsers. In fact, the BrowserScope network tab (also containing Steve Souders’ excellent UA profiler work) is at the basis of our solution: Instead of firing up every browsers executable out on the market and trying to puppeteer it to load a page, on various platforms, for thousands of monitors, we embrace the BrowserScope network parameters and exploit the fine-grained control that we have over a single engine and simulate the different browsers.
Some of the key factors in browsers that affect the performance are:
- the number of parallel connections that the browser can open to the same host
- the way the browser handles concurrent downloads of scripts, images and CSS files
Using these parameters we realized the emulation of a number of actual browsers, which we fine-tuned by comparing the waterfall charts of different browsers on various test sites to the waterfall charts rendered by the emulation.
More on the different parameters can be found on the site of the browserscope.org project. Some factors, however, I would like to discuss in more detail below.
Resolving host names
DNS lookups are nowadays handled by most browsers in a similar way: By resolving host names asynchronously as soon as they are available, even before there is a free TCP socket. The differences between browsers are therefore negligible when examining the full page load.
Some modern browsers (notably Chrome, Safari 5, and some versions of Firefox) also have a feature called DNS pre-fetching. This option is unrelated to the above and doesn’t affect loading times of single web pages. In multi-step transactions, however, the option enables resolving host names in links the user may click on in the near future. There are some privacy concerns with this and we will most probably leave it disabled in any future multi-step monitors. With multi-step transactions in mind, I propose to add a new BrowserScope network parameter, indicating whether DNS pre-fetching is enabled by default or not. In single-step monitors there should be no measurable influence on performance anyway.
But what about IPv6, or AAAA/A vs. A/AAAA lookups?
Still on the topic of resolve time, in case one of our stations has IPv6 connectivity it is configured to perform DNS lookups in accordance with RFC 3484. On the other hand, early versions of Windows seem to have had no or non-standard IPv6 support. In other words: Even with IPv6 connectivity available, Windows may be preferring IPv4 above IPv6. More modern versions (i.e. Windows Vista/Server after 2008) seem to adhere to RFC 3484 just like the nodes in our monitoring network.
Mac OS X also had it’s share of problems with RFC 3484, but the other way around: it always prefers IPv6. This situation doesn’t arise on our network, and unless someone would be specifically interested in synthetic testing using this broken set up, we can ignore it.
Summarizing: We ignore any platform-induced differences with respect to name resolution, but as explained above, this should have minimal impact on the way a single page will be loaded.
TCP connect times and simultaneous connections
The overall connect times are significantly influenced by the number of connections a browser opens to each host, and to all hosts in total. Our engine sets the number of connections based on the previously mentioned research by Souders. We can therefore expect (and confirmed this in our experiments) that the connect times are very similar to those of the real browser.
Response times
Response times can be split in two components: the actual server response time and any network-induced latency. The latter will depend only on geographical location, and hardly on OS or browser version. The former, the server response time, can be influenced by the number of concurrent connections opened to the server by the client, or in theory the ordering of the requests. However, as our engine sets the correct number of connections for each browser profile, only the ordering of the requests may be different. As of yet we did not see any difference in practice due to this.
Total pageload time
After obtaining a HAR file describing the pageload, we use many of the remaining BrowserScope network parameters to do the last step in fine-tuning the pageload profile to the selected browser. For example, looking at the JS/CSS concurrency levels and information from the HTML, together tells us how the elements are to be ordered.
JavaScript execution times
For now, we don’t touch these at all, at least until we have done more research on how to heuristically predict the changes.
Pros and cons of the profiling approach
Benefits:
- Accurate timings (using real browser events)
- Waterfalls for all profiles, not just one
- HTTP authentication and SSL client certificates for all profiles
- Matching on dynamic texts for all profiles
- Snapshots for all profiles
- Scalability, immediacy
Drawbacks:
- Single JavaScript engine
- Rendering quirks and artefacts won’t show
- No interpretation of conditional comments (like <!--[if IE]>)
An independent expert opinion
Further considerations are pointed out by Aaron Peters, an independent Web Performance consultant:
It’s great to see WatchMouse acknowledging the added value of Real Browser Monitoring and adding this as a service to their customers. Measuring page load times with real browsers is key to understanding the end user experience.
he says, while adding:
WatchMouse takes an interesting and innovative approach by using a single platform and then applying algorithms based on browser profiles to calculate the ‘actual’ page load times. The results I have seen so far are impressive
Aaron does believe, however, that for certain web pages the above mentioned drawbacks can have a significant impact on the results:
For example, the performance of JavaScript engines in browsers differ greatly. On a JavaScript-heavy page, the speed of the JavaScript engine can have a large impact on the overall page load times and user experience. In order to make their RBM service top of the bill, WatchMouse will have to get those last few important browser differences into their algorithm.
Conclusion
We believe that we have built a product that allows us to offer competitive, scalable and user-friendly real browser monitoring in a different way. We show that the profile approach has advantages as well as limitations, but when used in a synthetic monitoring environment, the benefits outweigh the imperfections by far.
If you feel you don’t get enough browser-specific detail from one of your checks, then it’s easy to do a single-shot measurement on the page in question using one of the many terrific tools out there.
When more insight is needed into different browsers on a continuous basis, passive (i.e. non-synthetic) methods may be a suitable alternative. Naturally, we’re working hard to get our version of Real User Monitoring (RUM) out there as soon as possible!
Next steps
Having this product out in the open allows us to gather bigger volumes of data required to tune the profiles further. A current topic of interest is to see if we can predict how the first-visual event translates across browsers. We’re also researching to install real browsers on a small set of nodes to be used as a reference platform.
But that’s not all and we’re certainly not finished yet! So let us know in case you have any thoughts on this matter.
Pieter Ennes
VP Engineering
References
- UA Profiler, http://stevesouders.com/ua/
- DNS Prefetching, http://www.chromium.org/developers/design-documents/dns-prefetching
- DNS Prefetching and Its Privacy Implications, http://www.usenix.org/event/leet10/tech/full_papers/Krishnan.pdf
- RFC 3484, http://www.ietf.org/rfc/rfc3484.txt
- RFC 3484 in Windows, http://support.microsoft.com/kb/969029
- Measuring and combating IPv6 brokenness, http://ripe61.ripe.net/presentations/162-ripe61.pdf
Posted by Pieter Ennes on November 18th, 2009
Today, we received the final results from the second IPv6 questionnaire we sent out to our hosting partners.
While the results of the first poll, which was held back in 2007, revealed a total lack of interest in IPv6 at the time, the new results (two years later) show something different.
To summarize, almost 40% of our current partners offer native IPv6 already, or expect to be able to do so within the next six months. The table below shows a breakdown for different regions in the world:
| Region | # hosters | IPv6 penetration |
|---|
| North-America | 7 | 14% |
| Europe | 18 | 50% |
| Asia & Australia | 7 | 43% |
| Rest of world | 2 | 0% |
| Total | 34 | 38% |
Even though our statistics are coarse, the results seem to indicate a lack of interest in the North-American area. But they also show that in both Europe as well as in Asia and Australia, penetration is much better. Almost 50% of the parties in Europe can feed us with native IPv6 down the wire within six months. Go go go!
Are you a hosting provider offering quality IPv6-enabled hosting services in an area that is not currently covered by our network? Please do send us a note!
UPDATE: We’re very pleased to hear that WatchMouse were selected as one of the finalists for the Dutch IPv6 awards today!
Posted by Pieter Ennes on October 19th, 2009
Update: Post updated for v0.2, v0.3, and v0.5. As of v0.5 there is a valid upgrade path towards native IPv6 support in MySQL 5.6; please see below.
Hi folks,
While our own migration to IPv6-enabled monitoring is progressing nicely, we are making available our version of some rudimentary MySQL User Defined Functions (UDF’s) that allow you to work with IPv6 addresses in pre-6.0 versions of MySQL.
The currently implemented functions are inet6_pton() and inet6_ntop() to convert IPv6 (and IPv4) addresses between presentation and numeric (or binary) form directly in SQL. The former function converts a readable IP address string to a binary representation of either 4 bytes (IPv4) or 16 bytes (IPv6) long. The latter function does exactly the opposite, as shown in:
mysql> select inet6_ntop(inet6_pton('2001:4860:a005::68'));
+----------------------------------------------+
| inet6_ntop(inet6_pton('2001:4860:a005::68')) |
+----------------------------------------------+
| 2001:4860:a005::68 |
+----------------------------------------------+
1 row in set (0.00 sec)
mysql> select inet6_ntop(inet6_pton('1.2.3.4'));
+-----------------------------------+
| inet6_ntop(inet6_pton('1.2.3.4')) |
+-----------------------------------+
| 1.2.3.4 |
+-----------------------------------+
1 row in set (0.00 sec)
mysql> select
length(inet6_pton('1.2.3.4')) as ipv4len,
length(inet6_pton('2001:4860:a005::68')) as ipv6len;
+---------+---------+
| ipv4len | ipv6len |
+---------+---------+
| 4 | 16 |
+---------+---------+
1 row in set (0.00 sec)
You can neatly store the output of inet6_pton() in a VARBINARY(16) column (using a VARCHAR(16) would not be wise because because of possible character set conversion problems).
Of course, there are many ways one could store IPv6 addresses in a database, but the method described here seems to work best for us and is unlikely to conflict with any native support for IPv6 that MySQL will implement in future versions. Furthermore, the binary notation seems to work nicely with ranges, so you can use WHERE ip BETWEEN a AND b to do something like:
mysql> select * from log
where ip between inet6_pton('2001:db8::') and inet6_pton('2001:db9::');
to get all IP addresses in the 2001:db8::/32 range.
We’ve also included two address lookup functions that are IPv6 compatible:
mysql> select inet6_lookup('www.watchmouse.com');
+------------------------------------+
| inet6_lookup('www.watchmouse.com') |
+------------------------------------+
| 2001:1938:80:73::2 |
+------------------------------------+
1 row in set (2.08 sec)
mysql> select inet6_rlookup('2a02:f8::ffff:ffe5'), inet6_rlookup('64.128.190.61');
+-------------------------------------+--------------------------------+
| inet6_rlookup('2a02:f8::ffff:ffe5') | inet6_rlookup('64.128.190.61') |
+-------------------------------------+--------------------------------+
| it.watchmouse.com | ny.watchmouse.com |
+-------------------------------------+--------------------------------+
1 row in set (0.07 sec)
As you can see, inet6_lookup takes a host name and converts it to a readable IP address. The reverse lookup function, inet6_rlookup, works on both binary as well as human readable IP addresses (since v0.2).
The inet6_pton() and inet6_ntop() functions should be quite fast, but as each call to inet6_lookup() involves a DNS lookup, the lookup functions can make your queries extremely slow. Make sure you LIMIT your result set to something sensible. In general, I would not recommend to use the lookup functions in production code as I don’t know the impact of these queries on MySQL’s performance.
Once the UDF’s are installed in the MySQL plugin directory, they can be loaded at runtime using:
mysql> CREATE FUNCTION inet6_ntop RETURNS STRING SONAME "mysql_udf_ipv6.so";
mysql> CREATE FUNCTION inet6_pton RETURNS STRING SONAME "mysql_udf_ipv6.so";
mysql> CREATE FUNCTION inet6_mask RETURNS STRING SONAME "mysql_udf_ipv6.so";
mysql> CREATE FUNCTION inet6_lookup RETURNS STRING SONAME "mysql_udf_ipv6.so";
mysql> CREATE FUNCTION inet6_rlookup RETURNS STRING SONAME "mysql_udf_ipv6.so";
UPDATE: As suggested by Matthias in the comments, in version 0.3 I’ve included a mask calculation function that may help you when comparing IP address ranges. You could use it like this:
mysql> select inet6_mask(inet6_pton('2001:db8:1::1'), 64) = inet6_mask(inet6_pton('2001:db8:1::2'), 64) as samenet;
+---------+
| samenet |
+---------+
| 1 |
+---------+
1 row in set (0.00 sec)
These functions have helped us with our immediate needs for managing IPv6 addresses in our DBMS, and hopefully they can help you too. We have some more improvements in mind, so stay tuned if you’re interested!
The latest version of the UDF’s is available for download here. Additionally, all code is available via a BitBucket repository on:
http://bitbucket.org/watchmouse/mysql-udf-ipv6/
Note: Compilation was tested on Debian, Ubuntu, and FreeBSD. If you get it running on other platforms, send us a patch!
Pieter Ennes
WatchMouse
PS. I’m happy to note that MySQL finally added native IPv6 support functions in v5.6.3 of their product. The good news is that they chose to use exactly the same method, only using slightly different names. So I’ve now added aliases for the native names in v0.5 of the UDF, creating a valid upgrade path from v5.1+UDF towards the native functions in v5.6.
More information on the native functions can be found here:
http://dev.mysql.com/doc/refman/5.6/en/miscellaneous-functions.html#function_inet6-ntoa
For more information on the aliases, please refer to the README file in the BitBucket repository:
https://bitbucket.org/watchmouse/mysql-udf-ipv6/overview
Posted by stan on October 13th, 2009
Earlier today we promoted the 1.6 version of the WatchMouse API to ‘stable’. Next to some minor fixes, the new API now fully supports IPv6 and UTF-8.
The WatchMouse API enables developers to tie in features of the WatchMouse monitoring service in their own application or website. Using the API you can access the data and functions of the WatchMouse services.
Some example uses of the API:
- Perform a check of your site from one of the worldwide monitoring stations of WatchMouse
- Access your WatchMouse account and set up, activate, modify, or remove site monitoring rules
- Schedule a Vulnerability Scan for your server
- Send an SMS from one of the many SMS gateways of WatchMouse
- Perform a traceroute from one of these stations to an IP address of your choice. The just-trace site was build using these services.
- Find where a certain host is located (IP to location)
- Compute how much 10 Euro is in US Dollars.
- …
IPv6 support
IPv6 is an important feature in this release, and it actually took more than just the API to fully support it. We upgraded monitoring stations, the core monitoring software, our website, and the databases in our back-end systems to make it all work. And let’s not forget the hosting providers for our 40+ stations world wide. But that’s a story worthy of a full post coming soon 
For most API calls you will need a WatchMouse account (or an account with one of our partners) and the number of calls is rate-limited on hourly or daily basis, but some calls can be used anonymously too. See the Access to the API chapter for details.
Your mashup here
Already a number of mash-ups were created, like the Just-ping site and the Down-or-Not site, and we will see a number of other applications down the road. If you use our API, just let us know in a comment below, and don’t forget to list it as a mash-up at Programmable web!
For documentation, please read on here:
Hope to see a thousand applications bloom!
Stan P. van de Burgt
Posted by Pieter Ennes on September 8th, 2009
Which hosters are running IPv6, and what monitoring tools are needed? We are trying to find out!
With a bit of a delay we want to share the results of a poll we held among the hosting providers of our monitoring network in the beginning of 2008. We asked them whether they could offer native IPv6 to our stations, and if not, when they could.
The results were interesting, as none of them could at that time deliver native v6 traffic right down to our stations, and only two were up for a joint experiment with us. Very few responded that they had IPv6 on the roadmap at all; the rest indicating that they had not thought about it yet. Two responded that they were in fact waiting on some IPv6 address space from APNIC. Assuming that our selection of (at that time 25) providers is representative for hosters in general, pioneers are still in bad shape if they depend on upstream network infrastructure.
So what has changed since then? The AMS-IX IPv6 bandwidth charts for 2009 show that there has only been a minor increase in v6 traffic recently. You can also see that IPv6 traffic still accounts for only a tiny 0.2% of the total traffic volume. We will conduct a second poll now to see if matters have changed, although the above statistics are pessimistic. I will update this blog with a more detailed comparison study when we have new results.
With respect to WatchMouse, we think early adopters should have at least access to tools to monitor all the aspects of their systems during their transition to IPv6. We therefore are currently aggressively making our own infrastructure IPv6-ready to offer IPv6 monitoring as good as we can. For locations where we find no support from our upstream providers, we will be using v4 tunnels through HE, SixXS and other providers. Besides that, our developers are currently testing the necessary code changes for full IPv6 deployment.
Some of the things we are working on right now:
- Finalizing deployment of a IPv6 capable monitoring network using tunnel brokers
- Make our core systems, databases and checker software ready for IPv6
- Conduct a second poll amongst our hosters to verify the current state of v6 penetration
- Make the WatchMouse website itself available on v6
We will post any interesting information regarding our progress in this blog when we have it. In the meantime, we would like to ask what kind of tools you would need from us to be able to monitor your own transition; why not leave a comment on this site!