WatchMouse Laboratories

Posted by mark on March 29th, 2011

Release of open sourced script loadrunner2jmx

Today we released some free and open source code to convert HP LoadRunner scripts to Apache JMeter scripts.

Both LoadRunner and JMeter are used to record and replay multi-step transactions on web sites, web API’s etc. and can be used for both load testing and transaction monitoring. At WatchMouse we based our scripting monitors on JMeter because of its robustness, its very complete functionality and obviously, its open source license.

Why this conversion tool? Over the last year we have had a couple of requests from our clients to convert their existing LoadRunner scripts to JMeter so they could run the same tests in the WatchMouse monitoring network worldwide, without having to create or record these scripts again. Manual conversion from one format to the other is not straightforward, as the formats are very different, so we created a tool to do just that, and now decided to make that freely available.

Using the converter

The code can be downloaded from our bitbucket: https://bitbucket.org/watchmouse/loadrunner/.

The installation process simply consists of unzipping the archive in a directory of your choice.

Using the script is pretty straightforward. Run it without any arguments and the usage instructions are shown:

$ ./loadrunner2jmeter
Input filename(s) are required
Usage:
     ./loadrunner2jmeter <inputfile1.c>
               -or-
     ./loadrunner2jmeter <inputfile1.c> [ -o <outputfile.jmx> ]
               -or-
     ./loadrunner2jmeter <inputfile1.c> <inputfile2.c> <inputfile3.c> [...] -o <outputfile>

More can be read in the README.

Improving and extending the code

We had a limited set of LoadRunner scripts to test with, which means the script is probably not perfect, and doesn’t support all constructs yet, so your milage may vary. Your feedback is greatly appreciated and as the code has been released under the GPL open source license, anyone who wants to contribute to loadrunner2jmx is more than welcome to do so.

The script is written in Perl, so if you are a Perl hacker and you would like to help improving the code, please feel free to submit patches. The source can be found here: https://bitbucket.org/watchmouse/loadrunner/src

Also, if you have some LoadRunner scripts laying around, please let us know so we can use it to test and, if needed, upgrade loadrunner2jmeter.

Mark Pors
CTO & co-founder

Mark lives is Greece and loves Perl almost as much as he loves Emacs

Posted by Pieter Ennes on February 15th, 2011

A few weeks ago we formally launched Real Browser Monitoring after a soft introduction last December. We’re really excited about this new monitoring capability, and are working hard to add more meat to this type of monitoring. Currently we’re adding support for:

• Filling forms (POST) in a page
• Multi-step monitors with Real Browsers
• Flash (that’s actually a hard one to scale over all our customers)

In this post I’d like to share some of the implementation details, what additional improvements we’re working on, and a bit on other approaches we’re exploring.

Why Real Browser monitoring

As many website rely extensively on third party content (images, like-buttons, ads, …) and client side code (JavaScript), we see that in synthetic website monitoring too, there is a shift from basic measurements using HTTP probes, towards monitoring of complex pages including the third party content and code. And because the visitors of sites will use different browsers, there is also a need for insight into how the site loads in different browsers.

To satisfy this demand, many website monitoring companies have been competing to set up arrays of servers running Linux + Firefox, or Microsoft Windows + various flavours of Internet Explorer. For each real browser check, they need to fire up an instance of the browser (and in the case of IE, most likely including the surrounding virtual machine), load the web page, retrieve the results, and kill the instance. From an engineering point of view this scenario has room for optimisations, and from a operational point of view, with thousands of monitors that need to be executed every couple of minutes, scaling this up becomes key.

So at WatchMouse we looked at the problem from a different angle: by researching the main differences between browsers, then interviewing clients and partners which of these difference they found most important, and finally building a monitoring solution that is both satisfactory in terms of browser profiling, and has some extra features that are hard to offer using the more brute force method mentioned before. Let me try to describe some of the gory details.

Browser profiles

A lot of research on key inter-browser differences has been done by the great people involved in browserscope.org, a community-driven project for profiling web browsers. In fact, the BrowserScope network tab (also containing Steve Souders’ excellent UA profiler work) is at the basis of our solution: Instead of firing up every browsers executable out on the market and trying to puppeteer it to load a page, on various platforms, for thousands of monitors, we embrace the BrowserScope network parameters and exploit the fine-grained control that we have over a single engine and simulate the different browsers.

Some of the key factors in browsers that affect the performance are:

• the number of parallel connections that the browser can open to the same host
• the way the browser handles concurrent downloads of scripts, images and CSS files

Using these parameters we realized the emulation of a number of actual browsers, which we fine-tuned by comparing the waterfall charts of different browsers on various test sites to the waterfall charts rendered by the emulation.

More on the different parameters can be found on the site of the browserscope.org project. Some factors, however, I would like to discuss in more detail below.

Resolving host names

DNS lookups are nowadays handled by most browsers in a similar way: By resolving host names asynchronously as soon as they are available, even before there is a free TCP socket. The differences between browsers are therefore negligible when examining the full page load.

Some modern browsers (notably Chrome, Safari 5, and some versions of Firefox) also have a feature called DNS pre-fetching. This option is unrelated to the above and doesn’t affect loading times of single web pages. In multi-step transactions, however, the option enables resolving host names in links the user may click on in the near future. There are some privacy concerns with this and we will most probably leave it disabled in any future multi-step monitors. With multi-step transactions in mind, I propose to add a new BrowserScope network parameter, indicating whether DNS pre-fetching is enabled by default or not. In single-step monitors there should be no measurable influence on performance anyway.

But what about IPv6, or AAAA/A vs. A/AAAA lookups?

Still on the topic of resolve time, in case one of our stations has IPv6 connectivity it is configured to perform DNS lookups in accordance with RFC 3484. On the other hand, early versions of Windows seem to have had no or non-standard IPv6 support. In other words: Even with IPv6 connectivity available, Windows may be preferring IPv4 above IPv6. More modern versions (i.e. Windows Vista/Server after 2008) seem to adhere to RFC 3484 just like the nodes in our monitoring network.

Mac OS X also had it’s share of problems with RFC 3484, but the other way around: it always prefers IPv6. This situation doesn’t arise on our network, and unless someone would be specifically interested in synthetic testing using this broken set up, we can ignore it.

Summarizing: We ignore any platform-induced differences with respect to name resolution, but as explained above, this should have minimal impact on the way a single page will be loaded.

TCP connect times and simultaneous connections

The overall connect times are significantly influenced by the number of connections a browser opens to each host, and to all hosts in total. Our engine sets the number of connections based on the previously mentioned research by Souders. We can therefore expect (and confirmed this in our experiments) that the connect times are very similar to those of the real browser.

Response times

Response times can be split in two components: the actual server response time and any network-induced latency. The latter will depend only on geographical location, and hardly on OS or browser version. The former, the server response time, can be influenced by the number of concurrent connections opened to the server by the client, or in theory the ordering of the requests. However, as our engine sets the correct number of connections for each browser profile, only the ordering of the requests may be different. As of yet we did not see any difference in practice due to this.

Total pageload time

After obtaining a HAR file describing the pageload, we use many of the remaining BrowserScope network parameters to do the last step in fine-tuning the pageload profile to the selected browser. For example, looking at the JS/CSS concurrency levels and information from the HTML, together tells us how the elements are to be ordered.

JavaScript execution times

For now, we don’t touch these at all, at least until we have done more research on how to heuristically predict the changes.

Pros and cons of the profiling approach

Benefits:

• Accurate timings (using real browser events)
• Waterfalls for all profiles, not just one
• HTTP authentication and SSL client certificates for all profiles
• Matching on dynamic texts for all profiles
• Snapshots for all profiles
• Scalability, immediacy

Drawbacks:

• Single JavaScript engine
• Rendering quirks and artefacts won’t show
• No interpretation of conditional comments (like <!--[if IE]>)

An independent expert opinion

Further considerations are pointed out by Aaron Peters, an independent Web Performance consultant:

It’s great to see WatchMouse acknowledging the added value of Real Browser Monitoring and adding this as a service to their customers. Measuring page load times with real browsers is key to understanding the end user experience.

he says, while adding:

WatchMouse takes an interesting and innovative approach by using a single platform and then applying algorithms based on browser profiles to calculate the ‘actual’ page load times. The results I have seen so far are impressive

Aaron does believe, however, that for certain web pages the above mentioned drawbacks can have a significant impact on the results:

For example, the performance of JavaScript engines in browsers differ greatly. On a JavaScript-heavy page, the speed of the JavaScript engine can have a large impact on the overall page load times and user experience. In order to make their RBM service top of the bill, WatchMouse will have to get those last few important browser differences into their algorithm.

Conclusion

We believe that we have built a product that allows us to offer competitive, scalable and user-friendly real browser monitoring in a different way. We show that the profile approach has advantages as well as limitations, but when used in a synthetic monitoring environment, the benefits outweigh the imperfections by far.

If you feel you don’t get enough browser-specific detail from one of your checks, then it’s easy to do a single-shot measurement on the page in question using one of the many terrific tools out there.

When more insight is needed into different browsers on a continuous basis, passive (i.e. non-synthetic) methods may be a suitable alternative. Naturally, we’re working hard to get our version of Real User Monitoring (RUM) out there as soon as possible!

Next steps

Having this product out in the open allows us to gather bigger volumes of data required to tune the profiles further. A current topic of interest is to see if we can predict how the first-visual event translates across browsers. We’re also researching to install real browsers on a small set of nodes to be used as a reference platform.

But that’s not all and we’re certainly not finished yet! So let us know in case you have any thoughts on this matter.

Pieter Ennes
VP Engineering

References

1. UA Profiler, http://stevesouders.com/ua/
2. DNS Prefetching, http://www.chromium.org/developers/design-documents/dns-prefetching
3. DNS Prefetching and Its Privacy Implications, http://www.usenix.org/event/leet10/tech/full_papers/Krishnan.pdf
4. RFC 3484, http://www.ietf.org/rfc/rfc3484.txt
5. RFC 3484 in Windows, http://support.microsoft.com/kb/969029
6. Measuring and combating IPv6 brokenness, http://ripe61.ripe.net/presentations/162-ripe61.pdf

Posted by Pieter Ennes on August 11th, 2010

Hi folks,

At WatchMouse we’ve been offering worldwide Web Application Testing (a.k.a. ‘scripting’) based on a customised version of Apache JMeter for some years now. This has been working really well in a wide variety of applications, like:

• Full page load simulation (including page assets)
• Website transaction monitoring (including HTTP headers, cookies etc.)
• SOAP or REST-like API testing (including variable extraction and conditional execution)

But the real strength of using JMeter for our functional monitoring is that our users remain in control, as they can craft monitoring scripts on their desktop using powerful tools and then easily upload them via our website. Or they could reuse a set of already existing load testing scripts and upload them for periodical monitoring after deployment. There is no lock-in, so whenever you choose to leave, you can download your scripts and still be able to run them using free and open-source tools. WatchMouse just adds indispensible bling like professional alerting, visualisations, and a worldwide network of 50 stations.

The HTTP Archive format as common denominator

Even though our clients keep finding new ways to employ JMeter, what do these different types of scripts have in common? They are similar in that all sequences:

• Are based on HTTP
• Can be multi-step
• Can contain subrequests
• Measure ‘performance’

Looking at these factors, we were very excited to learn about the the introduction and rapid adoption of the HTTP Archive format (HAR), an open specification for page load measurements. Because what HAR can store is exactly all of the above, plus more if you want.

JMeter to HAR converter

Based on the observed similarities between JMeter and HAR, I decided to build a simple JMeter-to-HAR converter as a quick proof of concept. The output of the first conversion turned out to be accepted by Jan Odvarko’s authoritative online HAR viewer which displayed the multiple steps and subrequests correctly, as well as some of the HTTP properties. In a second version of the script it also proved easy to convert many of the other elements common to both JMeter and HAR, e.g.:

• HTTP status, response message, method and version
• HTTP request and response headers
• HTTP Cookies
• Mime types
• Redirected URL’s
• Page sizes and compression
• Most of the performance metrics (dns, connect, wait, receive)

Additionally, I chose to convert JMeter’s notion of assertions into a private _assertions element in the HAR file, although we don’t currently use this elsewhere. If someone wants to try it, I made the code to the converter is available on BitBucket. It’s written in PHP since that matches with our own web environment, and I’ll be updating the script occasionally to stay in line with the most recent version of HAR.

HAR at WatchMouse: Script breakdowns and waterfalls

Internally at WatchMouse, we have already been using the converter for some time. We’ve incorporated Jan’s excellent waterfall charts in our dashboard, making FireBug-like visualisations available for all standard script monitors. We are planning to add more support for HAR in the future!

Support for non-HTTP protocols?

However simple it is, one of the things the converter shows is the close relation between JMeter and HAR. Yet, there are incompatibilities, as JMeter can do much more than only HTTP monitoring. So can JMeter support HAR? As an output format for all it’s HTTP requests it certainly could. But the current HAR format is very much geared towards HTTP only (if only for it’s name), so as a generic output format it would not be suited.

In parallel, at WatchMouse we are steadily moving towards a new storage solution for our check data based on MongoDB, and for that, we are researching an archive format that supports storing any type of performance measurement in JSON format.

Sounds like a generalised version of HAR in both cases? It could be! The success of HAR shows that performance measurements are ubiquitous nowadays and that the sector can benefit a lot from open formats. I’m confident that a generic archive format, if done properly, can set a standard and be used by many tools (like JMeter) and companies (like WatchMouse), both internally or in public interfaces.

If you have any thoughts on this subject, please feel free to leave a comment or email me directly.

Repository: http://bitbucket.org/watchmouse/har/
Issue tracker: http://bitbucket.org/watchmouse/har/issues

Pieter Ennes
VP Engineeringfadap

Posted by mark on July 20th, 2010

This article, a follow up on an earlier blogpost, gives a more detailed look at the location of the Amazon CloudFront service. This location is derived from the time it takes to connect to it from a number of locations.

The summary is that CloudFront is on the average about 40-50 milliseconds away from a random point on the Internet.  This is pretty good compared to a site located in e.g. New York (120 milliseconds) and is in the same league as other content distribution networks. In specific markets, it is pretty near: San Francisco: 3 milliseconds, New York: 13 milliseconds, Western Europe: 1-30 milliseconds.

According to Amazon, CloudFront is in 16 locations, in contrast to the S3 storage service and the EC2 compute service, which have only 4 points of presence around the world.

The following table gives distances (in milliseconds) from selected locations of the monitoring network to Amazon CloudFront (cities annotated with CF have CloudFront locations):



As the table shows, the proximity of CloudFront is uneven around the world.

CloudFront changes its connectivity regularly, mostly for the better. An interesting data point for example is that on April 8, CloudFront created a presence near Hong Kong dropping the distance from 160 milliseconds to 4 milliseconds. The following graph gives more detail.

In line with our earlier research, this data too shows that maintaining good proximity in an ever changing Internet is not a trivial thing to do. See for example the ever changing proximities to New York and New Zealand.

Our research was done in collaboration with Jitscale (a cloud consultancy)  and WatchMouse. Distances are measured by measuring a TCP connect to an http URL of an object provided by Cloudfront, and does not include DNS lookup.

This is a guest column by Peter van Eijk, owner of Digital Infrastructures, a consultancy firm. He publishes a blog at http://petersgriddle.net and is currently setting up the Computer Measurement Group’s Dutch chapter. If you speak Dutch, follow the blog and join the NLCMG LinkedIn group.  Contact information is listed on LinkedIn and on www.digitalinfrastructures.nl/contact

Posted by stan on June 22nd, 2010

Not many people run public facing LDAP (Lightweight Directory Access Protocol) servers, but if you do, we can help you keeping tabs on the availability and performance of your LDAP services, as seen by our monitoring stations worldwide.

In this post I’ll explain how to set up an LDAP monitor in your account. Beware, this is not for the faint of heart! If you never encountered a LDAP server, or do not know the access parameters of your organisations’ LDAP service, this post it not for you… Okay, here we go!

Setting up an LDAP monitor

First, check that your LDAP server is indeed public facing, i.e. that it can be reached over the Internet. It may be secured with user name and password, and only accessible over SSL, but is should be reachable from our monitoring stations, or at least the ones you select for the monitoring of your server.

Then go to your monitoring settings, add a new monitor, and select ldap as its type. Then click on expert mode and fill out the following fields:

• Host: The host name of the server
• Port: The TCP port where the LDAP service is listening, typically 389, or 636 with SSL.
• Path: The key to the information you want to retrieve. See below.
• Encryption: SSL or none.
• User name: If your LDAP server is only available to registered users, specify the account here. Note that this will probably be a LDAP path too, for example: cn=stan,ou=mt,dc=watchmouse,dc=com
• Password: the password associated with the user name.
• Match string: optionally a string (or regular expression, when enclosed in /-signs) to use to verify the output of the LDAP query.

Optionally you may want to change timing settings, IPv6 preference, interval, certificate checks, etc.

Now the path is a bit complex, as it is composed of four parts, separated by question marks.

/base_dn?attributes?scope?filter

Here we follow the LDAP URL Format (RFC2255):

1. base_dn: Distinguished name (DN) of an entry in the directory. This DN identifies the entry that is the starting point of the search. If no base DN is specified, the search starts at the root of the directory tree.
2. attributes: The attributes to be returned. To specify more than one attribute, use commas to separate the attributes (for example, ”cn,mail,telephoneNumber”). If no attributes are specified in the URL, all attributes are returned.
3. scope: The scope of the search, which can be base, one, or sub. If no scope is specified, the server performs a base search.
   • base retrieves information about the distinguished name (base_dn) specified in the URL only.
   • one retrieves information about entries one level below the distinguished name (base_dn) specified in the URL. The base entry is not included in this scope.
   • sub retrieves information about entries at all levels below the distinguished name (base_dn) specified in the URL. The base entry is included in this scope.
4. filter: Search filter to apply to entries within the specified scope of the search.

An LDAP example

Host ldap.itd.umich.edu and path dc=umich,dc=edu?uid,cn?sub?uid=mcs

This path indicated that the search starts at dc=umich,dc=edu, that only the UID and CN attributes are retrieved, all levels below the start node are searched (sub), and only the nodes with uid equal to ‘mcs‘ are returned. Finally, we check if the string “Mark Christo Smith” is in the results.

Are you still with us? Good! Now try to set up a monitor for your own LDAP server. And while you set it all up, why not add this new monitor you created to your Public Status Page and share the status of your server with your visitors? Transparency is the new trend!

Hope you enjoyed this post, and if you have any problems setting up a monitor for your LDAP service, just open a ticket at the helpdesk, and we’ll follow up immediately.

- Stan

Posted by stan on April 26th, 2010