More and more web applications are taking advantage of asynchronous JavaScript (or AJAX) – and with good reason. It gives tremendous power to a web application, allowing the app to respond fluidly to the user without requiring a full page reload and making it feel more like a native desktop app. At the same time, it makes the application trickier to automate.
When I first started writing browser-based test automation, it was safe to assume that once the page loaded, if an element wasn’t there yet, it wasn’t going to be. With AJAX, that assumption goes out the window. Let’s take an example:
Someone’s registering for our website. The site prompts her for a username, which she enters. Now, in the olden days, she would fill out the rest of the page and then submit. The page would refresh…and often tell her that the username she selected is unavailable. Rinse and repeat until you finally find an available name.
With AJAX, the moment she clicks away from the username field, we can have feedback appear on the page, letting her know whether or not that name has been taken. She doesn’t need to wait for a page refresh before each attempt, the feedback may be nearly instantaneous.
Sounds good, right? Now I come along to automate a test of this page. If I write something like (in pseudocode):
set username = reserved_name
assert response_text says reserved_name is taken.
…much of the time, I’ll get an error saying that the response_text isn’t there on the screen.
How can I tell my script not to progress to the next step until the application is ready? I have a range of options here. At the cringe-inspiring end of the spectrum, I can write:
set username = reserved_name
sleep 10 seconds
assert response_text says reserved_name is taken.
The problem here is that one either sets the sleep too short, and the script errors out when the test server is under a bit of extra load — or one sets it too long, and the scripts begin to spend 90% of their time sleeping.
Somewhat better is to do like so:
set username = reserved_name
wait( up to 20 seconds) for response_text #then
assert text in page says reserved_name is taken.
The issue here is that you need to know exactly what you’re waiting for, may need to list multiple triggers to wait for, and your script risks of aging poorly as these triggers change.
To get better than this likely requires testability features in the application under test. One thing that’s worked well for me is having the application set a flag whenever it starts to execute AJAX, and unset that flag when the AJAX completes. At Freebase.com, we chose to have that flag be an attribute on the body tag, like so:
Whenever AJAX is kicked off, this is set to ajaxStart and whenever the last asynchronous process completes, it sets it to
Now I can say something like:
set username = reserved_name
wait_until_loaded
assert text in page says reserved_name is taken.
Where wait_until_loaded is defined like:
Wait( up to 20 seconds) while body.attribute(ajax) == ‘ajaxStart’
This pushes the burden of knowing when the page is ready for the next step back to the application under test. The downsides of this approach are: (1) it requires code in the application under text to make it work. Sometimes that’s difficult (politically or otherwise). (2) this solution’s only as robust as the code that sets to ajax flag. If the application under test can’t or won’t report this reliably, this solution will only cause you grief.
That said, the upside of this approach is potentially huge: It frees the tester from writing a custom wait trigger after every AJAX step, speeding up test script writing and often making the scripts more robust as well.
Depending on how familiar you are with the testing library you’re using, it may be worth considering putting this check deeper in your code. For example, one might change the click method to always wait_until_loaded before clicking. This has the potential to streamline test scripts that much more.
In fact, the Windmill test automation library has already done this on trunk. As of the next release of Windmill, the default behavior for everything other than asserts is to wait for that element for a set number of seconds — progressing immediately if it’s there right away, but only throwing an error if the timeout is reached. I suspect as web applications become more AJAX-heavy, this will become the standard in test automation libraries.
There isn’t a One Right Answer for any context, but I’ve been very happy with having the application under test set an ajax flag of some sort. If you’re testing an application where the page changes asynchronously, I encourage you to give it a try.
Jeff Fry on Testing 