I was just looking over the talks from the 2009 Google Test Automation Conference (GTAC) in Zurich
In test automation, the most difficult challenge is often finding a useful Test Oracle — a heuristic mechanism by which we can determine if our test has found a bug. This is perhaps particularly true when we’re trying to use automated checks to find layout problems. Given that, I was curious what Michael Tamm’s “Fighting Layout Bugs” would have to say…and have to say I’m impressed.
Tamm started out with running the HTML and CSS through the W3C validator, which is cool as far as it goes, but it got really interesting when he declared he has a heuristic oracle to determine if text has overrun a boundary (e.g. a name in a summary box which overflows the edge of that box). How does he do it?
Get the set of pixels that are text:
- inject jQuery
- jquery(‘*’).css(‘color’. ‘white’); #set all text to white
- take screenshot
- jquery(‘*’).css(‘color’. ‘black’); #set all text to black
- take second screenshot
- compare. the diff should be the set of all text pixels
Next get the borders:
- jquery(‘*’).css(‘color’. ‘transparent’); #make all text transparent
- Take screenshot
- Find vertical pixel sequences of a certain min length, where each pixel has the same color (or a very similar one)
- Only select those which have a high contrast to the left or right. This is the set of vertical borders.
- Do the same for horizontal borders.
At this point, we have two sets of pixels; any pixel which is in both sets is likely a layout bug.
Now, obviously all of these screenshots of browsers makes for slowly running tests, and I’m curious how many false negatives there would be…but the idea is intruiging, and I quite admire the creative test oracle. I’m not sure where this would be an effective use of resources in my current work on Freebase.com, but I wonder if there are areas where it’s worth the bang for the buck.
For the curious, here’s his code. I recommend the slides, which stand on their own quite well. Here’s the talk on YouTube, but be warned that it suffers from poor audio. If you don’t want to watch the entire talk I agree with the YouTube commenter dericbytes that the highlight is ~ minute 21 to 26.
Congrats to Michael Tamm for thinking outside the box and creating a novel test oracle for detecting certain classes of layout bugs!
Jeff Fry on Testing 
February 24, 2010 at 4:00 am |
At CAST in Seattle, Harry Robinson showed a similar technique for testing Google Maps. They would do the driving directions from point A to point B and capture a screenshot. And the same for B to A. And then they would overlay them. If there was greater than x% variance in the routes then it was likely a bug.
March 10, 2010 at 1:36 am |
Hi Jeff,
I have been reading all your blog posts. You blog posts are excellent. They provide good and extremely valuable information.
Keep it up.
Ashfaque Ahmed
Author: Software Testing as a Service
June 8, 2010 at 11:20 am |
Two things spring to my mind immediately.
1) This will only work for entry or display fields with colored outlines
2) Why do we care about the horizontal for this test?
June 8, 2010 at 1:35 pm |
Cowboytesting — checking out the slides or the video might help, but let me see if I can answer both of your questions without it.
As far as the borders are concerned, reread the algorithm above. Now lets take an example page from the site I test: http://www.freebase.com/edit/topic/en/the_lord_of_the_rings_the_fellowship_of_the_ring
From your comment, it sounds like you thought this would find only outlined borders, but the algorithm as described should find anyplace where color changes suddenly along a vertical or horizontal line of sufficient length. For example, it should find the light blue v. white “zebra striping” of the various performances (actor : character) listed.
The reason we want to capture horizontal borders is because we want to find places where text overflows a border above or below it. This can happen just because the text was misaligned, or because it was longer than planned for, and so is squeezing out the bottom.
Note, this page would be interesting to play with, because it uses gradient coloring several places. Consider the text “Top Contributors to this topic” (toward the top right of the page). The algorithm described in the talk should find the place above the word “Top” where the color suddenly changes from dark grey to light grey. Depending on how well it handles gradients though , it might or might not catch the borders to the left and the right. Looking down from there, find the text “See Topic History”. I think this algorithm would likely have a difficult time discovering if that text overflowed to the right.
Any oracle will have limitations. The fact that this one misses certain classes of bugs isn’t necessarily a problem. The important thing is for the tester to understand what the oracle is(n’t) good at discovering, and use it appropriately.
June 8, 2010 at 1:38 pm |
Ironically, the link I used in the above example demonstrates a bug that this oracle should be able to catch (depending on how much contrast the tester sets it for, in deciding when something is a border).
Note also that the same link here:
http://www.freebase.com/edit/topic/en/the_lord_of_the_rings_the_fellowship_of_the_ring still overflows a border to *my* eye…but it wouldn’t be caught by this oracle.