A Creative Heuristic for Automated Detection of Certain Layout Bugs

I was just looking over the talks from the 2009 Google Test Automation Conference (GTAC) in Zurich

In test automation, the most difficult challenge is often finding a useful Test Oracle — a heuristic mechanism by which we can determine if our test has found a bug. This is perhaps particularly true when we’re trying to use automated checks to find layout problems. Given that, I was curious what Michael Tamm’s “Fighting Layout Bugs” would have to say…and have to say I’m impressed.

Tamm started out with running the HTML and CSS through the W3C validator, which is cool as far as it goes, but it got really interesting when he declared he has a heuristic oracle to determine if text has overrun a boundary (e.g. a name in a summary box which overflows the edge of that box). How does he do it?

Get the set of pixels that are text:

  • inject jQuery
  • jquery(‘*’).css(‘color’. ‘white’);  #set all text to white
  • take screenshot
  • jquery(‘*’).css(‘color’. ‘black’);  #set all text to black
  • take second screenshot
  • compare. the diff should be the set of all text pixels

Next get the borders:

  • jquery(‘*’).css(‘color’. ‘transparent’); #make all text transparent
  • Take screenshot
  • Find vertical pixel sequences of a certain min length, where each pixel has the same color (or a very similar one)
  • Only select those which have a high contrast to the left or right. This is the set of vertical borders.
  • Do the same for horizontal borders.

At this point, we have two sets of pixels; any pixel which is in both sets is likely a layout bug.

Now, obviously all of these screenshots of browsers makes for slowly running tests, and I’m curious how many false negatives there would be…but the idea is intruiging, and I quite admire the creative test oracle. I’m not sure where this would be an effective use of resources in my current work on Freebase.com, but I wonder if there are areas where it’s worth the bang for the buck.

For the curious, here’s his code. I recommend the slides, which stand on their own quite well. Here’s the talk on YouTube, but be warned that it suffers from poor audio. If you don’t want to watch the entire talk I agree with the YouTube commenter dericbytes that the highlight is ~ minute 21 to 26.

Congrats to Michael Tamm for thinking outside the box and creating a novel test oracle for detecting certain classes of layout bugs!


5 Responses to “A Creative Heuristic for Automated Detection of Certain Layout Bugs”

  1. Adam Goucher Says:

    At CAST in Seattle, Harry Robinson showed a similar technique for testing Google Maps. They would do the driving directions from point A to point B and capture a screenshot. And the same for B to A. And then they would overlay them. If there was greater than x% variance in the routes then it was likely a bug.

  2. ahmedashfaque Says:

    Hi Jeff,
    I have been reading all your blog posts. You blog posts are excellent. They provide good and extremely valuable information.

    Keep it up.

    Ashfaque Ahmed
    Author: Software Testing as a Service

  3. cowboytesting Says:

    Two things spring to my mind immediately.

    1) This will only work for entry or display fields with colored outlines
    2) Why do we care about the horizontal for this test?

  4. testingjeff Says:

    Cowboytesting — checking out the slides or the video might help, but let me see if I can answer both of your questions without it.

    As far as the borders are concerned, reread the algorithm above. Now lets take an example page from the site I test: http://www.freebase.com/edit/topic/en/the_lord_of_the_rings_the_fellowship_of_the_ring

    From your comment, it sounds like you thought this would find only outlined borders, but the algorithm as described should find anyplace where color changes suddenly along a vertical or horizontal line of sufficient length. For example, it should find the light blue v. white “zebra striping” of the various performances (actor : character) listed.

    The reason we want to capture horizontal borders is because we want to find places where text overflows a border above or below it. This can happen just because the text was misaligned, or because it was longer than planned for, and so is squeezing out the bottom.

    Note, this page would be interesting to play with, because it uses gradient coloring several places. Consider the text “Top Contributors to this topic” (toward the top right of the page). The algorithm described in the talk should find the place above the word “Top” where the color suddenly changes from dark grey to light grey. Depending on how well it handles gradients though , it might or might not catch the borders to the left and the right. Looking down from there, find the text “See Topic History”. I think this algorithm would likely have a difficult time discovering if that text overflowed to the right.

    Any oracle will have limitations. The fact that this one misses certain classes of bugs isn’t necessarily a problem. The important thing is for the tester to understand what the oracle is(n’t) good at discovering, and use it appropriately.

  5. testingjeff Says:

    Ironically, the link I used in the above example demonstrates a bug that this oracle should be able to catch (depending on how much contrast the tester sets it for, in deciding when something is a border).

    Note also that the same link here:
    http://www.freebase.com/edit/topic/en/the_lord_of_the_rings_the_fellowship_of_the_ring still overflows a border to *my* eye…but it wouldn’t be caught by this oracle.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: