A Creative Heuristic for Automated Detection of Certain Layout Bugs

I was just looking over the talks from the 2009 Google Test Automation Conference (GTAC) in Zurich

In test automation, the most difficult challenge is often finding a useful Test Oracle — a heuristic mechanism by which we can determine if our test has found a bug. This is perhaps particularly true when we’re trying to use automated checks to find layout problems. Given that, I was curious what Michael Tamm’s “Fighting Layout Bugs” would have to say…and have to say I’m impressed.

Tamm started out with running the HTML and CSS through the W3C validator, which is cool as far as it goes, but it got really interesting when he declared he has a heuristic oracle to determine if text has overrun a boundary (e.g. a name in a summary box which overflows the edge of that box). How does he do it?

Get the set of pixels that are text:

• inject jQuery
• jquery(‘*’).css(‘color’. ‘white’);  #set all text to white
• take screenshot
• jquery(‘*’).css(‘color’. ‘black’);  #set all text to black
• take second screenshot
• compare. the diff should be the set of all text pixels

Next get the borders:

• jquery(‘*’).css(‘color’. ‘transparent’); #make all text transparent
• Take screenshot
• Find vertical pixel sequences of a certain min length, where each pixel has the same color (or a very similar one)
• Only select those which have a high contrast to the left or right. This is the set of vertical borders.
• Do the same for horizontal borders.

At this point, we have two sets of pixels; any pixel which is in both sets is likely a layout bug.

Now, obviously all of these screenshots of browsers makes for slowly running tests, and I’m curious how many false negatives there would be…but the idea is intruiging, and I quite admire the creative test oracle. I’m not sure where this would be an effective use of resources in my current work on Freebase.com, but I wonder if there are areas where it’s worth the bang for the buck.

For the curious, here’s his code. I recommend the slides, which stand on their own quite well. Here’s the talk on YouTube, but be warned that it suffers from poor audio. If you don’t want to watch the entire talk I agree with the YouTube commenter dericbytes that the highlight is ~ minute 21 to 26.

Congrats to Michael Tamm for thinking outside the box and creating a novel test oracle for detecting certain classes of layout bugs!