I was just talking with coworkers about the pros and cons of test plans with expected results. I tend to organize my testing with a series of questions to investigate, and or a rough map (often in .xls format) of areas to explore. Here we have some very well thought out test cases, including clear steps to follow and clear expected results.
Cem Kaner refers to having a test script with expected results as inattentional blindness by design. I agree wholeheartedly. In discussing this, an analogy occured to me to a 2004 Wired article about Hans Monderman, a Dutch traffic planner who has succeeded in making roads safer by making what to do more ambiguous. Here they discuss his solution to a frantic intersection that was the site of many accidents:
It’s the confluence of two busy two-lane roads that handle 20,000 cars a day, plus thousands of bicyclists and pedestrians. Several years ago, Monderman ripped out all the traditional instruments used by traffic engineers to influence driver behavior – traffic lights, road markings, and some pedestrian crossings – and in their place created a roundabout, or traffic circle. The circle is remarkable for what it doesn’t contain: signs or signals telling drivers how fast to go, who has the right-of-way, or how to behave. There are no lane markers or curbs separating street and sidewalk, so it’s unclear exactly where the car zone ends and the pedestrian zone begins. To an approaching driver, the intersection is utterly ambiguous – and that’s the point.
The general goal is to keep the driver’s brain engaged. If they enter the intersection thinking “I know just what needs to be done here”, they risk running on autopilot. If they enter and think “Whoa. This is a bit confusing, I better pay attention!” there are a lot less accidents.
I think this has clear parallels to testing. If we go to a section of the application and think “Ahh. I know just what needs to be checked here. I’ve followed this script before.” we are much more likely to miss the elephant in the middle of the room.
To quote Monderman,
“The trouble with traffic engineers is that when there’s a problem with a road, they always try to add something,” Monderman says. “To my mind, it’s much better to remove things.”
In many cases, I couldn’t agree more.
[Edit, 2007, June 1]
I originally titled this “Why You Should Make Your Test Plans Less Clear”, which at the moment seemed like an interesting teaser/title. I just changed it to “Adding A Little Ambiguity To Your Plan”. Why? “…You Should…” was meant tongue-in-cheek, but I read it again today and it irked me. Who is this yahoo telling me what I should do? Oh yeah, it’s me. And so I changed the title.