Here’s a 22 minute video lecture on using controlled experiments to discover what your customers think, and if you work on web software I suspect you’ll find it 22 minutes well spent. Kohavi points out several examples where the results of tests were quite surprising, as well as some interesting suggestions for how to organize them.
Note, this is related testing software in the broad sense. The bugs that split tests find are the (plentiful) bugs in our mental models, not in our implementation.
If you haven’t been to Stack Overflow yet, it’s an interesting forum for asking technical questions — and sorting through the answers — written by Joel Spolsky and Jeff Atwood.
I noticed a question on Model-Based Testing over there that I had something to say about. I wanted to link to articles by Harry Robinson, Ben Simo and James Bach…but as a new user, I’m allowed to add only one link. What to do? How about using my one link to go to my blog. Here’s the original question.
And here’s my answer, complete with links:
First, a quick note on terms. I tend to use James Bach’s definition of Testing as “Questioning a product in order to evaluate it”. All test rely on /mental/ models of the application under test. The term Model-Based Testing though is typically used to describe programming a model which can be explored via automation. For example, one might specify a number of states that an application can be in, various paths between those states, and certain assertions about what should occur in on the transition between those states. Then one can have scripts execute semi-random permutations of transitions within the state model, logging potentially interesting results.
There are real costs here: building a useful model, creating algorithms for exploring it, logging systems that allow one to weed through for interesting failures, etc. Whether or not the costs are reasonable has a lot to do with *what are the questions you want to answer?* In general, start with “What do I want to know? And how can I best learn about it?” rather than looking for a use for an interesting technique.
All that said, some excellent testers have gotten a lot of mileage out of automated model-based tests. Sometimes we have important questions about the application under test are best explored by automated, high-volume semi-randomized tests. Here’s one very colorful example from Harry Robinson (one of the leading theorists and proponents of model-based testing) where he discovered many interesting bugs in Google driving directions using a model-based test (written with ruby’s Watir library).
Robinson has used MBT successfully at companies including Bell Labs, Microsoft, and Google, and has a number of helpful essays.
Ben Simo (another great testing thinker and writer) has also written quite a bit worth reading on model-based testing.
Finally, a few cautions: To make good use of a strategy, one needs to explore both its strengths and its weaknesses. Toward that end, James Bach has an excellent talk on the limits and challenges of Model-Based Testing. This blog post of Bach’s links to his hour long talk (and associated slides).
I’ll end with a note about what Boris Beizer calls the Pesticide Paradox: “Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffective.” Scripted tests (whether executed by a computer or a person) are particularly vulnerable to the pesticide paradox, tending to find less and less useful information each time the same script is executed. Folks sometimes turn to model-based testing thinking that it gets around the pesticide problem. In some contexts model-based testing may well find a much larger set of bugs than a given set of scripted tests…but one should remember that it is still fundamentally limited by the Pesticide Paradox. Remembering its limits — and starting with questions MBT addresses well — it has the potential to be a very powerful testing strategy.
On the watir-general email list, George Sanders recently wrote
“It seems that I’ve been encountering more people within my workplace (and, alas, even within my own QA team!) that are not sold on test automation. From what I’ve learned so far, there seems that automation will never cover 100% of what needs to be tested, but this doesn’t negate the need.
Another frustration is that I’ve been tasked to write automation scripts as part of my year-end goals. However, I haven’t been assigned hours in my work week to do them! All of my script development has been after-hours and weekends (notice I’m posting this on a Saturday!).
Has anyone else run into naysayers? How can I convince the decision-makers that this is a worthwhile effort?”
I responded on-list, but want to improve my answer (and change my analogy). This conversation tends to be quite unproductive when it becomes “Is or isn’t test automation useful?” It’s a lot like “Are ladders useful?” Ladders won’t solve any problems on their own. There are plenty of problems that ladders will not help with a bit. Problems that are helped by step-ladders may only be made worse by a 20 foot ladder. Yet there are some problems where the right ladder well used will add tremendous value.
Similarly with testing, I like to look at what are the problems that need to be solved. Then I like to think about each of them, and consider solutions. Is one of your problems basic functionality
breaking when code is changed? Unit tests might be a great solution to your problem. Do you suspect intermittent production failures are the result of concurrency issues? Likewise, automation-assisted tests
might really help in reproducing the problem in house.
I think discussing the relative costs/benefits of automated browser-based regression testing is a good idea, and getting real experience reports helps a lot. Within this realm in particular — depending on many contextual factors — there may be some problems that’re helped through automated browser-based tests and others where the cost is too high.
Note, for some concrete examples of problems where Watir has been useful for me, see my post about a few of the most compelling cases where I’ve used Watir over the past five years.
Getting time added into the schedule for test automation is a different question, but one that might become easier if you’re able to focus the conversation around solutions to particular problems. (On the other hand, there are a lot of reasons — sensible and not — why test automation might not get onto the schedule.) In any event, focussing the conversation on specific problems and potential solutions for them is likely to increase the odds of a productive conversation.
A sign of the times, I know a handful of great testers and coders who’ve been laid off in recent months. One I just heard about today is Chris McMahon. I first encountered Chris as a contributor on the Watir, Software-testing, and Agile-testing mailing lists. At the time, I was QA Manager for a company that was having a rough time filling a position for a very technical API tester and test automation expert. I emailed Chris, one thing led to another, and I had the good fortune to work with Chris for a period.
Several jobs later for both of us, he fell prey to economic downsizing today. I have to say I was very impressed (though not really surprised) to see what a stir it has made in the software testing communities I’m a part of. The impact is clearest in all the conversation about and testimonals for Chris on Twitter.
Is this because Chris is a very skilled tester? Absolutely it is…but there are other very skilled testers out there who just aren’t as known. He blogs, he has had articles published in several journals, and he actively contributes to multiple online testing communities. And by contributes I mean he engages in dialog, he offers ideas, he offers help to folks who ask good questions. On the Watir list, he claimed an unofficial spot some time back as the Answerer Of Off Topic Questions. When someone raises something that’s more of a ruby / test design / other library related question, he has frequently had something helpful to contibute, and he’s done so, even as he worked for the last year or so at a company that uses Selenium instead.
Being a part of a testing community has many benefits: Exposure to new ideas, meeting colleagues, a chance to have our ideas tested and improved via feedback, etc. — but this is a place where it’s particularly clear. It may turn out to be a tough time to be looking for a remote testing position, but the way Chris has chosen to live his professional life over many years seems to be reaping major dividends right about now.
I’ve just started my second round of teaching Black Box Software Testing Foundations for AST members. Besides the obvious benefits of what I consider to be a very well designed and rigorous course, there are the side benefits of meeting interesting folks from all over the world, and the discussions that arise. One student who’s currently between jobs, and has extensive experience testing embedded software but is looking for a job testing web applications asked me for advice.
Looking back on my answer to her, it seems like something others might appreciate as well, so I decided to publish it here as well.
Specific ideas for transitioning from testing one type of software to another:
And then there’s the general advice:
David Christiansen, the editor of the AST Update, has put out a nice new issue…including an essay of mine on working with Inattentional Blindness. It’s available in print and as a PDF, and there’s a lot of good in it. Check it out!
I’ve been home from the 3rd annual Conference of the Association of Software Testing in Toronto for three weeks now, and am still thinking about all I’ve learned.
This was my 3rd time attending CAST. As always, the keynotes, tutorials and track sessions were excellent…and as always, even better than that was the conferring. You see, CAST knows that it attracts testers with an impressive array of experience, and that making time for them to riff off of each other is very valuable. Toward that end, every single track session and keynote included substantial time devoted to discussion, with an explicit welcome to challenge the presenter or to otherwise test their assertions. Inevitably, the I found the lively conversation spilling over into dinner, or following us to the pub.
While I’ve been testing for eight years now, and have worked on a range of web and client-server applications, all of my work so far is limited to the San Francisco Bay Area. It’s a tremendous pleasure discussing testing with folks from vastly different industries, and from around the world.
One story of many: Over dinner I mentioned a problem that I thought was best solved by starting a conversation with the programmer. Scott Barber responded with something like “Providing the tester doesn’t get in trouble for talking to her.” I was surprised, and asked if that’s really something that happens much in 2008…and was informed by folks at the table from more regulated industries or on government projects that that’s not uncommon. Partly this made me very happy to be a tester for in very human culture we have at Freebase, but it also served to reinforce how easy it is to overgeneralize about the field of testing, when I’m really thinking about testing in a particular context or contexts.
Discussing our particular challenges, sharing stories, and questioning testers with very different perspectives really made CAST 2008 a joy for me…and I’m looking forward to 2009 in Colorado Springs!
This is only loosely related to software testing…being some of the things that keep me healthy while I do it. This started as an email template, which I’ve sent to many friends and colleagues at their request – enough that I decided it was worth posting here:
Keyboard
The Goldtouch Adjustable Keyboard is what I use at work, and like it a lot. I found it to have next to no learning curve. I know others who prefer the Kinesis Contoured Keyboard…which friends say takes up to a week to learn to use, but I know a couple folks who my keyboard did nothing for, and who love these.
My biggest advice for folks though is to try several options out, and (for folks in the SF Bay Area) Office Relief is a great place in San Leandro to do so.
Mouse
Again, very personal, and trying out options is definitely the way to go. What I’ve settled on is a fairly traditional mouse on my left, and an Evoluent VerticalMouse 3 on my right. Some days I use mostly one or the other, most days I switch between them. Note, I was excited to try their left handed version, but it is still at v2 and I really didn’t like the way it fit in my hand.
Taking Breaks
The last thing I’ll say is that the single most helpful ergonomic adjustment I’ve made is software that helps me to pace my breaks. There are good free options for PC (Workrave) or Mac (Anti-RSI). Both are very configurable.
Scott Barber just wrote an amusing and insightful reflection on what sorts of titles testing professionals have, entitled Software Testers: Identity Crisis Or Illusions of Grandeur?.
I personally am inclined toward simple, descriptive titles, and think that ‘Software Tester’ (including variants like ‘Senior’, ‘Lead’, etc.) describes what I do pretty well. Several times at previous companies I’ve discussed changing the titles we use from Software Quality Assurance Engineer to Software Tester. I’ve gotten several interesting reactions, including:
Both of these objections seem to be compelling reasons to raise the conversation. Do folks around you not understand what software testing is, and why it’s a challenging, high-skill activity? Well, that’s a good conversation to get to have.
More concerning to me is when folks think that my team should “assure quality”. At this point I really want to make clear my perspective: Everyone in software development should consider quality to be there job. I am here to discover and communicate important quality-related information about the product and project – but I cannot and shouldn’t try to “assure quality”.
Rationales that I’m more sympathetic to for having high-fallutin’ titles include:
Scott ends with
Since there seems to be a prevalent desire for software testers to have fancy sounding titles, maybe we should consider “Software Quality Forecaster” instead. At least that would help our teammates better understand what we really do.
Maybe. I like Forecast way better than I like Assure. Perhaps “Software Quality Investigator”? For someone who doesn’t understand software testing, each title communicates what the job means imperfectly. To capture the flavor and variety of what testing can be, I expect that no title will be a substitute for a good conversation.
Certainly, I would love to see Software Tester considered an esteemed title, and to see the practice of mislabeling testing as ‘quality assurance’ fade away. For now I’m flexible as to what I’m called, but believe it’s important to consider what inaccuracies any given title communicates – and what biases it represents – and then to use the title as a springboard for conversation.
Bret Pettichord, lead Watir developer just announced on the watir-general discussion list that he’s taken a new job “working full-time with Prototest and Pete Dignan (CEO of Prototest) to accelerate the development of new versions of Watir.”
One of his top priorities is getting FireWatir and Watir in sync with each other – something I’m very excited about.
He just asked for stories regarding “Why do you use Watir?“
I haven’t blogged for a while, and I’ve meant to tell this story here for some time, so here goes:
I had just started a new job as the first dedicated tester at a company with an established product. In my first few months, the testing that made the most sense to do was hands-on, sapient testing…but I’d heard some buzz about a new open source tool under development, nearing a 1.0 release, and I was itching to try it.
One afternoon a report came in from the field: Someone had been attempting to import members to a project, and had seen “Permission Denied”…with someone else’s name up in the right hand corner! Had their members been imported into some stranger’s project? They tried again and succeeded, but were understandably spooked.
I asked around, “Have we ever heard reports like this before?”
“You know, now that you mention it, once or twice a year someone makes a similar complaint. Each time, a coder has poked about in the code, changed something, and said ‘I bet that’ll fix it.’” The problem was no one had ever succeeded at reproducing the problem, and so no one had a way to test if it was fixed.
I had a hunch that there was some concurrency / race condition going on, and that Watir might help me diagnose it. Watir turned out to be simple enough that in less than I day, and without any previous ruby experience, I wrote a script that performed these actions over and over, concurrently, as 5 different users. Lo and behold, I could consistently reproduce this issue in under 30 minutes.
The coders I worked with were thrilled to have a consistent repro case, and after a few false leads, they had a fix.
Since then, I’ve used Watir for tasks including concurrency testing (above), creating large data sets, and measuring end-to-end page performance across revisions, not to mention regression testing. Inspired by some of Harry Robinson’s work doing high-volume, semi-random testing of Google directions using Watir, my latest project is venturing into using Watir for some automated model-based testing. Along the way I’ve learned a real language (not a vendorscript) and am part of a friendly and thriving community of users and contributors…to whom I’m very grateful.
