The 2nd Annual CAST Tester Competition

I had the pleasure of competing in CAST‘s testing competition as captain of Team “Hey, David”, and I’m proud (and a bit stunned) to say that we won 1st or 2nd place in all four categories!

Needless to say, it was a 100% team effort, and I was lucky to work with my teammates Dee Pizzica and Grace Hensley. James Bach talks about certifying testers he as worked with closely, and (while the Jeff Fry certification may not be as broadly recognized) I have to say I am proud to certify that Dee and Grace are stellar folks to pair with for a 6 hour testing frenzy!

I should say as well that Team Canada would have knocked us to 2nd place for the top prize, if they hadn’t been disqualified for having Paul Holland, a AST facilitator on their team…a rule which wasn’t entirely clear to them. I’ve seen bits of Team Canada’s record and heard more. They clearly did a phenomenal job. Hopefully this loss’ll teach the rest of them not to associate with that shady Paul Holland character. ;0)

All that said, let me explain how things worked. David Gilbert of SiriusSQA generously offered to expose some alpha quality software he’d been working on in his spare time to our testing. This meant we got to test real software…with the real developer/project owner on site. Specifically, we tested ShapeUp, an installed Windows exercise management program. James Bach ran the show, and explained that they didn’t want to have to police for cheating, and so they were going to allow as much as possible: One could ask for help from anyone they wanted to, talk to anyone they wanted to, and could listen in on each other if they chose to…but the first team who logged a bug was going to be the only one who got credit for it for the Best Bug List category.

The four categories, and the associated prizes were:

BEST TEST REPORT AWARD: $1500
<Team Canada, (Paul Holland, Captain)>
Team Hey David, (Jeff Fry, Captain)
Team Louise, (Louise Perod, Captain)

BEST BUG LIST AWARD: New Laptop
Team Blue, (Martin Taylor, Captain)
Team Hey David, (Jeff Fry, Captain)

THANK YOU TESTER AWARD FOR BEST BUG: Video Ipod
Bug 110, Team Hey David, (Jeff Fry, Captain)
Bug 117, Team Crazy Canuck, (Jason Coutu, Captain)

DEVELOPERS CHOICE: The cash in James’ wallet ($120)
Team Hey David, (Jeff Fry, Captain)
Team Louise (Louise Perod, Captain)

The first two categories are described a bit more on the Association for Software Testing site.

One could test solo or in teams, and I believe folks chose to work on teams anywhere from one to five testers. Prizes were to be split amongst the team, regardless of the size of the team.

I teamed up with Grace Hensley and Dee Pizzica, and while we rotated roles, at most times two of us were pair-testing on my Mac using Parallels, and a third was entering bugs in the bug tracker on Grace’s Mac (which lacks Windows and so couldn’t run ShapeUp). If I had to point to one single decision we made that led to our victory, it was deciding to sit an arm’s length from the developer. The second we got our install CDs, most testers ran for a corner or another room, presumably to keep their finds hidden. We chose an opposite strategy – sitting an arm’s length away from the developer.

Now, the job of a tester is to provide important quality-related information about the state of the product, yes? And the importance of the information is defined by the developers and project owners. I appreciate Gregory Bateson’s definition of information as “a difference that makes a difference”, and it always helps me to know what makes a difference to the people I’m testing for. From a different angle, lets take James Bach’s definition of testing as “Questioning a product in order to evaluate it.” Many of those questions I explore through experimentation, but (especially when I am first exploring an application) others are best answered by talking to a developer, customer or stakeholder. Here where the developer and product owner were the same person, we very quickly started asking David if and how he preferred to be approached with questions. As it happened, he wasn’t coding during the contest, he was mostly evaluating bug reports and answering questions, so he invited us (and anyone else that thought to ask) to go ahead and pepper him.
A final benefit of sitting by the developer, one I’ve enjoyed every single time I’ve had the pleasure of working side by side with the folks whose work I was testing: Developers have all kinds of important conversations that they don’t pass along to the test team. This isn’t necessarily about being non-communicative; thoughout the day developers will have conversations with each other or others on the project team, and those conversations are often peppered with helpful nuggets for the test team. In this case, when other testers came to ask David questions, we often kept an ear open, and got useful clues as a result.

There was of course much more to our 6 hour testing spree than colocation. I found that pairing for the testing was fantastic. It kept our spirits up as the night wore on. It helped us focus our testing, and enabled us to catch more problems and to brainstorm more experiments than any one of us testing alone. For next years contenders, I strongly recommend:

• Colocation – I think that it’ll be quite crowded around the developer next time around.
• Testing as a team – Grace, Dee and I didn’t know each other at all before the start of the weekend. Testing as a team helped me to test better, learn more about testing, build community, and have a much better time.
• Test reports – I learned a lot about this from Team Canada, who turned in an amazing pile of test artifacts including video captures of bug reports, video demonstrations of testing techniques (both ones that found bugs and those that didn’t), and a voluminous description of all their testing.

Most of all I recommend joining the fun next year! I believe we had around fifteen teams competing this year, and from the conversations I’ve had it seems folks in general had a great time.