Why Would You Call Yourself THAT?

Scott Barber just wrote an amusing and insightful reflection on what sorts of titles testing professionals have, entitled Software Testers: Identity Crisis Or Illusions of Grandeur?.

I personally am inclined toward simple, descriptive titles, and think that ‘Software Tester’ (including variants like ‘Senior’, ‘Lead’, etc.) describes what I do pretty well. Several times at previous companies I’ve discussed changing the titles we use from Software Quality Assurance Engineer to Software Tester. I’ve gotten several interesting reactions, including:

• “SQAE is a more impressive title. Why would you want a title that makes you sound low-skill?”, and
• “But we want you to assure the quality here. Don’t try to back out of your responsibilities!”

Both of these objections seem to be compelling reasons to raise the conversation. Do folks around you not understand what software testing is, and why it’s a challenging, high-skill activity? Well, that’s a good conversation to get to have.

More concerning to me is when folks think that my team should “assure quality”. At this point I really want to make clear my perspective: Everyone in software development should consider quality to be there job. I am here to discover and communicate important quality-related information about the product and project – but I cannot and shouldn’t try to “assure quality”.

Rationales that I’m more sympathetic to for having high-fallutin’ titles include:

• “SQAE is a form of shorthand to communicate to budget setters that testing is a high-skill activity”,
• “It’s the standard for companies like ours. Job applicants will know what we mean”, and the related
• “Some applicants might think that Software Tester is a lower-pay position, and so listing the job as such might turn off some.”

Scott ends with

Since there seems to be a prevalent desire for software testers to have fancy sounding titles, maybe we should consider “Software Quality Forecaster” instead. At least that would help our teammates better understand what we really do.

Maybe. I like Forecast way better than I like Assure. Perhaps “Software Quality Investigator”? For someone who doesn’t understand software testing, each title communicates what the job means imperfectly. To capture the flavor and variety of what testing can be, I expect that no title will be a substitute for a good conversation.

Certainly, I would love to see Software Tester considered an esteemed title, and to see the practice of mislabeling testing as ‘quality assurance’ fade away. For now I’m flexible as to what I’m called, but believe it’s important to consider what inaccuracies any given title communicates – and what biases it represents – and then to use the title as a springboard for conversation.

Why Do You Use Watir?

Bret Pettichord, lead Watir developer just announced on the watir-general discussion list that he’s taken a new job “working full-time with Prototest and Pete Dignan (CEO of Prototest) to accelerate the development of new versions of Watir.”

One of his top priorities is getting FireWatir and Watir in sync with each other – something I’m very excited about.

He just asked for stories regarding “Why do you use Watir?“

I haven’t blogged for a while, and I’ve meant to tell this story here for some time, so here goes:

I had just started a new job as the first dedicated tester at a company with an established product. In my first few months, the testing that made the most sense to do was hands-on, sapient testing…but I’d heard some buzz about a new open source tool under development, nearing a 1.0 release, and I was itching to try it.

One afternoon a report came in from the field: Someone had been attempting to import members to a project, and had seen “Permission Denied”…with someone else’s name up in the right hand corner! Had their members been imported into some stranger’s project? They tried again and succeeded, but were understandably spooked.

I asked around, “Have we ever heard reports like this before?”

“You know, now that you mention it, once or twice a year someone makes a similar complaint. Each time, a coder has poked about in the code, changed something, and said ‘I bet that’ll fix it.’” The problem was no one had ever succeeded at reproducing the problem, and so no one had a way to test if it was fixed.

I had a hunch that there was some concurrency / race condition going on, and that Watir might help me diagnose it. Watir turned out to be simple enough that in less than I day, and without any previous ruby experience, I wrote a script that performed these actions over and over, concurrently, as 5 different users. Lo and behold, I could consistently reproduce this issue in under 30 minutes.

The coders I worked with were thrilled to have a consistent repro case, and after a few false leads, they had a fix.

Since then, I’ve used Watir for tasks including concurrency testing (above), creating large data sets, and measuring end-to-end page performance across revisions, not to mention regression testing. Inspired by some of Harry Robinson’s work doing high-volume, semi-random testing of Google directions using Watir, my latest project is venturing into using Watir for some automated model-based testing. Along the way I’ve learned a real language (not a vendorscript) and am part of a friendly and thriving community of users and contributors…to whom I’m very grateful.

AST’s Online Testing Courses, and Power of Two Bugs in Excel 2007

The Association for Software Testing has recently started offering the Black Box Testing Course, designed by Cem Kaner, James Bach, and Becky Fiedler, as a free benefit to AST members. I took the first round (taught by Cem, Scott Barber, and Jon Hagar) and was very impressed. I expected that any course designed by Cem would reflect a deep understanding of testing – and it did – but what surprised me was the deep understanding of teaching. Pedagogically, this course stood head-and-shoulders above the previous online courses I’ve taken from UC Berkeley Extension and Foothill College.

I’ll write more about it soon, but for now I want to highlight one exercise in particular. We were told:

You are testing a program that includes an Integer Square Root function. The function reads a 32-bit word that is stored in memory, interprets the contents as an unsigned integer and then computes the square root of the integer, returning the result as a floating point number.

And then asked a number of thoughtful questions about what test strategy we would take, how many square roots we would(n’t) test, etc. It’s an interesting problem, and many different suggestions were made. One of common thread was the importance of testing around powers of two, especially just over and under boundaries of 8, 16, and 32-bit integers. Now, Doug Hoffman has an excellent article showing that interesting errors can occur far from any of these boundaries…but testing around powers of two was one of our baselines.

That said, I’m fascinated to learn that Excel 2007 has a major calculation error at 2**16-1. Many (or all?) formulas that should evaluate to 65535 instead evaluate as 100000. So for example:

• 850 x 77.1 – 1 = 65534
• 850 x 77.1 = 100000

As a test engineer, I fully understand that not every bug will ever be caught…but it fascinates me that Microsoft’s test team doesn’t seem to have thoroughly tested the boundary between 16-bit and 32-bit integers. This is of course being discussed on Slashdot, Digg, and many other places. I’m curious how Microsoft’ll react, and what the repercussions will be.

The 2nd Annual CAST Tester Competition

I had the pleasure of competing in CAST’s testing competition as captain of Team “Hey, David”, and I’m proud (and a bit stunned) to say that we won 1st or 2nd place in all four categories!

Needless to say, it was a 100% team effort, and I was lucky to work with my teammates Dee Pizzica and Grace Hensley. James Bach talks about certifying testers he as worked with closely, and (while the Jeff Fry certification may not be as broadly recognized) I have to say I am proud to certify that Dee and Grace are stellar folks to pair with for a 6 hour testing frenzy!

I should say as well that Team Canada would have knocked us to 2nd place for the top prize, if they hadn’t been disqualified for having Paul Holland, a AST facilitator on their team…a rule which wasn’t entirely clear to them. I’ve seen bits of Team Canada’s record and heard more. They clearly did a phenomenal job. Hopefully this loss’ll teach the rest of them not to associate with that shady Paul Holland character. ;0)

All that said, let me explain how things worked. David Gilbert of SiriusSQA generously offered to expose some alpha quality software he’d been working on in his spare time to our testing. This meant we got to test real software…with the real developer/project owner on site. Specifically, we tested ShapeUp, an installed Windows exercise management program. James Bach ran the show, and explained that they didn’t want to have to police for cheating, and so they were going to allow as much as possible: One could ask for help from anyone they wanted to, talk to anyone they wanted to, and could listen in on each other if they chose to…but the first team who logged a bug was going to be the only one who got credit for it for the Best Bug List category.

The four categories, and the associated prizes were:

BEST TEST REPORT AWARD: $1500
<Team Canada, (Paul Holland, Captain)>
Team Hey David, (Jeff Fry, Captain)
Team Louise, (Louise Perod, Captain)

BEST BUG LIST AWARD: New Laptop
Team Blue, (Martin Taylor, Captain)
Team Hey David, (Jeff Fry, Captain)

THANK YOU TESTER AWARD FOR BEST BUG: Video Ipod
Bug 110, Team Hey David, (Jeff Fry, Captain)
Bug 117, Team Crazy Canuck, (Jason Coutu, Captain)

DEVELOPERS CHOICE: The cash in James’ wallet ($120)
Team Hey David, (Jeff Fry, Captain)
Team Louise (Louise Perod, Captain)

The first two categories are described a bit more on the Association for Software Testing site.

One could test solo or in teams, and I believe folks chose to work on teams anywhere from one to five testers. Prizes were to be split amongst the team, regardless of the size of the team.

I teamed up with Grace Hensley and Dee Pizzica, and while we rotated roles, at most times two of us were pair-testing on my Mac using Parallels, and a third was entering bugs in the bug tracker on Grace’s Mac (which lacks Windows and so couldn’t run ShapeUp). If I had to point to one single decision we made that led to our victory, it was deciding to sit an arm’s length from the developer. The second we got our install CDs, most testers ran for a corner or another room, presumably to keep their finds hidden. We chose an opposite strategy – sitting an arm’s length away from the developer.

Now, the job of a tester is to provide important quality-related information about the state of the product, yes? And the importance of the information is defined by the developers and project owners. I appreciate Gregory Bateson’s definition of information as “a difference that makes a difference”, and it always helps me to know what makes a difference to the people I’m testing for. From a different angle, lets take James Bach’s definition of testing as “Questioning a product in order to evaluate it.” Many of those questions I explore through experimentation, but (especially when I am first exploring an application) others are best answered by talking to a developer, customer or stakeholder. Here where the developer and product owner were the same person, we very quickly started asking David if and how he preferred to be approached with questions. As it happened, he wasn’t coding during the contest, he was mostly evaluating bug reports and answering questions, so he invited us (and anyone else that thought to ask) to go ahead and pepper him.
A final benefit of sitting by the developer, one I’ve enjoyed every single time I’ve had the pleasure of working side by side with the folks whose work I was testing: Developers have all kinds of important conversations that they don’t pass along to the test team. This isn’t necessarily about being non-communicative; thoughout the day developers will have conversations with each other or others on the project team, and those conversations are often peppered with helpful nuggets for the test team. In this case, when other testers came to ask David questions, we often kept an ear open, and got useful clues as a result.

There was of course much more to our 6 hour testing spree than colocation. I found that pairing for the testing was fantastic. It kept our spirits up as the night wore on. It helped us focus our testing, and enabled us to catch more problems and to brainstorm more experiments than any one of us testing alone. For next years contenders, I strongly recommend:

• Colocation – I think that it’ll be quite crowded around the developer next time around.
• Testing as a team – Grace, Dee and I didn’t know each other at all before the start of the weekend. Testing as a team helped me to test better, learn more about testing, build community, and have a much better time.
• Test reports – I learned a lot about this from Team Canada, who turned in an amazing pile of test artifacts including video captures of bug reports, video demonstrations of testing techniques (both ones that found bugs and those that didn’t), and a voluminous description of all their testing.

Most of all I recommend joining the fun next year! I believe we had around fifteen teams competing this year, and from the conversations I’ve had it seems folks in general had a great time.

Definitions of Testing

Elisabeth Hendrickson has just posted a great blog entry about definitions of testing. She lists several older definitions of testing, and then gives one she and Dale Emery crafted:

Dale Emery and I discussed this at length and decided that we agree that: Testing is a process of gathering information by making observations and comparing them to expectations.

I added two others that work well for me, from James Bach and Cem Kaner respectively, taken from this post on James Bach’s blog:

Testing: questioning a product in order to evaluate it (Bach version); technical investigation of a product, on behalf of stakeholders, with the objective of exposing quality-related information of the kind they seek (Kaner version).

I am a big fan of these three definitions, and think that both their similarities and their differences are quite interesting. I like that they emphasize testing as fundamentally a learning / questioning / investigative process. I like the Hendrickson/Emery emphasis on “…comparing them to expectations.” The open-endedness of ‘expectations’ versus the narrower ‘requirements’. This phrase reminds me that there are many documented and undocumented ways any software or system is expected to work, and harkens back to Jerry Weinberg’s definition of quality as “Value to some person”. This in turn can reminds me that we need to think about whose expectations matter to us, and what might be missing from our current map of the (typically vague, often conflicting) expectations that matter to our stakeholders.

Kaner approaches a very similar notion with a very different flavor when he says (my emphasis) “on behalf of stakeholders, with the objective of exposing quality-related information of the kind they seek.” Kaner is much more explicit that there is a certain set of stakeholders who determine what kind of information we are trying to discover. The set of expectations that we want to compare against are the set of expectations that matter to the stakeholders. These might be the stakeholders’ expectations…or they might be the expectations of others, e.g. valued customers, who matter to the stakeholders.

The origin of Elisabeth’s post was a question from a group of testers who’d been discussing the definitions of a test and testing. In the end, I appreciate knowing several definitions, precisely because of the conversation and reflection that the set as a whole generates.

Adding A Little Ambiguity To Your Plan

I was just talking with coworkers about the pros and cons of test plans with expected results. I tend to organize my testing with a series of questions to investigate, and or a rough map (often in .xls format) of areas to explore. Here we have some very well thought out test cases, including clear steps to follow and clear expected results.

Cem Kaner refers to having a test script with expected results as inattentional blindness by design. I agree wholeheartedly. In discussing this, an analogy occured to me to a 2004 Wired article about Hans Monderman, a Dutch traffic planner who has succeeded in making roads safer by making what to do more ambiguous. Here they discuss his solution to a frantic intersection that was the site of many accidents:

It’s the confluence of two busy two-lane roads that handle 20,000 cars a day, plus thousands of bicyclists and pedestrians. Several years ago, Monderman ripped out all the traditional instruments used by traffic engineers to influence driver behavior – traffic lights, road markings, and some pedestrian crossings – and in their place created a roundabout, or traffic circle. The circle is remarkable for what it doesn’t contain: signs or signals telling drivers how fast to go, who has the right-of-way, or how to behave. There are no lane markers or curbs separating street and sidewalk, so it’s unclear exactly where the car zone ends and the pedestrian zone begins. To an approaching driver, the intersection is utterly ambiguous – and that’s the point.

The general goal is to keep the driver’s brain engaged. If they enter the intersection thinking “I know just what needs to be done here”, they risk running on autopilot. If they enter and think “Whoa. This is a bit confusing, I better pay attention!” there are a lot less accidents.

I think this has clear parallels to testing. If we go to a section of the application and think “Ahh. I know just what needs to be checked here. I’ve followed this script before.” we are much more likely to miss the elephant in the middle of the room.

To quote Monderman,

“The trouble with traffic engineers is that when there’s a problem with a road, they always try to add something,” Monderman says. “To my mind, it’s much better to remove things.”

In many cases, I couldn’t agree more.

[Edit, 2007, June 1]

I originally titled this “Why You Should Make Your Test Plans Less Clear”, which at the moment seemed like an interesting teaser/title. I just changed it to “Adding A Little Ambiguity To Your Plan”. Why? “…You Should…” was meant tongue-in-cheek, but I read it again today and it irked me. Who is this yahoo telling me what I should do? Oh yeah, it’s me. And so I changed the title.

Entering the Metaweb

I am two weeks into a new position as Senior QA Engineer at Metaweb. The current plan is for me to lead testing of Freebase, our client application (currently in an invite-only Alpha). It was a bit surreal of a decision to make. It meant leaving the most fun work environment I’ve ever had – and after only 6 months, something I’ve never done before – but the opportunity to be a part of what Metaweb is trying to do is too rich an opportunity for me to pass up.

And what are we doing that’s so exciting? In short, we aim to organize the world’s knowledge – in ways that are more meaningful to machines than pages of text. We hope that Freebase will compliment the fantastic ways that Google and the Wikipedia already help us to access the information we need – solving certain problems that are challenging using current options.

You can read more about what we’re doing at the NY Times, at Tim O’Reilly of O’Reilly Books’ blog here and here, or listen to an in depth interview of our “Minister of Information”.

I am only a few weeks into the job. One of my first impressions (besides the usual excited blur as I learn a new application and explore a new domain) is that it’s very refreshing to come to an environment where there are already experienced testers. The last two places I’ve worked, I’ve enjoyed getting to create the test processes from scratch. Now I’m enjoying learning from what others have established…and providing thoughts and ideas from the perspective of an outsider coming in to an existing organization, rather than building from scratch.

Limeuristics

We’ve read James’, Mike’s and Jonathan’s compelling arguments for using mnemonics to keep useful testing heuristics at the tip of our tongue. But where, I ask you, can one find heuristics that rhyme?

In anticipation of National Limerick Day (May 12th), I humbly suggest using Limeuristics.

I’ve written a handful to get us started:

There are those who say testing’s a bore
And the moment they start it they snore.
Now my secret I’ll tell:
If you learn to test well
You’ll discover a thirst to explore.

A tester I know (who loves bisquits) -
Her peers thought her gifted by mystics.
“You’ve got it all wrong,”
She’d explain to the throng.
“It’s just that I use good heuristics”

Certifications are always a-fruitin’
To make resumes highfalutin’.
If they don’t certify
What they try to imply -
Our craft, I’m afraid they’re pollutin’

Every tester at times gets contusions
When their findings leave egos with bruisin’s.
But don’t say we’re unkind
For the things that we find
We must test to dispell our illusions

…and finally, with thanks to Chris McMahon and “the man from Japan” for the inspiration:

A limerick ’bout testing sublime
Should have meter and rhythm and rhyme
But I might bend a rule
While a bug I pursue
So don’t be surprised if I decide to add a boundary test into the last line.

Have a very happy National Limerick Day. If you have a Limeuristic you’d like me to add to the list, I’d happily add yours as well.

Cheers!

Why Do You Enjoy Testing?

My first career was as an educator. I worked for 10 years with youth of all ages, in and out of the school system. One of my favorite jobs was running leadership training programs for teenagers at Hidden Villa. I had some very gratifying times, but was feeling ready for a change just as a grant-funded position of mine was drying up. That was September of 2000, at the tail end of that wacky boom when someone could get a testing job just by being bright and willing to learn.

When I started testing, I decided to try it out for a year – pay off my tenacious student loan debt – and then decide what was next. I knew very little about what to expect, and what I discovered surprised me. I’ve now been a software tester for five and a half years and my satisfaction with the work continues to grow. Many folks in my life see that I seem pretty pleased with my work, but continue to be perplexed as to why exactly this is so stimulating for me. Recently a friend who doesn’t work in technology asked “…But don’t you miss interacting with people?”

To this friend, I started by saying that a great deal of the work I do is interactive – I commonly spend much of the day talking to other testers, to software developers, to the folks who asked for the software or who represent our users…and in fact just about everyone in the company. In my current position, I may well have regular interactions with more folks in the company than anyone else. Along the way, I get to ask myself and others thorny questions. I challenge myself to seek out and illuminate unaware assumptions (my own first, and then those of everyone else connected to the project). I imagine what the potential risks are in the product we are building. (What might be broken? What might have unintended consequences? What proposed solution might not really solve our users’ problems?) I think as creatively and as strategically as I’m able about how to explore those risks (What else haven’t I considered yet? What’s another angle this could be approached from?) and then as I explore those risks I keep thinking, generating new test ideas and refining my strategy. Along the way I am learning constantly – about the product, underlying technologies, the users we want to serve, etc.

To me, this is in many ways a dream job. My friend clearly didn’t understand. She is both a voracious reader and a writer, so my next tact was describing the books I’m currently reading to learn and grow as a tester. While I’ve learned a good deal from testing and programming books, that’s not what I’m reading at the moment. I recently finished The Logic Of Failure (mentioned in an earlier post) – a fascinating study of how our thinking can break down in the face of complex systems, often leading to dire results. The next (barely begun) book is Jerry Weinberg’s Introduction To General Systems Thinking…which I can already tell is one for me to read slowly and to reread – it is dense with insight into how complex systems work.

This meant a bit more to her. She still couldn’t quite picture what I did (which is fine) but was intrigued that social psychology and general systems theory were on a tester’s reading list, and decided based on that that whatever-it-is-I-do must be more than she thought it was.

I’ve been thinking about the job of a tester a lot recently, partly because I’ve been hiring (or attempting to hire) testers…and having a hard time. I know there’s a marketing problem here, because (a) so few folks (outside of tech companies) seem to have even heard of testing as a job, and (b) those who have heard of it tend to have heard either that it’s “a job for programmers” or that it’s “boring and repetitive”. Now, programming skills will almost always help (and sometimes are necessary) but frankly I think that the technical skills involved in testing are often easier to train folks on than the just-as-crucial creativity, organization, communication, and strategy. As far as repetitiveness: I know that every job contains repetitive elements, but I would suggest that testing well minimizes the repetitive aspects while maximizing covering new territory…because covering new territory (or finding ways to cover old territory in a new way) tends to provide more useful information about the state of the product to its stakeholders.

All that said, do you enjoy testing? If you do, why? And if you’re feeling bold enough, how do we get the word out to smart, creative, organized folks that exploring software is a fascinating and lucrative way to make a living?

A Rose By Any Other Name

There was an interesting conversation about test terminology a few weeks back on the Agile Testing list. It started with Chris McMahon forwarding an amusing post looking for a Non-Functional Tester…and led to an interesting conversation about variation amongst test terminology, and whether we should be trying to standardize it. I’m feeling the urge to sum up and synthesize what I’m

First, let me go on record stating that I think trying to hire a “non-functional tester” is an painful miss-use of language. It reminds me of a white fellow I knew in college who commented after being at a black event that “It was interesting to be the only majority in the room”.

Regarding what language to use – There are no meaningful standard definitions for testing terms of which I am aware. There was an interesting conversation about this on the software-testing yahoo group a few weeks back. Matt posted a bit about it here: http://xndev.blogspot.com/2007/02/non-functional-testing.html

When folks talk of non-functional or parafunctional testing, I think they tend to mean “all forms of testing other than testing a particular function of the software.” This tends to include some combination of: performance/load/stress, scalability, integration, usability, and security testing…and probably a good deal more.

Someone on the Agile Testing list suggested we find a way to name it positively rather than negatively. It’s a good challenge. For a term that’s used as a catch-all for “everything other than ____”, can there be a way to state it positively, other than to use a list? I tend to think that the only thing that ties these classes of tests together is that they aren’t functional tests. That in turn makes me wonder if it’s a concept with much value, whatever name we give it.

I think the real issue is that, like many terms for “everything other than ____”, it’s a funny bucket to try to define. People seem to want the bucket though, and given that I think that using a term that describes it as accurately as possible is a good thing. “Para-” can mean “beside” or “in addition to” (as in paramedic). For that reason, plus the fact that I have yet to hear a more descriptive term suggested for it, and because it’s starting to get (at least a bit of) acceptance amongst testers, parafunctional works for me.

If we go back to the description above, Parafunctional is (to me) at least as clear as non-functional, and has the additional virtue of not sounding foolish.