Testing != test execution

Testing includes:

  1. Building a useful mental model of the application under test, including what value it should provide and what risks could threaten that value,
  2. Designing powerful tests, using that model to investigate important questions about the application’s quality,
  3. Test execution (which might be automated entirely, partly, or not at all),
  4. Analyzing the results to determine if there’s a problem, how severe it might be, or otherwise answer stakeholder questions, and
  5. Test reporting that clearly communicates the impact of the bug, and how to quickly reproduce it.

We often talk about testing as if it’s only test execution, yet often the most interesting, challenging, skill-intensive aspects of testing are in creating a mental model that helps us understand the problem space, designing tests to quickly and effectively answer key questions, analyzing what specifically the problem is, and communicating it effectively.

About these ads

9 Responses to “Testing != test execution”

  1. Tal E. Says:

    this is why it’s good to have an end-to-end test management tool that covers the entire process, and not just spreadsheets or bug-tracking software.

  2. software testing nut Says:

    Actually the execution is maybe the least important. The design of the tests using a well thought out model and then the analysis and reporting are perhaps the most important. If we do no analysis afterwards, how can we figure out if what we did worked? and how can we improve without good analysis? Improving should be one of our key objectives, not only in the quality of the sw itself, but also in our testing processes and test effectiveness. Otherwise, why test at all?

  3. ronaldbwilsonjr Says:

    You are exaclty correct, in fact the test execution part is the smallest component of overall testing. Without proper planning, and designing of great test cases, you will not find the critical defects that will affect your customers.

  4. Ralf Says:

    I’m right there with you! Nice post! I want to add:
    it might be depending on which types of tests or testlevels you are working on.
    – Example: short module tests: model and test cases are easier to create
    – other example: integration testing – workflow testing: test model and creation of testcases is more complex. You have to consider for examples some interfaces between the single modules too – that means: more effort.
    When boundary testing or equivalence class building is the task: Then it also will be usefull to create a short model. General in black box testing, when it is more then single module/function testing, the creation of a model and after that the creation of the test cases will be usefull and I think the right way!

    KR, Ralf

  5. Justin Hunter (@Hexawise) Says:

    Jeff,

    I love this post. I’ve attempted to deliver similar messages myself earlier but was not as elegant and concise. From here on out, I’m just going to quote you on this topic.

    – Justin

  6. Software Testing Carnival #2 — Hexawise Says:

    […] Testing != test execution by Jeff Fry – “We often talk about testing as if it’s only test execution, yet often the most interesting, challenging, skill-intensive aspects of testing are in creating a mental model that helps us understand the problem space, designing tests to quickly and effectively answer key questions, analyzing what specifically the problem is, and communicating it effectively.” […]

  7. edge bander Says:

    thanks for sharing this good points about testing with us..
    i really liked & note down them in my note book…..

  8. ahmedashfaque Says:

    Hi Jeff
    We often forget simple things. A check list always helps. Also people do not realize a complete picture and think work is done by just doing a fraction of the work.

  9. Anna Baik Says:

    I’m going to second Justin here – I keep quoting this blog post to people, as it’s just so elegant and concise. Thank you for writing this!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: